[subkanthi]

closes: ##2394

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Read clustering columns from delta-kernel-rs and use that for data skipping.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [92d9e6b]

[subkanthi]

This PR is dependent on changes to delta-kernel-rs
created PR
ClickHouse/delta-kernel-rs#18

[subkanthi]

Audit Report: Altinity/ClickHouse PR #2395

PR: Altinity/ClickHouse#2395 — Read clustering columns from delta lake and use it for data skipping

AI audit note: This review comment was generated by AI (claude-opus-4.6).

---

Scope

This PR adds support for reading Delta Lake liquid clustering columns from domain metadata (delta.clustering) and using them for a predicate-fallback data-skipping strategy. When the full filter DAG cannot be translated to a delta-kernel expression, the code restricts the filter to only clustering-column conjuncts and pushes that weaker predicate for file-level min/max skipping.

Files changed

• src/Storages/ObjectStorage/DataLakes/DeltaLake/TableSnapshot.cpp — core logic
• src/Storages/ObjectStorage/DataLakes/DeltaLake/TableSnapshot.h — new member + methods
• src/Common/ProfileEvents.cpp — new profile event
• src/Core/Settings.cpp — new setting delta_lake_clustering_predicate_fallback
• src/Core/SettingsChangesHistory.cpp — setting history entries
• .gitmodules / contrib/delta-kernel-rs — submodule pointer change
• tests/integration/helpers/cluster.py — unrelated --ignore-pull-failures
• tests/integration/test_storage_delta/test.py — 3 new integration tests

---

Call Graph

iterate()
  -> initOrUpdateSchemaIfChanged()             [lazy, mutex-held]
  -> initOrUpdateClusteringColumns()            [lazy, mutex-held]
  -> getKernelSnapshotState()
  -> Iterator ctor
       stores: clustering_columns (copy), enable_clustering_predicate_fallback
       -> scanDataFunc() [background thread]
            -> initScanState()
                 -> getEnginePredicate(full filter)
                 [on failure + fallback enabled + clustering cols non-empty]:
                    -> restrictFilterDAGToInputs(filter, clustering_inputs)
                    -> getEnginePredicate(restricted DAG)
                    -> ffi::scan(snapshot, engine, fallback_predicate)

getClusteringColumns() [public, mutex-held]
  -> initOrUpdateClusteringColumns()

initOrUpdateClusteringColumns() [mutex-held]
  -> getKernelSnapshotState()
  -> ffi::get_domain_metadata_internal("delta.clustering")
  -> JSON parse clusteringColumns array
  -> physical-to-logical name mapping via schema->physical_names_map

---

Transition Matrix

State	Trigger	Next State
clustering_columns == nullopt	initOrUpdateClusteringColumns	clustering_columns == Names{...} or Names{}
Full predicate OK	getEnginePredicate succeeds	Use full predicate scan
Full predicate fails	exception + fallback enabled + clustering cols present	Try restricted DAG
Restricted DAG non-trivial	getEnginePredicate on restricted DAG succeeds	Use fallback predicate scan
Restricted DAG trivial / fallback fails	fallthrough	Use original no-predicate scan

---

Confirmed Defects

Medium: Clustering columns not invalidated on credential-rotation snapshot rebuild

• Impact: After a stale-token recovery path resets kernel_snapshot_state (line 760), clustering_columns remains cached from the previous snapshot state. If the table's clustering columns changed between snapshots (schema evolution), stale clustering info could cause the fallback to push wrong columns or miss new ones. In practice this is low-probability because the snapshot version is pinned per-query, but the invalidation pattern is inconsistent with how schema is handled.
• Anchor: TableSnapshot.cpp / initOrUpdateClusteringColumns and stale-token recovery path (line 760)
• Trigger: Credential rotation causes kernel_snapshot_state.reset(), then initOrUpdateClusteringColumns returns early because clustering_columns.has_value() is true from the previous fetch.
• Why defect: clustering_columns is semantically derived from the snapshot state, but is not invalidated when the snapshot state is rebuilt. The existing schema field has the same pattern (also not invalidated), so this is an inherited design issue — but the PR extends it to a new cached field without acknowledging the existing gap.
• Fix direction: When kernel_snapshot_state is reset, also reset clustering_columns to std::nullopt.
• Regression test direction: Test with a schema-evolving Delta table across credential rotation.

Code evidence (TableSnapshot.cpp, stale-token recovery):

// Line 760: kernel_snapshot_state is reset, but clustering_columns is NOT
kernel_snapshot_state.reset();
initOrUpdateSnapshot();
snapshot_stats = getSnapshotStatsImpl();
// clustering_columns still holds old value here

Code evidence (TableSnapshot.cpp, early-return guard):

void TableSnapshot::initOrUpdateClusteringColumns() const
{
    if (clustering_columns.has_value())  // <-- returns early with stale data
        return;
    // ...
}

---

Medium: restrictFilterDAGToInputs only handles AND — OR-rooted filters silently drop clustering predicates

• Impact: If the user's WHERE clause is a = 42 OR cityHash64(b) != 0 (an OR), restrictFilterDAGToInputs does NOT replace non-computable children with true — it only does so for and nodes (ActionsDAG.cpp line 3848). The entire OR becomes non-computable, so has_function is false, the fallback path is skipped, and no file skipping occurs. This is correct but surprising (no data corruption), however the LOG_INFO message and the ProfileEvent would not fire, giving no observability that the fallback was attempted and rejected.
• Anchor: TableSnapshot.cpp / initScanState fallback block; ActionsDAG.cpp:3848
• Trigger: WHERE clause uses OR with a clustering column and an untranslatable expression.
• Why defect: The fallback silently does nothing for OR predicates. A TRACE or DEBUG log for "restricted DAG was trivial, skipping fallback" would improve diagnosability.
• Fix direction: Add a log message in the else branch of if (has_function) explaining the restricted DAG was trivial.
• Regression test direction: Integration test with OR predicate verifying fallback is NOT applied but a diagnostic log exists.

Code evidence (ActionsDAG.cpp:3847-3856, only and is handled):

if (frame.node->type == ActionsDAG::ActionType::FUNCTION)
{
    const auto & name = frame.node->function_base->getName();

    /// Replace non-computable child in "and" with constant true.
    if (name == "and")
    {
        auto const_column = child->result_type->createColumnConst(0, 1);
        copy_map[child] = &actions.addColumn(std::move(const_column), child->result_type, child->result_name);
        it->second = true;
    }
    // NOTE: "or" is not handled — the non-computable child makes the whole OR non-computable
}

Code evidence (TableSnapshot.cpp, no logging on trivial DAG):

if (has_function)
{
    // ... fallback logic ...
}
// else: no log, no metric — silently falls through

---

Low: initOrUpdateClusteringColumns depends on schema but does not call initOrUpdateSchemaIfChanged

• Impact: initOrUpdateClusteringColumns reads schema.has_value() and schema->physical_names_map (line 1148) without ensuring schema has been initialized. In the current call site (iterate), initOrUpdateSchemaIfChanged is always called first (line 985), so this is safe today. But getClusteringColumns (line 1081-1085) calls initOrUpdateClusteringColumns WITHOUT calling initOrUpdateSchemaIfChanged first. If getClusteringColumns is called before iterate on a fresh TableSnapshot where schema == nullopt, then initOrUpdateClusteringColumns will take the else branch (line 1160: logical_columns = std::move(physical_columns)) and return physical names instead of logical names, which would not match the filter DAG's column names.
• Anchor: TableSnapshot.cpp:1081-1085 (getClusteringColumns) and TableSnapshot.cpp:1148 (initOrUpdateClusteringColumns)
• Trigger: Call getClusteringColumns before iterate or getTableSchema on a fresh snapshot with physical name mapping.
• Why defect: Missing precondition — the function depends on schema being populated but does not ensure it.
• Fix direction: Call initOrUpdateSchemaIfChanged at the start of initOrUpdateClusteringColumns, or at the start of getClusteringColumns.
• Regression test direction: Unit test calling getClusteringColumns on a fresh snapshot with physical name mapping before any other accessor.

Code evidence (TableSnapshot.cpp:1081-1085, missing schema init):

const DB::Names & TableSnapshot::getClusteringColumns() const
{
    std::lock_guard lock(mutex);
    initOrUpdateClusteringColumns();  // does NOT call initOrUpdateSchemaIfChanged()
    return *clustering_columns;
}

Code evidence (TableSnapshot.cpp:1148, conditional on schema):

DB::Names logical_columns;
if (schema.has_value() && !schema->physical_names_map.empty())
{
    // uses schema->physical_names_map for reverse mapping
}
else
{
    // falls through to physical names — wrong if schema just wasn't loaded yet
    logical_columns = std::move(physical_columns);
}

---

Low: Settings history has duplicate entry across two version blocks

• Impact: delta_lake_clustering_predicate_fallback is registered in both the "26.7" and "26.6" blocks of SettingsChangesHistory.cpp. This is likely an error from rebasing. Having the same setting in two version blocks may cause confusion during upgrades, and the compatibility setting resolution picks the latest version entry, so the 26.7 entry wins and the 26.6 entry is dead code.
• Anchor: src/Core/SettingsChangesHistory.cpp (both version blocks)
• Trigger: N/A — cosmetic / maintenance issue.
• Why defect: Redundant entry across version blocks; should be in exactly one.
• Fix direction: Remove the entry from the version block where it does not belong (likely keep 26.7 only, or whichever is the target version).
• Regression test direction: N/A.

Code evidence (SettingsChangesHistory.cpp, both entries):

// In the 26.7 block:
{"delta_lake_clustering_predicate_fallback", false, true,
 "New setting to enable clustering-aware predicate fallback ..."},

// In the 26.6 block:
{"delta_lake_clustering_predicate_fallback", false, true,
 "New setting to enable clustering-aware predicate fallback ..."},

---

Coverage Summary

• Scope reviewed: TableSnapshot.cpp (Iterator ctor, initScanState fallback, initOrUpdateClusteringColumns, iterate, getClusteringColumns), TableSnapshot.h (new members), SettingsChangesHistory.cpp, Settings.cpp, ProfileEvents.cpp, cluster.py, test.py (3 new tests).
• Categories passed: JSON parsing (well-guarded with Poco exception catch), FFI resource management (uses KernelPointerWrapper RAII wrappers correctly), predicate fallback logic (correct for AND-only filters), physical-to-logical name mapping (correct when schema is populated), thread safety of new members (Iterator copies values, no shared mutable state in hot path), integration tests (cover basic read, data skipping, and predicate fallback).
• Categories failed: Stale cache invalidation (clustering_columns not reset on snapshot rebuild), missing schema precondition in getClusteringColumns, duplicate settings history entry.
• Assumptions/limits: Static analysis only; delta-kernel-rs submodule changes (get_domain_metadata_internal FFI) were not inspected beyond the C++ call site. The restrictFilterDAGToInputs behavior was verified by reading its implementation.

[subkanthi]

AI audit note: This review comment was generated by AI (claude-4.6-opus).

Audit update for PR Altinity/ClickHouse#2395 (Delta Lake liquid-clustering predicate fallback):

---

Scope

Partition	Files
Core feature (predicate fallback)	TableSnapshot.cpp, TableSnapshot.h
Settings / profile events	ProfileEvents.cpp, Settings.cpp, SettingsChangesHistory.cpp
Submodule change	.gitmodules, contrib/delta-kernel-rs
Test infra	tests/integration/helpers/cluster.py
Integration tests	tests/integration/test_storage_delta/test.py

---

Call Graph

iterate()                          [locks mutex]
  ├─ initOrUpdateSchemaIfChanged()
  │    └─ getKernelSnapshotState() → initOrUpdateSnapshot()
  ├─ initOrUpdateClusteringColumns()          ← NEW
  │    ├─ getKernelSnapshotState()
  │    ├─ ffi::get_domain_metadata_internal()  ← NEW FFI call
  │    ├─ Poco::JSON parse
  │    └─ physical→logical column-name remap
  └─ new Iterator(…, *clustering_columns, …)
       └─ [background thread] scanDataFunc()
            └─ initScanState()
                 ├─ getEnginePredicate(filter)
                 ├─ ffi::scan(predicate)
                 └─ [if exception + fallback enabled + clustering_cols non-empty]  ← NEW
                      ├─ restrictFilterDAGToInputs(clustering_inputs)
                      ├─ has_function check
                      ├─ getEnginePredicate(restricted_dag)
                      └─ ffi::scan(fallback_predicate)

getClusteringColumns()             [locks mutex]           ← NEW public API
  └─ initOrUpdateClusteringColumns()

getSnapshotStats() stale-token retry:
  └─ clustering_columns.reset()                            ← NEW invalidation

---

Confirmed defects:

Low: Trivial AND(true, true) passes has_function gate, causing wasted ffi::scan

• Impact: When the original predicate fails to translate and no filter conjunct references a clustering column, restrictFilterDAGToInputs produces AND(true, true). The AND node passes the has_function check, so getEnginePredicate + ffi::scan are called for a predicate that is semantically equivalent to "no filter". The result is a second scan object that provides zero data-skipping benefit. Not a correctness issue — the scan result is identical — but it adds unnecessary latency (one extra FFI round-trip).
• Anchor: TableSnapshot.cpp / initScanState() — the has_function loop (~line 220).
• Trigger: WHERE cityHash64(b) != 0 AND length(c) > 5 on a table clustered by column a (no clustering column in the filter, but top-level AND with 2+ non-computable branches).
• Why defect: Wasted FFI work on every such query; the trivial-predicate scan is functionally identical to the already-created no-predicate scan.
• Fix direction: After restrictFilterDAGToInputs, also check that at least one input node in the restricted DAG matches a clustering column (i.e. the restricted DAG actually references a clustering column, not just constants).
• Regression test direction: Add a case where the filter has no clustering-column conjuncts and verify DeltaLakeClusteringPredicateFallbacks is not incremented.

---

Coverage summary:

• Scope reviewed: All changed C++ code (TableSnapshot.cpp/.h, Settings.cpp, SettingsChangesHistory.cpp, ProfileEvents.cpp), integration tests, docker infra change.
• Categories executed & passed:
  • Thread safety / data-race analysis — pass (Iterator owns copies of all mutable data; clustering_columns is guarded by mutex with TSA annotations; no shared mutable access from the background scan thread).
  • Memory lifetime / use-after-free — pass (filter is cloned; clustering_columns is copied by value into Iterator; restricted_dag and fallback_predicate are locals with clear lifetimes).
  • Iterator / reference invalidation — pass (no container mutation concurrent with iteration).
  • Exception safety / partial-update — pass (initOrUpdateClusteringColumns either sets the optional or throws; clustering_columns stays nullopt on failure, safe to retry).
  • Cache invalidation consistency — pass (stale-token retry in getSnapshotStats resets clustering_columns; normal credential-rotation in initOrUpdateSnapshot does not need to reset because the snapshot version is pinned and domain metadata is version-stable).
  • Integer overflow / signedness — pass (loop indices are size_t; static_cast<unsigned int>(i) for Poco API is safe within realistic array sizes).
  • Lock-order / deadlock — pass (single mutex; no nested lock acquisition).
  • Rollback / partial-update — pass (scan replacement in fallback is a simple move-assignment; no intermediate state inconsistency).
  • Settings history — pass (added to both "unreleased" and "26.6" sections, consistent with existing entries in the Altinity fork).
  • Logical name consistency — pass (clustering_columns are mapped to logical names; filter DAG uses logical names; restrictFilterDAGToInputs and getEnginePredicate both operate in the logical-name space).
  • JSON parsing robustness — pass (getArray null checks present; empty sub-arrays skipped via continue; Poco exceptions caught and re-thrown as DB::Exception).
  • FFI ownership — pass (result from get_domain_metadata_internal wrapped in unique_ptr<std::string> matching KernelUtils::allocateString contract).
• Categories not applicable:
  • Security / auth bypass — N/A (feature is read-only data skipping; no auth changes).
  • Protocol / API parity — N/A (no new external API; internal-only change).
• Assumptions / limits: Static analysis only; delta-kernel-rs submodule update (Altinity fork, commit 625c2fa) was not audited at the Rust level. The behavior of ffi::get_domain_metadata_internal and restrictFilterDAGToInputs was assessed by API contract, not by reading their full implementations.

Additional observations (not defects):

1. --ignore-pull-failures in cluster.py: This is a docker-compose v1 flag. Docker Compose v2 added it later for backward compatibility, but very old v2 releases may not support it. If the CI images use a sufficiently recent Docker Compose v2, this is fine. Otherwise it could cause a noisy (but non-fatal) warning.

2. getClusteringColumns() public API: Currently only called indirectly via iterate(). If called independently before initOrUpdateSchemaIfChanged() (e.g., from a new caller), the physical→logical name mapping would fall back to identity (the else branch at line ~1164), which is correct for most tables but could diverge for tables with column mapping enabled. Not a current bug since iterate() calls initOrUpdateSchemaIfChanged() first.

[subkanthi]

TESTING

docker run --rm -i \
  --user 0:0 \
  -v "$PWD/tmp/liquid_delta:/delta" \
  -v "$PWD/tmp/ivy:/tmp/ivy" \
  --entrypoint /opt/spark/bin/spark-sql \
  apache/spark:3.5.3 \
  --master 'local[2]' \
  --packages io.delta:delta-spark_2.12:3.3.0 \
  --conf spark.driver.extraJavaOptions='-Divy.cache.dir=/tmp/ivy -Divy.home=/tmp/ivy' \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  --conf spark.sql.warehouse.dir=/tmp/warehouse \
  --conf spark.sql.files.maxRecordsPerFile=500 \
  --conf spark.databricks.delta.optimize.maxFileSize=32768 <<'SQL'
CREATE TABLE delta.`/delta/table` (a INT, b STRING)
USING DELTA
CLUSTER BY (a)
TBLPROPERTIES (
  'delta.checkpointPolicy' = 'v2',
  'delta.checkpointInterval' = '1'
);

INSERT INTO delta.`/delta/table` SELECT id, CAST(id AS STRING) FROM range(0, 1000);
INSERT INTO delta.`/delta/table` SELECT id, CAST(id AS STRING) FROM range(1000, 2000);
INSERT INTO delta.`/delta/table` SELECT id, CAST(id AS STRING) FROM range(2000, 3000);
INSERT INTO delta.`/delta/table` SELECT id, CAST(id AS STRING) FROM range(3000, 4000);
INSERT INTO delta.`/delta/table` SELECT id, CAST(id AS STRING) FROM range(4000, 5000);

OPTIMIZE delta.`/delta/table` FULL;
DESCRIBE DETAIL delta.`/delta/table`;
SHOW TBLPROPERTIES delta.`/delta/table`;
SELECT count(*), min(a), max(a), sum(a) FROM delta.`/delta/table`;
SQL

mkdir -p /root/Documents/ClickHouse/programs/server/user_files
mv /root/Documents/ClickHouse/tmp/liquid_delta/table \
   /root/Documents/ClickHouse/programs/server/user_files/liquid_delta_table

Ubuntu-2404-noble-amd64-base :) SELECT count(), min(a), max(a), sum(a)
FROM deltaLakeLocal('/root/Documents/ClickHouse/programs/server/user_files/liquid_delta_table');

SELECT
    count(),
    min(a),
    max(a),
    sum(a)
FROM deltaLakeLocal('/root/Documents/ClickHouse/programs/server/user_files/liquid_delta_table')

Query id: 0a665eca-7f52-42b5-8e56-53bbdb4305bb

   ┌─count()─┬─min(a)─┬─max(a)─┬───sum(a)─┐
1. │    5000 │      0 │   4999 │ 12497500 │ -- 12.50 million
   └─────────┴────────┴────────┴──────────┘

[Ubuntu-2404-noble-amd64-base] 2026.10.08 19:06:07.807155 [ 3215907 ] {dca3e07d-9491-4aba-9eaa-9023cb26d1f3} <Trace> DeltaLakeMetadata: Clustering columns: a

[ianton-ru]

Must be only in Antalya section above.

[subkanthi]

removed.

[ianton-ru]

I did not find where is this method called. Is it used, or just dead code?

[subkanthi]

yes this function is not used, removed.

[ianton-ru]

I think make sense to add a table name in message.

[subkanthi]

TableSnapshot does not store the table name, but added the table location. IKernelHelper::getTableLocation()

[ianton-ru]

Make sense to add table name here too.

[subkanthi]

Same here added IKernelHelper::getTableLocation()

[ianton-ru]

I thing BAD_ARGUMENTS is incorrect code here. It is used for arguments in queries, when here is some metadata incosistency.

[subkanthi]

Replaced with INCORRECT_DATA error code.

[ianton-ru]

The same as above about error code.
And what is going on when this exception thrown? Does customer have an opinion to execute query in old way?

[subkanthi]

Replaced with INCORRECT_DATA error code.

[ianton-ru]

As I understand this method called in any case, but clustering_columns is used only when delta_lake_clustering_predicate_fallback=1. Make sense to check this setting here to avoid useless snapshot parsing when feature is turned off.

[subkanthi]

Added logic to run initOrUpdateClusteringColumns only if delta_lake_clustering_predicate_fallback is set.

const bool need_clustering_columns = filter_dag
        && settings[DB::Setting::delta_lake_enable_engine_predicate]
        && settings[DB::Setting::delta_lake_clustering_predicate_fallback];
    if (need_clustering_columns)
        initOrUpdateClusteringColumns();

[subkanthi]

@ianton-ru can u please check this, i have addressed all comments.

[ianton-ru]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-23T13:56:42.648494Z	e8453b9	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9dbbc8ab0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Retry clustering metadata reads after credential rotation

When a cached snapshot uses catalog-vended temporary credentials that refresh only after an ExpiredToken, this new domain-metadata read runs with the stale engine and lets the resulting DELTA_KERNEL_ERROR escape. Because it happens before constructing Iterator, the existing scan retry cannot call refreshCredentials; consequently, the first filtered query after token expiry fails even though ordinary scans and stats recover. Apply the same refresh/rebuild/one-shot retry used by the other snapshot operations around this FFI call.

Useful? React with 👍 / 👎.

[subkanthi]

Addressed this

[chatgpt-codex-connector]

Keep pull failures visible to the retry wrapper

For any ordinary registry failure, this flag suppresses the failure that run_and_check must observe to trigger its three retries. Docker documents --ignore-pull-failures as pulling what it can while ignoring failed images (Docker CLI reference); therefore a flaky pull can silently leave a stale same-tag image in place and run integration tests against it, or defer a missing-image failure until compose up without the intended pull retries. Limit the exception to the known local-only image rather than applying it to every service image.

AGENTS.md reference: AGENTS.md:L153-L153

Useful? React with 👍 / 👎.

[subkanthi]

addressed

[chatgpt-codex-connector]

Record the setting at its actual release version

This commit is based on 26.6.8.20002, but the newly introduced setting is recorded as having changed in 26.6.2.20001. As a result, a user selecting compatibility with any intervening release such as 26.6.7 will not have the setting reverted to false, even though that release predates the feature. Move this history entry to the version that actually ships the new default so compatibility profiles reproduce the requested release behavior.

Useful? React with 👍 / 👎.

[subkanthi]

moved to 26.6.8.20002 section

[ianton-ru]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8453b94bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Trigger fallback for unsupported predicate nodes

For the intended a = 42 AND cityHash64(b) != 0 case, unsupported functions return VISITOR_FAILED_OR_UNSUPPORTED from EngineIterator::getNextImpl without setting engine_predicate_exception; only thrown exceptions populate that pointer. Consequently this condition is false, the clustering fallback and its profile event never run, and the newly added fallback test observes zero hits. Detect unsupported translation explicitly rather than using only the exception pointer.

Useful? React with 👍 / 👎.

[subkanthi]

addressed

[chatgpt-codex-connector]

Force the baseline query to scan files

With delta-kernel enabled, an unfiltered SELECT count() uses totalRows() and the trivial-count optimization, whose stats visitor does not increment DeltaLakeScannedFiles. Therefore full_scan_files is zero while the filtered point query scans at least one file, so the later point_scan_files < full_scan_files assertion fails instead of testing pruning. Disable optimize_trivial_count_query for this baseline or read an actual column.

Useful? React with 👍 / 👎.

[subkanthi]

addressed

[ianton-ru]

@balu-ai

[subkanthi]

@blau-ai

[blau-ai]

CI triage for data_lake_clustering_read @ e8453b9

Verdict: 6 failing checks — 1 PR-caused, 5 pre-existing / flaky / infra.

The only failure actually caused by this PR is the settings regression snapshot (it flags the newly added delta_lake_clustering_predicate_fallback setting — expected whenever a setting is added). The other five are in test areas this PR doesn't touch (CAS-S3 I/O, stress hung-check, swarm node restart, lightweight-delete concurrency/hard-restart) and are unrelated to the DeltaLake read path.

---

🔴 PR-caused

RegressionTestsRelease / Common (settings) / settings

/settings/default values/delta_lake_clustering_predicate_fallback — Fail
tests/default_values.py line 40 in 'check_default_value'
snapshot: default_values.py.default values>=26.6_antalya.snapshot

This PR adds a new server setting:

DECLARE(Bool, delta_lake_clustering_predicate_fallback, true, ...)   // src/Core/Settings.cpp

The regression settings suite compares the live system.settings default values against a checked-in snapshot, and the new setting isn't in that snapshot yet, so check_default_value fails. This is the normal "new setting ⇒ snapshot out of date" signal, not a code bug.

Fix: regenerate the default-values snapshot in the regression repo (Altinity/clickhouse-regression, file settings/tests/snapshots/default_values.py.default values>=26.6_antalya.snapshot) so it includes delta_lake_clustering_predicate_fallback. That file lives outside this repo (ClickHouse itself), so it can't be fixed by a commit on this PR branch — it needs a snapshot-update PR on clickhouse-regression (or a maintainer re-run with snapshot regeneration). Settings.cpp / SettingsChangesHistory.cpp are already correct here (the change is recorded under 26.6.x).

---

🟡 Not PR-caused (flaky / infra / pre-existing)

Stateless tests (amd_binary, cas s3 storage, parallel) — 1 failed
04023_rollup_totals_memory_limit: expected serverError 241 (MEMORY_LIMIT_EXCEEDED) but got 210 (NETWORK_ERROR):

CAS write could not be committed (... gave up at the policy deadline after 38 attempt(s)); retrying later
... (NETWORK_ERROR)

A CAS-S3 commit/read gave up at its retry deadline — a storage-backend/infra hiccup, nothing to do with rollup/memory or DeltaLake. Note it only fails in the cas s3 storage variant; the amd_asan_ubsan, cas s3 storage and plain variants of the same suite passed. → Safe to re-run.

Stress test (amd_asan_ubsan) — Failures 1/5
Only sub-result: Hung check failed, possible deadlock found. No specific test is reported and there are zero references to delta/clustering/TableSnapshot in the hung-check output (the stacks are generic FilterStep / dictGet query paths). Hung-check is a well-known flaky stress result. The amd_debug, amd_debug cas s3 and amd_asan_ubsan cas s3 stress variants all passed. → Re-run; not PR-related.

Regression release swarms — 1 scenario failed
/swarms/feature/node failure/check restart swarm node (node_failure.py:108 run_long_query). A swarm-node restart-resilience test — timing/resource sensitive and unrelated to DeltaLake. → Flaky/infra, re-run.

Regression release cas_lightweight_delete_4 — 1 scenario failed
/lightweight delete/concurrent delete/VersionedCollapsingMergeTree/concurrent delete without overlap. Concurrent-delete timing test on VersionedCollapsingMergeTree under CAS-S3; no DeltaLake code involved. → Flaky, re-run.

RegressionTestsRelease / LightweightDelete (2, hard restart) / lightweight_delete_2
/lightweight delete/hard restart/SIGKILL/hard restart. SIGKILL hard-restart recovery test — infra/timing sensitive, unrelated to this PR. → Flaky/infra, re-run.

---

Suggested next steps

1. Update the default_values snapshot in Altinity/clickhouse-regression for the new delta_lake_clustering_predicate_fallback setting (the one real fix).
2. Re-run the five flaky/infra checks (stateless cas-s3, stress asan_ubsan, swarms, both lightweight-delete suites). If any reproduces across re-runs I'll re-triage.

(The core builds, stateless/integration/unit tests, and the targeted DeltaLake integration test test_storage_delta are all green, so the DeltaLake change itself looks healthy.)

[subkanthi]

@blau-ai

[blau-ai]

CI triage for #2395

Summary: none of the failures look caused by this PR. I checked 7 failing suites plus DCO. All the failures are either already on antalya-26.6 or are known infra/flaky issues.

There's no CI result yet for the current head 2f70736 (the merge of antalya-26.6, pushed 02:38 UTC). The run has only just started, and the long "skipping" list in gh pr checks comes from the Community PR workflow, which is expected. So this triage covers the last finished report, for b86ee04 (result_pr.json).

The PR only changes DeltaLake (EnginePredicate, TableSnapshot), settings, ProfileEvents, the delta-kernel-rs submodule and test_storage_delta. None of the failing tests touch Delta Lake. Integration tests (amd_asan_ubsan, targeted) passed (0/14 failures), and no shard reported a test_storage_delta failure.

Check	Failing tests	Verdict
Integration tests (amd_asan_ubsan, db disk, old analyzer, 3/8)	test_database_glue/* (29), test_storage_iceberg_no_spark/test_writes_statistics_by_minmax_pruning (6)	Already on base
Integration … 4/8	test_storage_iceberg_with_spark/test_minmax_pruning (12), test_remote_initiator::…global_join_without_analyzer[s3]	Already on base
Integration … 7/8	test_cas_gcs/* (37), test_datalake_glue_token_forwarding/* (3), test_storage_s3/test_sts.py (4)	Already on base
Integration … 8/8	test_cas_gcs_relink_liveness::test_both_queues_drain_under_slow_checkpoints	Already on base
Stateless (amd_asan_ubsan, distributed plan, parallel, 2/2)	05387_s3_cluster_remote_initiator_keeps_stripped_join	Already on base
same job	05055_format_schema_source_query_background_access	Not PR-related: a test added to base 2 days ago fails under randomized settings
Stateless (amd_binary, cas s3 storage, parallel) / (amd_asan_ubsan, cas s3 storage, parallel, 1/2)	Logical error: GroupingAggregatedTransform pushed bucket A twice (STID: 3994-4c1d) → Server died	Already on base (server crash, unrelated)
DCO	—	Infra/policy: not required, and fails on other Altinity PRs too

Evidence

• Integration failures (glue / iceberg minmax / cas_gcs / sts) and 05387: unrelated PR Antalya 26.6 Backport of #102156 - Fix Nullable(X)->Y cross-type ALTER MODIFY COLUMN #2713 (a backport of a Nullable ALTER fix) on the same base fails the same set: 29× test_database_glue, 37× test_cas_gcs, 12× test_minmax_pruning, 6× test_writes_statistics_by_minmax_pruning, 4× test_sts, 3× test_datalake_glue_token_forwarding, test_cas_gcs_relink_liveness, test_remote_initiator, and 05387. So these come from the base branch.
  • 05387 failure: Code: 499 … Failed to get object info … HTTP response code: 403 … while reading '05387_test_….tsv' in bucket 'test'. The diagnosis step pins it to --prefer_localhost_replica 0.
• GroupingAggregatedTransform pushed bucket A twice: the same logical error and server death show up on unrelated PR Antalya 26.6 Backport of #123436 - 26.8: Fix ALIAS row policies on legacy distributed reads #2655 (an ALIAS row-policy backport). It's an aggregation-pipeline crash and has nothing to do with Delta reading.
• 05055_format_schema_source_query_background_access:

  -background flush refused: 1
  +background flush refused: 0
  

  The diagnosis says "Passes without randomization" (6/6), and the culprit list includes enable_parallel_replicas 1 / prefer_localhost_replica 0. The test only reached antalya-26.6 on 2026-10-26 (via the Antalya 26.6 Backport of #127810 - Backport #127419 to 26.8: Bump brotli from v1.2.0 to v1.2.1 #2685 merge), and it tests format-schema access during background Buffer flushes, not Delta. I found no existing issue for it, and it hasn't shown up on other PRs yet. It probably needs a no-parallel-replicas-style tag or a fix on the base branch, not in this PR.
• DCO: the commits have no Signed-off-by, and Antalya 26.6 Backport of #102156 - Fix Nullable(X)->Y cross-type ALTER MODIFY COLUMN #2713 fails DCO in the same way. It isn't a required check.

Next steps

• No code changes needed in this PR for CI.
• Wait for run 40622141656 on 2f70736. The failures above should repeat until the base branch fixes them.
• A follow-up on antalya-26.6 would help for 05055 (randomized-settings and parallel-replicas sensitivity), and the glue/cas_gcs/sts/iceberg-minmax integration breakage deserves its own issue.

_{Triage based on the praktika S3 reports for this PR, #2713 and #2655. Nothing was re-run locally.}

[subkanthi]

@blau-ai

[blau-ai]

CI triage for 92d9e6b

Summary: 6 failing checks. 1 failure is caused by this PR, and it only needs a snapshot update in the regression repo. Everything else also fails on unrelated antalya-26.6 PRs or is flaky. Several suites (old-analyzer integration, stress, sequential stateless, SQLLogic) were still running when I looked, so this covers only the failures reported so far.

Check	Failure	Verdict
Regression release settings	/settings/default values/delta_lake_clustering_predicate_fallback	🔴 PR-caused (new setting has no snapshot)
Regression release settings	iceberg_min_data_file_size_bytes, iceberg_target_data_file_size_bytes	⚪ pre-existing (settings that come from the base branch)
Regression release ssl_server_3	/ssl server/part 3/zookeeper/secure connection	⚪ pre-existing / base-branch change
Integration tests (amd_asan_ubsan, targeted), 19/34	iceberg test_minmax_pruning ×12, test_writes_statistics_by_minmax_pruning ×6, test_remote_initiator_global_join_without_analyzer[s3]	⚪ pre-existing
Stateless (amd_asan_ubsan, distributed plan, parallel, 2/2)	05387_s3_cluster_remote_initiator_keeps_stripped_join	⚪ pre-existing
Stateless (amd_asan_ubsan, distributed plan, parallel, 2/2)	01661_extract_all_groups_throw_fast	⚪ flaky (0/3 reruns failed)
Stateless (amd_asan_ubsan, distributed plan, parallel, 1/2)	Logical error: GroupingAggregatedTransform pushed bucket A twice → Server died	⚪ pre-existing

---

🔴 Regression release settings: delta_lake_clustering_predicate_fallback

report

/settings/default values/delta_lake_clustering_predicate_fallback  Fail
AssertionError ... SnapshotNotFoundError(

The settings suite compares the default value of every setting with a stored snapshot. This PR adds DECLARE(Bool, delta_lake_clustering_predicate_fallback, true, ...) in src/Core/Settings.cpp, and the snapshot has no entry for it yet. This is expected for any new setting, and the setting code itself is fine.
Fix: add a snapshot entry for delta_lake_clustering_predicate_fallback (default 1) to the settings suite in Altinity/clickhouse-regression, or have the regression team regenerate the snapshots for 26.6. No change is needed in this PR.

The other two failures in the same suite (iceberg_min_data_file_size_bytes, iceberg_target_data_file_size_bytes) are also SnapshotNotFoundError. Those settings come from the base branch (they sit in the 26.6.8.20003.altinityantalya block of SettingsChangesHistory.cpp), not from this PR. The same snapshot update should cover them.

⚪ Regression release ssl_server_3: zookeeper secure connection

Code: 999. DB::Exception: All connection tries failed while connecting to ZooKeeper. nodes: secure://172.18.0.3:2281
Poco::Exception. Code: 1000, e.code() = 0, Certificate validation error: Unacceptable certificate from 172.18.0.3: application verification failure

This is a TLS certificate validation failure on the ClickHouse→Keeper connection. This PR only touches DeltaLake code. The likely cause is #2669 ("Verify the server certificate hostname on outbound TLS connections"), which was merged into antalya-26.6 on 2026-10-25. CI runs on the PR merge ref, so that change is in this build. Here the node is reached by IP (172.18.0.3), so the certificate is now probably rejected by the hostname/IP check. In #2669's own CI the SSLServer job was skipping, so this suite has likely not run since the change landed. I couldn't confirm the cause from a base-branch run, though.
Next step: not for this PR. The ssl_server regression's zookeeper certificates (SAN) or config probably need updating after #2669.

⚪ Integration tests (amd_asan_ubsan, targeted): iceberg minmax pruning + remote initiator

test_minmax_pruning.py:116          E   assert 0 == 3   (pruned files)
test_writes_statistics_by_minmax_pruning.py:89   E   assert 0 == 3
test_remote_initiator_global_join_without_analyzer[s3]: Code: 206 ... No alias for subquery or table function in JOIN

These are the same 19 tests, failing the same way, in the old-analyzer integration shards of unrelated backport PRs #2698 and #2685. Nothing here goes through the DeltaLake code this PR changes. The PR's test_storage_delta changes made the "targeted" job run these shards, which is why they show up here. Safe to ignore for this PR.

⚪ Stateless: 05387_s3_cluster_remote_initiator_keeps_stripped_join

The test fails only with the randomized --prefer_localhost_replica 0 (S3_ERROR ... HTTP response code: 403) and passes 35/35 without randomization. The same test fails in the same suite on #2685. It's pre-existing and not related to DeltaLake.

⚪ Stateless: 01661_extract_all_groups_throw_fast

The test expected TOO_LARGE_ARRAY_SIZE (128) and got code 241 (MEMORY_LIMIT_EXCEEDED) under ASan, then failed 0 of 3 reruns. That is a timing/memory flake, and the query doesn't touch any code this PR changes.

⚪ Stateless 1/2: GroupingAggregatedTransform pushed bucket 4 twice (server died)

This is a logical error in MergingAggregatedMemoryEfficientTransform.cpp:63, which is in aggregation and has nothing to do with DeltaLake. The same STID 3994-4c1d also fires on #2698 (Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)). It's a known pre-existing crash on the base branch.

---

Bottom line: none of the failures point to a bug in the DeltaLake clustering code. The only PR-caused item is the missing settings snapshot for the new setting, and that is fixed in clickhouse-regression, not here. I didn't push any changes to this branch.

[subkanthi]

@ianton-ru can u please check