Iceberg nullable minmax pruning by ianton-ru · Pull Request #2657 · Altinity/ClickHouse

ianton-ru · 2026-10-23T13:54:26Z

Changelog category (leave one):

Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Iceberg file min/max pruning and Parquet row-group and page min/max pruning can now skip a nullable column when its non-null range misses the predicate and SQL NULL cannot make the predicate true. IS NULL, assumeNotNull, and IN / NOT IN with transform_null_in still read a chunk that contains NULL.

Documentation entry for user-facing changes

No new setting or syntax. Existing Iceberg manifest min/max pruning and Parquet filter pushdown (input_format_parquet_filter_push_down, input_format_parquet_page_filter_push_down) apply the predicate to the non-null bounds of a nullable column.

CI/CD Options

Exclude tests:

Regression jobs to run:

Motivation

Unity liquid clustering on Iceberg is a sort order plus per-file lower_bounds / upper_bounds. It is not a partition spec. For a nullable clustering column, ClickHouse already had those bounds, but it refused to use them unless null_value_counts was present and exactly 0. Parquet did the same for row groups and pages: a nullable column with any null widened the min/max to infinity, so a predicate such as partner_id = 'sg2o27m' could not skip a chunk whose other values missed the probe.

A null does not satisfy that comparison. The chunk can be skipped. IS NULL must still read it.

Related: #2395 reads Delta delta.clustering only. It does not change the Iceberg or Parquet readers.

Approach

KeyCondition::mayBeTrueOnNull walks the predicate and reports whether it can be true when one key column is SQL NULL. A bare comparison is unknown on NULL, so WHERE does not keep that row. IS NULL is true. IS NOT NULL is false.

Iceberg manifest pruning skips a file when the non-null range misses and either the manifest says the column has no nulls or mayBeTrueOnNull is false. Parquet row-group and page stats keep the finite non-null range in that same case. An all-null row group has no finite min/max and is dropped only when NULL cannot match.

Two predicates are true for NULL and must not be pruned away:

assumeNotNull replaces NULL with the nested value, so a monotonic wrapper is treated as able to match.
nullIn / notNullIn, which transform_null_in rewrites IN / NOT IN into, can match NULL. The RPN stores both the normal and the null-comparing forms as the same set atom, so a chunk that contains NULL is kept for every IN / NOT IN.

A null-preserving wrapper such as toDate is also kept when the chunk contains NULL. The rows are unchanged. The chunk is not pruned from that column's bounds.

Tests

05386_iceberg_nullable_minmax_prunes_null_rejecting_predicate: an Iceberg file whose non-null range misses partner_id = '...' or NOT partner_id = '...' is pruned even if it contains NULL. IS NULL keeps it.
05387_parquet_nullable_minmax_null_rejecting: the same rule for Parquet row groups and pages, including an all-null row group, assumeNotNull, and IN / NOT IN with transform_null_in = 1.

…ch NULL. A clustering column can contain nulls while its manifest bounds still describe the other values. `partner_id = '...'` cannot match those nulls, so the file can be skipped. `IS NULL` still keeps a file that may contain nulls. Co-authored-by: Cursor <cursoragent@cursor.com>

…cate cannot match NULL. `KeyCondition::mayBeTrueOnNull` is the shared check for Iceberg files and Parquet chunks. An equality that rejects NULL still skips a chunk whose other values miss, including an all-null row group. `IS NULL` keeps it. Co-authored-by: Cursor <cursoragent@cursor.com>

…in` matches NULL. `mayBeTrueOnNull` treated every comparison and `IN` as unknown on NULL, so a chunk whose other values missed the probe was skipped even though the NULL rows matched. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru · 2026-10-23T13:54:46Z

@codex review

chatgpt-codex-connector · 2026-10-23T13:55:05Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-27T09:20:59.700170Z	`3be5d18`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions · 2026-10-23T13:56:12Z

Workflow [PR], commit [0a43e58]

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9732cac9ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-23T14:10:28Z

+            case RPNElement::FUNCTION_IN_RANGE:
+            case RPNElement::FUNCTION_NOT_IN_RANGE:
+            case RPNElement::FUNCTION_ARGS_IN_HYPERRECTANGLE:
+            case RPNElement::FUNCTION_POINT_IN_POLYGON:
+                /// A bare comparison is unknown on NULL. `assumeNotNull` and other wrappers can make it true.
+                stack.push_back(uses_this_column && element.monotonic_functions_chain.empty()
+                    ? TruthOnNull::isUnknown() : TruthOnNull::any());


Preserve null-safe comparison semantics under NOT

When the filter is NOT isNotDistinctFrom(nullable_col, c), KeyCondition represents the inner null-safe equality as FUNCTION_IN_RANGE, but this branch classifies its result on NULL as unknown; FUNCTION_NOT therefore leaves it unknown even though the actual result is true. For a Parquet row group/page or Iceberg file containing NULLs whose non-null values are all c, the finite range makes the negated predicate appear impossible and the new pruning path drops the chunk, losing every NULL row that should match. Preserve the atom's null-safe origin/null truth value in the RPN, or conservatively keep nullable chunks when that distinction is unavailable.

Useful? React with 👍 / 👎.

Co-authored-by: Cursor <cursoragent@cursor.com>

…e-minmax-pruning

ianton-ru · 2026-10-23T15:22:07Z

@codex review

chatgpt-codex-connector · 2026-10-23T15:42:36Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-23T15:46:50Z

@codex review

chatgpt-codex-connector · 2026-10-23T16:07:20Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-26T10:23:53Z

@codex review

ianton-ru · 2026-10-26T13:12:21Z

@codex review

chatgpt-codex-connector · 2026-10-26T13:32:52Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-26T16:00:52Z

@codex review

chatgpt-codex-connector · 2026-10-26T16:21:20Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-26T17:53:23Z

@codex review

chatgpt-codex-connector · 2026-10-26T18:13:47Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-27T09:03:11Z

@codex review

chatgpt-codex-connector · 2026-10-27T09:20:58Z

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 3be5d18bcc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-authored-by: Cursor <cursoragent@cursor.com>

arthurpassos

lgtm

ianton-ru · 2026-10-27T15:50:56Z

@blau-ai

blau-ai · 2026-10-27T15:56:52Z

CI triage for `0a43e58`

Verdict: 6 red checks. None are caused by this PR. Every failure also shows up on unrelated PRs against antalya-26.8. No fixes are needed in this PR.

The checks that cover the code this PR touches are green:

Fast test: 11444 passed
Stateless tests (amd_debug, parallel): 0 failed / 13639 passed
both targeted ASan stateless jobs
Regression release iceberg_1, iceberg_2, parquet, parquet_aws_s3, parquet_minio

Check	Failing tests	Classification
Integration (amd_asan_ubsan, db disk, old analyzer, 5/8)	`test_storage_iceberg_with_spark/test_export_partition_iceberg.py` x21	Pre-existing
Integration … 5/8	`test_cas_gcs_relink_liveness::test_both_queues_drain_under_slow_checkpoints`	Pre-existing / harness
Integration … 6/8	`test_database_backup/test.py` x17	Pre-existing (server crash, S3 multipart)
Integration … 1/8	`test_storage_s3/test_sts.py` x7, `test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py` x2	Pre-existing / harness
Integration … 7/8	`test_datalake_glue_token_forwarding` x3, `test_custom_s3_disk_remote_url_allow_hosts.py` x2	Pre-existing / harness
Regression release oauth	`/oauth/access control`, `/oauth/groups`	Pre-existing
Regression release swarms	`cluster discovery`, `node failure/cpu overload*`, `profile events with overloaded node`	Pre-existing

PR is just the aggregate gate.

Details

1. test_export_partition_iceberg.py (21 tests). This is the only failure in the Iceberg area. It is an access-control error in the test setup, not a pruning problem:

Code: 497. DB::Exception: default: Not enough privileges. To execute this query, it's necessary to have the grant SHOW NAMED COLLECTIONS SECRETS ON s3. (ACCESS_DENIED)
(query: CREATE TABLE IF NOT EXISTS spark_identity_... ENGINE=IcebergS3(s3, ...)

The same 21 tests fail on #2700 (a cert-bundle backport) and #2687 (a brotli bump), neither of which touches Iceberg. They also fail on #2704. The default user lacks a named-collection grant, so this needs a test-config or grant fix on the base branch.

2. test_database_backup (17 tests). The server aborts on a debug assertion in the S3 backup write path:

<Fatal> : Logical error: 'Chosen multipart upload for an empty file. This must not happen'.
BaseDaemon: Signal description: Aborted

The remaining tests in the module then fail with Connection refused. The same 16–17 tests fail on #2704 and #2703. This PR doesn't touch backup or S3 upload code.

3. Startup timeouts: test_sts.py, test_datalake_glue_token_forwarding, test_cas_gcs_relink_liveness, test_custom_s3_disk_remote_url_allow_hosts.py. The helper containers (sts.amazonaws.com, sts, fakegcs, node) never start:

Timed out while waiting for instance `sts.amazonaws.com' ... to start.
bash: line 1: /usr/bin/clickhouse: cannot execute: required file not found

This is a test-harness or image problem, not server behaviour. The same modules fail on #2700, #2704 and #2703.

4. Regression oauth / swarms. These are HTTP status-code assertions (assert http_code == status_code) in oauth, and cluster-discovery / CPU-overload scenarios in swarms. Both suites fail the same way ("1 feature (1 failed)" / "1 module (1 failed)") on #2700 and #2687.

Next steps

Nothing to change in this PR.
Re-running won't help: these failures repeat on every PR against antalya-26.8. The base branch needs fixes for (1), (2) and (3).

_{Evidence: result_pr.json for this PR and for #2700, #2687, #2704, #2703 (praktika S3), plus the regression report.html files.}

DimensionWieldr · 2026-10-27T18:06:18Z

Stateless tests covering this change are passing.

CI looks fine, no fails caused by this PR.

AI audit returned no defects.

LGTM

Going to add some regression tests; not expecting any issues.

DimensionWieldr · 2026-10-27T19:54:02Z

Added regression tests for this in clickhouse-regression/parquet/tests/nullable_minmax_pruning.py and clickhouse-regression/iceberg/tests/iceberg_engine/nullable_minmax_pruning.py which are passing.

ianton-ru and others added 3 commits October 22, 2026 17:17

chatgpt-codex-connector Bot reviewed Oct 23, 2026

View reviewed changes

ianton-ru and others added 2 commits October 23, 2026 17:03

Keep nullable min/max chunks when NOT (key <=> c) is true for NULL

4086c26

Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'antalya-26.8' into feature/antalya-26.8/iceberg-nullabl…

3be5d18

…e-minmax-pruning

ianton-ru changed the title ~~iceberg nullable minmax pruning~~ Iceberg nullable minmax pruning Oct 26, 2026

ianton-ru added antalya port-antalya PRs to be ported to all new Antalya releases antalya-26.8 Session label (releasy session config) labels Oct 26, 2026

ianton-ru marked this pull request as ready for review October 27, 2026 09:27

Do not print 05386 count rows that only fill query_log

0a43e58

Co-authored-by: Cursor <cursoragent@cursor.com>

arthurpassos approved these changes Oct 27, 2026

View reviewed changes

DimensionWieldr added the verified Approved for release label Oct 27, 2026

Conversation

ianton-ru commented Oct 23, 2026

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Motivation

Approach

Tests

Uh oh!

ianton-ru commented Oct 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Oct 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 23, 2026

Uh oh!

ianton-ru commented Oct 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 23, 2026

Uh oh!

ianton-ru commented Oct 26, 2026

Uh oh!

ianton-ru commented Oct 26, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026

Uh oh!

ianton-ru commented Oct 26, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026

Uh oh!

ianton-ru commented Oct 26, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026

Uh oh!

ianton-ru commented Oct 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026

Uh oh!

arthurpassos left a comment

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Oct 27, 2026

Uh oh!

blau-ai commented Oct 27, 2026

CI triage for 0a43e58

Details

Next steps

Uh oh!

DimensionWieldr commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DimensionWieldr commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chatgpt-codex-connector Bot commented Oct 23, 2026 •

edited

Loading

github-actions Bot commented Oct 23, 2026 •

edited

Loading

CI triage for `0a43e58`

DimensionWieldr commented Oct 27, 2026 •

edited

Loading