Repository navigation
Conversation
|
|
github-actions
Bot
added
Component: C++
awaiting committer review
… sets using a sorted value set Since apacheGH-46777 is_in predicates with more than 50 values are not simplified against row group statistics at all, because filtering the value set and rebinding the call for every guarantee was prohibitively expensive. Large value sets are now simplified with a binary search instead: - The is_in kernel init keeps a copy of the value set sorted in the order of the comparison kernels, with nulls removed. An already sorted value set is reused as is after a single O(N) check; an unsorted one is sorted lazily and thread-safely on the first simplification, so eager kernel calls never pay for sorting. Value sets containing NaN are not searchable, since statistics cannot bound NaN. - SimplifyWithGuarantee applies all inequality guarantees on a field to a large is_in call at once, narrowing an index range into the sorted value set with lower/upper bound searches. If the range is empty the call simplifies to false (subject to the null matching behavior), otherwise it is left unchanged. This is exact for any number of values, costs O(log N) per guarantee and never rebinds. Applying both bounds together is what allows pruning a row group inside the hull of the value set but holding none of its values, which no per-bound check can do. Value sets up to 50 values keep using the existing filter-and-rebind path.
…d of via sort_indices + take Sorting a copy of the values with std::sort is about 14x faster than the generic sort_indices + take path for a shuffled set of 10M int64 values (0.19s vs 2.6s). Nulls are dropped while copying and a set containing NaN is reported as not searchable before sorting, since NaN would break the strict weak ordering std::sort requires. Non-primitive types (binary, decimal) keep using the generic path.
…in is_in pruning Extract nullable min/max bounds from Parquet statistics for large value sets. Respect GetNullMatchingBehavior() in both simplification paths and when rebinding narrowed sets. Add expression and Parquet regressions for nullable ranges and conflicting deprecated skip_nulls options.
…and prune nullable ranges in every simplification path Inequality guarantees are now restated through the order preserving casts wrapping a filter's argument before they are applied: strict bounds are weakened to inclusive ones and the bound is put through the same casts, since e.g. f64 = 51.00000001 satisfies f64 > 51 but casts to 51f. This applies to plain comparisons and small is_in value sets as well as the large value set path, which previously was the only one handling it. The nullable range guarantee produced by Parquet statistics for row groups with nulls, (x >= min and x <= max) or is_null(x), is now extracted for all simplifications rather than only for large is_in value sets, and an is_valid(x) member of the guarantee rules out nulls for the inequalities on x.
rok
force-pushed
the
GH-55292-is-in-row-group-pruning
branch
from
November 3, 2026 17:12
a3f317c to
79750a7
Compare
…L and INCONCLUSIVE calls - Defer concatenating, checking and sorting the value set to the first GetSortedValueSet() call (std::call_once), so plain is_in kernel calls pay nothing extra at init. - Simplify an is_in call without possible matches to its exact result for every null matching behavior: invert(true_unless_null(arg)) for EMIT_NULL and INCONCLUSIVE, or a null literal for INCONCLUSIVE with a null in the value set, so these prune row groups too. - Factor out Comparison::IsOrderPreservingCast, derive IsSearchableValueSetType from the accessor trait, and drop unreachable NaN, type and comparison checks. - Test chunked value sets and the new null matching results.
…pruning and condense tests - Sort the value set with sort_indices + take and search it with search_sorted instead of hand-written visitors. Decimal and fixed size binary value sets are no longer searchable since search_sorted does not support them. - Make the large is_in simplification tests table-driven, merge the null matching tests, and share a helper which checks that simplified filters evaluate like the originals. - Merge the two Parquet row group pruning tests into one file whose row groups contain nulls.
…and max instead of sorting them Sorting a value set costs 1.5-3x as much as building the is_in lookup table, so for 10M shuffled ids the one-time sort outweighed the row groups it pruned and scans got slower than without pruning. Value sets longer than kIsInMaxSortedValueSet (2^17) are now only bounded by their smallest and largest value (min_max), which is cheap at any size and prunes as well when the values are clustered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Since #46859,
is_inpredicates with more than 50 values are not simplified against Parquet row group statistics at all, so such filters read every row group. In addition,SimplifyWithGuaranteedoes not recognize the statistics guarantee of a row group that may contain nulls,(x >= min and x <= max) or is_null(x), so no filter prunes those row groups. See #55292 for the background. Theis_inproblem was reported and fixed in Polars in pola-rs/polars#29923.What changes are included in this PR?
is_inkernel state keeps the value set and sorts it (sort_indices+take, nulls dropped) the first time it is used for simplification. Plain kernel calls pay nothing extra. Sorting costs 1.5 to 3 times as much as building the kernel's lookup table, so value sets longer than 2^17 are only bounded by their smallest and largest value (min_max), which is cheap at any size and prunes as well when the values are clustered. Value sets containing NaN are not searched, since statistics cannot bound NaN.is_insimplification.SimplifyWithGuaranteeapplies all inequality guarantees on a field to a largeis_incall at once and looks up the admitted range withsearch_sorted. Using both row group bounds together is what lets us skip row groups that lie inside the hull of the value set but hold none of its values (for value sets of up to 2^17 values). If no value can match, the call is replaced by what it then evaluates to:false,invert(true_unless_null(x))forEMIT_NULLandINCONCLUSIVEon nullable data, or a null literal forINCONCLUSIVEwith a null in the value set. Otherwise the call is left unchanged, with no rebinding. Sets of up to 50 values keep using the existing filter-and-rebind path.(cmp(x, a) and cmp(x, b)) or is_null(x)now yield both inequalities, and anis_valid(x)guarantee rules out nulls. Comparisons and smallis_insets now prune row groups with nulls too.is_insimplification: givenf64 > 51,cast(f64, float32) == 51was simplified tofalsealthoughf64 = 51.00000001casts to51.Comparison::StripOrderPreservingCastsno longer strips same-width unsigned to signed casts that may wrap (allow_int_overflow).is_inpath now honors the deprecatedskip_nullsoption (GetNullMatchingBehavior()).Are these changes tested?
Yes. New expression tests:
SimplifyIsInLargeValueSet,SimplifyIsInLargeValueSetNullMatching(which also checks that simplified filters evaluate like the originals on rows with nulls),SimplifyWithNullableRangeGuarantee,SimplifyComparisonThroughNonInjectiveCast,SimplifyIsInDeprecatedSkipNullsandSimplifyLargeIsInThenExecute. A new Parquet dataset test checks which row groups are read for largeis_infilters under every null matching behavior, and for comparisons on row groups with nulls.Scan timings on a 49.3M row file with 47 row groups of 2^20 rows and a sorted
idcolumn (release build, file in page cache, median of 5 scans, each binding a fresh filter). The value sets are shuffled distinct ids clustered in a few row groups:is_invalue setAre there any user-facing changes?
No API changes.
is_infilters with more than 50 values prune Parquet row groups again (except for decimal, fixed size binary and view value sets, and dictionary columns). Range filters and smallis_infilters now also prune row groups that contain nulls. Filters with explicit narrowing floating point casts or wrapping unsigned to signed casts could previously be simplified tofalseand skip matching rows; they no longer are.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by:
The code was drafted with an AI assistant; I reviewed, tested and benchmarked it and iterated on the design.