Skip to content

fix slow filters on multi-valued properties in joined tables - #663

Open
cportele wants to merge 1 commit into
api-validation-fixes-part2from
filter-array-semijoins
Open

cportele wants to merge 1 commit into
api-validation-fixes-part2from
filter-array-semijoins

Conversation

@cportele

Copy link
Copy Markdown
Contributor

Filters on values in a joined table (e.g. a value array with A_CONTAINS or A_OVERLAPS, or a negation of such a filter) took seconds to minutes on large tables when the value is frequent, because the planner had to materialize every matching row of the whole table:

  • A_CONTAINS with a single value and A_OVERLAPS no longer add a GROUP BY without HAVING to the IN semi-join; it does not change the result, but it kept PostgreSQL from turning the subquery into a semi-join that probes the joined table per candidate
  • A_CONTAINS with several values is the conjunction of one semi-join per value instead of a single subquery with GROUP BY ... HAVING count(distinct ...) = n; this also matches a list with duplicate values
  • the negation of a single semi-join is NOT EXISTS over the unchanged subquery instead of NOT (... IN (...)), which the planner can only evaluate per candidate or hash; it is the same predicate, as the key of the main table is never null, and the planner turns it into an anti-join

The derived table of NOT EXISTS has an alias without AS, which Oracle does not accept there.

@cportele cportele self-assigned this Oct 19, 2026
@cportele
cportele added this pull request to stack #658 October 19, 2026 17:45
@cportele
cportele requested a review from azahnen as a code owner October 19, 2026 17:45
Filters on values in a joined table (e.g. a value array with A_CONTAINS
or A_OVERLAPS, or a negation of such a filter) took seconds to minutes on
large tables when the value is frequent, because the planner had to
materialize every matching row of the whole table:

- A_CONTAINS with a single value and A_OVERLAPS no longer add a GROUP BY
  without HAVING to the IN semi-join; it does not change the result, but
  it kept PostgreSQL from turning the subquery into a semi-join that
  probes the joined table per candidate
- A_CONTAINS with several values is the conjunction of one semi-join per
  value instead of a single subquery with GROUP BY ... HAVING
  count(distinct ...) = n; this also matches a list with duplicate values
- the negation of a single semi-join is NOT EXISTS over the unchanged
  subquery instead of NOT (... IN (...)), which the planner can only
  evaluate per candidate or hash; it is the same predicate, as the key of
  the main table is never null, and the planner turns it into an
  anti-join

The derived table of NOT EXISTS has an alias without AS, which Oracle
does not accept there.
@cportele
cportele force-pushed the filter-array-semijoins branch from 7eb08a0 to c6bd2f6 Compare October 19, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant