Skip to content

perf: certify full-text Top-K and reuse write buffer settings - #1245

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/fts-bounded-topk
Oct 28, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/fts-bounded-topk

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Closes #1243.

Global full-text search retains every archive's candidates before applying the caller's Top-K. Bound per-shard candidates when equivalence to the existing final result can be certified, and provide a reproducible comparison against exhaustive collection.

Scalar global-index sorting also reuses Java's write-buffer-size setting instead of the recently introduced Rust-only sort-memory option.

Brief change log

  • Select certified Top-K automatically for FAST-mode global searches with disjoint shard ranges, no raw sources, and no deletion exclusions or enabled deletion vectors.
  • Evaluate the complete native query and filter with K+1 hits. Keep K only when the cutoff score is strict and the selected row IDs are distinct; otherwise retry with the original full candidate limit.
  • Carry the optional candidate limit through scheduled search tasks, preserving exhaustive behavior for older frames.
  • Keep the exhaustive reference reader inside the existing internal-api feature for benchmarks and tests. The production optimization has no table-level enable/disable option.
  • Reuse write-buffer-size for scalar sorting, with Java's 256 MiB default. Preserve builder-option overrides, spill/merge behavior, oversized-key errors, and FM's existing streaming path.
  • Add a production-path release benchmark with exact ID/score assertions, alternating mode order, distinct-score and tied-score corpora, protocol and raw sample results.

Validation

  • Core library with Vortex: 4,647 passed, 6 ignored.
  • Related data-evolution search/sidecar integration target: 2 passed.
  • Workspace Clippy, all targets with Vortex, denying warnings: passed.
  • Release benchmark through the internal reader: 10,000 rows, four shards, K=10, three measured iterations per mode; exact IDs and scores matched for both corpora.
  • Formatting, diff, dependency-report and generated release-license checks: passed.
  • Independent review found no actionable issues.
  • Regression assertions distinguish three exhaustive candidates from two bounded candidates after filtering. Sort tests cover invalid budgets, oversized keys without snapshot publication, builder overrides, and small-budget spill behavior for BTree v1/v2, Bitmap, Multivalue and composite keys.

A local all-targets attempt exceeded available disk during linking. The complete core library and related integration target passed; the remaining integration/platform matrix is left to CI.

API and Format

The existing Java option write-buffer-size replaces sorted-index.sort-memory for scalar sorting; the default changes from 64 MiB to 256 MiB. The optional search-task candidate limit has a default for older frames. Index archives and score semantics are unchanged.

Documentation

Document the automatic optimization, internal benchmark command and Java-compatible sort-memory setting. Preserve the historical performance sample with its original source revision.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi force-pushed the codex/fts-bounded-topk branch from 87dc89c to 2d31b97 Compare October 28, 2026 09:17
@JingsongLi JingsongLi changed the title perf: certify bounded full-text Top-K candidates perf: certify full-text Top-K and reuse write buffer settings Oct 28, 2026
@JingsongLi
JingsongLi merged commit 2300ad5 into apache:main Oct 28, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add certified full-text Top-K candidates and a reproducible benchmark

2 participants