Skip to content

fix(table): align native vector index configuration semantics - #1270

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/fix-pk-vector-persisted-metric
Nov 1, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/fix-pk-vector-persisted-metric

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Fix native index compatibility failures reported by Paimon Native CI. Existing PK vector payloads can use a different metric from the current schema, and portable build requests can contain opaque implementation parameters.

Brief change log

  • Use the first nonempty captured ANN payload's metric for indexed scoring, exact fallback and refinement. Validate every searched payload header against that metric, and retain table metrics for raw-only plans.
  • Reuse the vector metadata/header reader for bucket-local payloads without serializing header reads across all segments.
  • Pass recognized generic build parameters to strict native validation while retaining opaque options in captured plans. Known invalid or implementation-incompatible parameters still fail.
  • Cover empty segments, mixed metrics, captured sources, and single/batch/refined searches over indexed and raw data.

Tests

  • 59 focused Rust unit tests across PK vector reads, vector search/options and generic index builds.
  • 2 Java-fixture integration tests.
  • cargo +1.98.0 fmt --all -- --check.
  • cargo +1.98.0 clippy --locked -p paimon --lib --tests -- -D warnings.

API and Format

No API signature or storage-format changes. Generic builds ignore opaque implementation parameters consistently with Java/PyPaimon; the low-level strict options API is unchanged.

Documentation

No new configuration options.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 1ce455e into apache:main Nov 1, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants