Skip to content

[lake/paimon] Avoid repeated manifest scans for historical tiering - #4731

Open
beryllw wants to merge 4 commits into
apache:mainfrom
beryllw:tiering/historical-snapshot-sequence
Open

beryllw wants to merge 4 commits into
apache:mainfrom
beryllw:tiering/historical-snapshot-sequence

Conversation

@beryllw

@beryllw beryllw commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Fixes #4656.

Historical primary-key splits write to multiple Paimon partitions and buckets, repeatedly scanning manifests during writer initialization.

Brief change log

Default write.sequence-number-init-mode to snapshot when creating a Paimon primary-key table with historical partition access enabled and automatic compaction disabled. Persist the option so writers and external compactors inherit it, while preserving explicit settings. Log tables and existing Paimon tables are unchanged.

Tests

  • Cover creation-time defaults, explicit overrides, sequence-number progression, external compaction, and subsequent updates/deletes.
  • Use real catalog/writer paths and capture Paimon scan-completion logs: a historical writer targeting two partitions performs 0 plans with the default optimization, versus 2 with explicit scan or automatic compaction enabled.
  • Paimon module verification passed: 217 unit tests and 59 integration tests, plus Spotless and Checkstyle.

API and Format

No API or storage-format changes.

Documentation

Document the creation-time default, Paimon 2.0 requirement, and scan opt-out under historical partition access.

@beryllw
beryllw requested a review from luoyuxia October 19, 2026 08:14
@beryllw
beryllw force-pushed the tiering/historical-snapshot-sequence branch 2 times, most recently from 1a14081 to 50ba644 Compare October 23, 2026 04:23
@beryllw
beryllw force-pushed the tiering/historical-snapshot-sequence branch from 50ba644 to 1b16131 Compare October 28, 2026 03:18
@beryllw

beryllw commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@luoyuxia cc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Historical Paimon tiering repeatedly scans manifests

1 participant