Skip to content

feat: add deletion vector compaction planning and execution - #921

Draft
wirybeaver wants to merge 2 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-executor
Draft

wirybeaver wants to merge 2 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-executor

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026 •

Copy link
Copy Markdown

Note

depends on #918

Summary

Deliver DV-aware compaction as one feature, combining the former standalone planner (#918) with its executor.

  • form deterministic, snapshot-bound groups within partition/spec boundaries using file size and file-scoped delete pressure

  • use ordinary double-precision heuristic thresholds; remove the redundant maximum-size ratio and exact binary-rational comparison machinery

  • read real surviving rows through the existing scan reader, write replacement files, and atomically commit them with obsolete delete-entry removal

  • preserve row lineage, reject stale/duplicate plans, retain shared Puffin objects, and safely handle output cleanup and unknown commit states

  • treat empty planner results as no-ops, centralize execution-result handling, and reuse shared cleanup and native stream ownership

  • consolidate before/after-creation fault injection and cover successful multi-file compaction together with a fully deleted group

DV-PR-4 in #916. Depends only on DV-PR-1 / #919 for shared cleanup behavior and end-to-end test setup. The complete compaction feature is one commit above #919. No prerequisite commits from #913, #914, #915, or the separate #918 remain.

Test plan

GCC 14 focused checks on the four-feature integration tree based on 3c5715c; the final changed branch sources were verified to match the tested tree.

  • data_test --gtest_filter='*Compaction*': 14 passed

Coverage includes empty-plan no-op, successful multi-file/two-group compaction with a fully deleted group, exact surviving rows/lineage, shared Puffin retention, stale/duplicate plans, atomic commit failure, cleanup retry, and before/after-creation failures. Planner logic was unchanged in this refactor.

  • Repository pre-commit checks and git diff --check: passed

@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch 5 times, most recently from 55c6b56 to b3834d9 Compare September 6, 2026 00:19
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch from b3834d9 to a7d3361 Compare October 1, 2026 19:57
@wirybeaver wirybeaver changed the title feat: add deletion vector compaction execution feat: add deletion vector compaction planning and execution Oct 1, 2026
Treat empty plans as no-ops and centralize owned-output cleanup without weakening unknown-commit or stale-plan safety. Consolidate fault injection tests and cover successful multi-file rewriting with a fully deleted group.
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch from a7d3361 to 877ff3e Compare October 1, 2026 21:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant