Goal
Deliver end-to-end Iceberg v3 deletion vectors through a small set of feature PRs: table-aware deletion, sparse position storage, delete inspection, and compaction. Follow-up to #637 and the core DV support in #777.
Summary
The IDs below replace the previous eight-ticket numbering. Tests travel with the feature they verify; there is no fixture-only or footer-hardening prerequisite.
| ID |
Task |
PR |
Dependencies |
| DV-PR-1 |
End-to-end DV delete, commit, and read |
#919 |
— |
| DV-PR-2 |
Sparse bitmap and full non-negative int64 positions |
#917 |
— |
| DV-PR-3 |
position_deletes metadata inspection |
#920 |
— |
| DV-PR-4 |
DV-aware compaction planning and execution |
#921 |
DV-PR-1 |
Roadmap
DV-PR-1 — End-to-end table-aware deletion — #919
Accept position deletes against live data files, resolve their partition/spec, merge existing deletes, write v3 Puffin DVs, and atomically commit replacement delete entries. Preserve v2 Parquet deletion behavior. Keep narrow partition/spec consistency checks and existing sequence-number error semantics; do not silently skip inconsistent DVs.
Acceptance uses real Parquet data: delete and commit, reload the table, check exact surviving rows, delete again, and verify earlier deletes remain applied. Cover partitioned/unpartitioned tables, partition evolution, an actual concurrent data-file removal, commit-state-unknown output safety, and failure cleanup. Missing-file deletion is explicitly idempotent for safe cleanup.
DV-PR-2 — Sparse bitmap storage — #917
Represent positions through INT64_MAX with sparse ordered high-bit buckets rather than allocating intervening buckets. Preserve the wire format, emit populated buckets in order, and retain framing-size prechecks. Keep the production interface small: no test-only configurable writer factory. Half-open range calls still add INT64_MAX separately.
This is independent of the table-aware update and does not depend on cross-language Puffin fixture PRs.
DV-PR-3 — Position-delete inspection — #920
Expose logical deletion rows from both Parquet position deletes and Puffin DVs, including optional stored row values and partition metadata. Support mixed upgraded tables, partition evolution, multi-blob Puffin, empty results, and top-level projections.
Use the current metadata-table Arrow stream API with bounded output batches. Detect the physical Parquet row field structurally rather than parsing error strings. Inspection still materializes results before returning the stream; nested projection such as row.id is outside this PR. No full Puffin footer cross-validation is required.
DV-PR-4 — DV-aware compaction — #921
Combine the planner and executor into one usable maintenance feature: choose and group files within partition/spec boundaries, apply their deletes, rewrite surviving rows, and atomically replace data files and obsolete delete entries.
Keep deterministic grouping and simple double-precision size/delete-ratio heuristics; remove the redundant maximum-size-ratio knob and exact binary-rational threshold machinery. Preserve snapshot-bound stale-plan rejection, row lineage, shared-Puffin safety, and output cleanup. DV-PR-1 supplies shared cleanup semantics and the table-aware update used by end-to-end tests.
Scope consolidation
Prioritize DV-PR-1. Bitmap completeness, inspection, and compaction must not block the core delete/commit/read path.
Goal
Deliver end-to-end Iceberg v3 deletion vectors through a small set of feature PRs: table-aware deletion, sparse position storage, delete inspection, and compaction. Follow-up to #637 and the core DV support in #777.
Summary
The IDs below replace the previous eight-ticket numbering. Tests travel with the feature they verify; there is no fixture-only or footer-hardening prerequisite.
position_deletesmetadata inspectionRoadmap
DV-PR-1 — End-to-end table-aware deletion — #919
Accept position deletes against live data files, resolve their partition/spec, merge existing deletes, write v3 Puffin DVs, and atomically commit replacement delete entries. Preserve v2 Parquet deletion behavior. Keep narrow partition/spec consistency checks and existing sequence-number error semantics; do not silently skip inconsistent DVs.
Acceptance uses real Parquet data: delete and commit, reload the table, check exact surviving rows, delete again, and verify earlier deletes remain applied. Cover partitioned/unpartitioned tables, partition evolution, an actual concurrent data-file removal, commit-state-unknown output safety, and failure cleanup. Missing-file deletion is explicitly idempotent for safe cleanup.
DV-PR-2 — Sparse bitmap storage — #917
Represent positions through
INT64_MAXwith sparse ordered high-bit buckets rather than allocating intervening buckets. Preserve the wire format, emit populated buckets in order, and retain framing-size prechecks. Keep the production interface small: no test-only configurable writer factory. Half-open range calls still addINT64_MAXseparately.This is independent of the table-aware update and does not depend on cross-language Puffin fixture PRs.
DV-PR-3 — Position-delete inspection — #920
Expose logical deletion rows from both Parquet position deletes and Puffin DVs, including optional stored row values and partition metadata. Support mixed upgraded tables, partition evolution, multi-blob Puffin, empty results, and top-level projections.
Use the current metadata-table Arrow stream API with bounded output batches. Detect the physical Parquet
rowfield structurally rather than parsing error strings. Inspection still materializes results before returning the stream; nested projection such asrow.idis outside this PR. No full Puffin footer cross-validation is required.DV-PR-4 — DV-aware compaction — #921
Combine the planner and executor into one usable maintenance feature: choose and group files within partition/spec boundaries, apply their deletes, rewrite surviving rows, and atomically replace data files and obsolete delete entries.
Keep deterministic grouping and simple double-precision size/delete-ratio heuristics; remove the redundant maximum-size-ratio knob and exact binary-rational threshold machinery. Preserve snapshot-bound stale-plan rejection, row lineage, shared-Puffin safety, and output cleanup. DV-PR-1 supplies shared cleanup semantics and the table-aware update used by end-to-end tests.
Scope consolidation
Prioritize DV-PR-1. Bitmap completeness, inspection, and compaction must not block the core delete/commit/read path.