Skip to content

feat: add table-aware position delete updates - #919

Draft
wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:feature/v3-dv-write-pipeline
Draft

wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:feature/v3-dv-write-pipeline

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026 •

Copy link
Copy Markdown

Summary

Deliver table-aware end-to-end position deletion: write v3 Puffin DVs, preserve the v2 Parquet path, merge prior deletes, supersede old entries, and commit through RowDelta.

  • resolve each target's live partition/spec and keep minimal scan consistency checks with existing sequence-number fail-fast behavior

  • retain conflict validation, unknown-commit output preservation, and retry-safe cleanup; define missing-file deletion as idempotent at the shared FileIO seam

  • verify actual surviving Parquet rows after two commits and a fresh table load, including partition evolution

  • exercise a real concurrent data-file removal during commit/retry

  • resolve requested files lazily, reuse the existing delete-write result, and centralize commit-result handling with a small shared cleanup helper

  • consolidate duplicate routing/order tests and reuse the existing retry test harness without weakening conflict or unknown-commit coverage

DV-PR-1 in #916. Supersedes the standalone applicability work in #915. Based directly on main at 3c5715c; no dependency on #913, #914, or #915. Full Puffin footer hardening and standalone fixtures are deliberately excluded.

Test plan

GCC 14 focused checks on the four-feature integration tree based on 3c5715c; the final changed branch sources were verified to match the tested tree.

  • data_test --gtest_filter='*PositionDeleteUpdateTest*:*PositionDeleteV3Test*': 16 passed

Includes actual Parquet row survival after two commits/reload, partition evolution, a persisted concurrent removal, cleanup/retry, unknown commit ownership, and v2 physical delete ordering.

  • Repository pre-commit checks and git diff --check: passed

@wirybeaver
wirybeaver force-pushed the feature/v3-dv-write-pipeline branch from 658315c to f256e78 Compare October 1, 2026 21:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant