Skip to content

feat: add position_deletes metadata table - #920

Draft
wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:feature/position-deletes-metadata-table
Draft

wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:feature/position-deletes-metadata-table

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026 •

Copy link
Copy Markdown

Summary

Add general position_deletes metadata inspection for Parquet delete files and Puffin DVs, independent of the high-level delete-update API.

  • preserve stored row payloads, subset-row nulls, and partition evolution with collision-free metadata field IDs

  • support mixed upgraded tables, multi-blob DVs, empty scans, and complete top-level projections

  • adapt to the current metadata-table ArrowArrayStream API, emitting bounded batches

  • detect optional physical Parquet row fields through a reader schema-presence query, not error-message matching

  • construct and validate projection state once per scan, reusing it across batches; open each position-delete file once while still distinguishing absent row columns from invalid null rows

  • fix decimal partition appending/decoding and honor precision-dependent Avro fixed widths in both encoder paths; preserve logical date values represented by primitive storage literals

DV-PR-3 in #916. Based directly on main at 3c5715c; no dependency on #913, #914, or #915. Reuse the existing range-based DV reader without full footer cross-validation.

Test plan

GCC 14 focused checks on the four-feature integration tree based on 3c5715c; the final changed branch sources were verified to match the tested tree.

  • position_deletes_table_test: 19 passed
  • manifest_test --gtest_filter='*ManifestWriter*:*ManifestReader*': 72 passed, 7 version-specific skips
  • avro_test --gtest_filter='*Decimal*': 6 passed
  • data_test --gtest_filter='ArrowRowBuilderTest.*': 8 passed

Coverage includes decimal/date partition write/read, schema-sized signed decimal encoding, reordered projection across multiple batches, optional/subset stored rows, and empty/stale-layout scans.

  • Repository pre-commit checks and git diff --check: passed

Deliberate limits

Results are materialized before returning the stream; bounded batches are not a claim of lazy or memory-bounded input scanning. Nested projection such as row.id remains outside this feature and is rejected explicitly.

Recreate the inspector after source schema/format/partition-spec changes; scans reject a stale layout rather than mixing schemas.

@wirybeaver
wirybeaver force-pushed the feature/position-deletes-metadata-table branch 3 times, most recently from 74edbb6 to 7965dcd Compare September 5, 2026 23:34
@wirybeaver
wirybeaver force-pushed the feature/position-deletes-metadata-table branch from 7965dcd to 0a41688 Compare October 1, 2026 19:57
Reuse one reader and projection context per scan. Append and decode decimal partitions as Decimal128, and encode Avro fixed decimals using their schema width rather than all 16 Arrow bytes. Preserve logical date storage values and cover the complete partition write/read path.
@wirybeaver
wirybeaver force-pushed the feature/position-deletes-metadata-table branch from 0a41688 to 2e7240e Compare October 1, 2026 21:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant