Skip to content

[rust] Decode only selected columns in remote Arrow logs - #4922

Open
spoon-lz wants to merge 1 commit into
apache:mainfrom
spoon-lz:codex/remote-log-decode-projection-20261028
Open

spoon-lz wants to merge 1 commit into
apache:mainfrom
spoon-lz:codex/remote-log-decode-projection-20261028

Conversation

@spoon-lz

Copy link
Copy Markdown
Contributor

Purpose

Closes #4921.

Avoid decoding unrequested columns in remote Arrow logs for both row and batch scans, including historical-schema reads with projection.

Brief change log

  • Pass source-column selections to the Arrow IPC decoder.
  • Remap historical-schema alignment while preserving missing-column nulls, output order, duplicates, and row counts.
  • Keep full-column and server-projected decoding paths unchanged.

Remote download size is unchanged. No nested leaf pruning is introduced.

Tests

  • Cover NONE, LZ4, and ZSTD; nested columns; reordered and duplicate selections; empty batches; and missing historical columns.
  • Exercise public row and batch poll paths with file-backed historical logs in both schema modes.
  • Verify that unselected compressed columns are not decompressed.
  • Rust workspace tests, 914 core unit tests, Clippy, and formatting checks passed locally.

Live-cluster integration tests were not run.

API and Format

No public API, configuration, RPC, or storage-format changes.

Documentation

Updated the affected code comment. No new user-facing configuration.

Generative AI disclosure

Generative AI tools used: OpenAI Codex (GPT-6).
Generated-by: OpenAI Codex (GPT-6).

Push column selection into Arrow IPC decoding and remap historical schema alignment to the selected output. Preserve row and batch scan semantics, missing columns, and full-column decoding.

Generated-by: OpenAI Codex (GPT-6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Decode only selected columns in remote Arrow logs

1 participant