Repository navigation
[client] Prune Arrow buffers before client-side decompression. - #4883
loserwang1024 wants to merge 1 commit into
Conversation
a5058ea to
2541827
Compare
|
@leonardBang @platinumhamburg @swuferhong , WDYT? |
platinumhamburg
left a comment
platinumhamburg
left a comment
There was a problem hiding this comment.
LGTM, just a couple of minor suggestions.
|
|
||
| /** | ||
| * Deserializes the Arrow batch and loads only the selected fields; buffers of unselected fields | ||
| * are neither copied nor decompressed. Buffers of a previously loaded batch are replaced. |
There was a problem hiding this comment.
The full IPC body is still deserialized and copied before pruning. Please describe this as skipping vector loading and decompression for unselected columns, rather than saying their buffers are not copied.
| if (target == null || isProjectionPushDowned() || schemaId == target.schemaId) { | ||
| return null; | ||
| if (logFormat == LogFormat.ARROW) { | ||
| return getArrowReadPlan(schemaId).createOutputRow(); |
There was a problem hiding this comment.
Arrow iteration already obtains its output mapping directly from ArrowReadPlan.createOutputRow(), while the remaining callers of this method handle INDEXED and COMPACTED records. Consider restricting this method to those formats and keeping Arrow output mapping on the plan. This would remove a redundant entry point and avoid creating Arrow read resources when querying an output mapping.
Select Arrow field nodes and buffers before vector loading so row scans only decompress requested columns. Cache per-schema projection plans and preserve schema evolution mappings, output order and batch checksums.
2541827 to
c38df4d
Compare
Select Arrow field nodes and buffers before vector loading so row scans only decompress requested columns. Cache per-schema projection plans and preserve schema evolution mappings, output order and batch checksums.
Cover compression codecs, nested fields, missing columns, buffer ownership and failed loads.
AI-Model: gpt-6
AI-Contributed/Feature: 257/257
AI-Contributed/UT: 447/447
Purpose
Linked issue: close #4882
Brief change log
Tests
API and Format
Documentation