[C++] IPC metadata V4 union field-subset reads can return incorrect values #55369

Describe the bug, including details regarding any error messages, version, and platform.

When IpcReadOptions::included_fields is set, RecordBatchFileReader reads only the byte ranges of a record batch body needed by the selected fields. ReadFieldsSubset in cpp/src/arrow/ipc/message.cc determines those ranges by running LoadFieldsSubset (cpp/src/arrow/ipc/reader.cc) against an IoRecordedRandomAccessFile, which assumes the default V5 metadata buffer layout.

The buffer layouts differ for union arrays between V4 and V5 metadata: in V4 metadata a union array includes a validity bitmap buffer slot, whereas in V5 it does not. For an IPC message written with V4 metadata that contains a union column (including Feather V2 files written with V4 metadata when reading a column subset), the byte ranges recorded by ReadFieldsSubset correspond to the wrong buffer offsets. When the batch is subsequently loaded using the message's V4 metadata version, it reads from unfilled regions of the body buffer and returns corrupted values for the selected union column.

Expected behavior

Reading a subset of fields via IpcReadOptions::included_fields from a V4 metadata IPC stream/file containing sparse or dense union columns should return the same array values as reading the full record batch and selecting those columns (for example, by falling back to reading the full message body in ReadFieldsSubset when metadata_version() < MetadataVersion::V5 while keeping selective reads for V5).

Component(s)

C++