[Mandukhai-Alimaa]

Rationale for this change

• Closes [Format] Define Canonical Extension Type for FILE type #51143

As storage formats standardize file-level references, Arrow needs a canonical in-memory representation to preserve these semantics across IPC streams, the C Data Interface, Arrow Flight, and language bindings. Without a canonical extension type, file fields lose their semantic metadata during serialization.

[DISCUSS] thread in the mailing list

What changes are included in this PR?

Proposal and documentation for the arrow.parquet.file canonical extension type.

Are these changes tested?

N/A

Are there any user-facing changes?

Yes, this is an extension to the arrow format.

• GitHub Issue: [Format] Define Canonical Extension Type for FILE type #51143

[github-actions]

⚠️ GitHub issue #51143 has been automatically assigned in GitHub to PR creator.

[Mandukhai-Alimaa]

CC: @rok

[wgtmac]

I think we need to move forward with #50892, which is blocked by this proposal to some extent because we want interoperability.

WDYT? @alamb @emkornfield @mapleFU @pitrou @zeroshade

[wgtmac]

For this section, should we quote the Parquet spec verbatim with explicit attribution at the start, or keep the current wording and label it as “adapted from the Parquet FILE logical type specification”? Either way, it would help to make the relationship explicit so readers know whether this is a direct quotation or a restatement intended to preserve the same semantics.

[alamb]

unless there is a good reason to restate all the content, I think it should simply point at the Parquet spec

[alamb]

Thank you for working on this @Mandukhai-Alimaa

[Mandukhai-Alimaa]

Thanks for the feedback. I’ve removed the duplicated text and linked to the Parquet FILE spec instead.

[rok]

Thanks for working on this @Mandukhai-Alimaa! (and appologies for my slowness).

No form comments, only one on semantics.

[HuaHuaY]

Suggested change

	point for the relative path, an extension type will not carry no such
	information.
	point for the relative path, an extension type will not carry such information.

I guess here is a typo.

[rok]

@HuaHuaY how do you feel about the relative path handling proposed here? It was my suggestion, but I'm not entirely happy with it.

[HuaHuaY]

@rok Given that a service might receive Arrow RecordBatch data, I think the interpretation of the fields should be left to the service's developers. Arrow itself does not need to understand the content of the data. But if I understand correctly, Parquet itself does not know how to read the content by the URI either. So I’m not sure we need to explicitly state this constraint, and I think it may be already inherent in the Parquet standard.

[rok]

Parquet itself does not know how to read the content by the URI either.

Agreed! Parquet (intentionally) doesn't say much about about relative URIs, so this is implicitly up to the application. I'll propose a new wording to better describe parquet here. Sorry for the back and forth @Mandukhai-Alimaa !