[EdwardShen125]

Rationale for this change

#566 reports encryption and decryption failures between pyarrow and arrow-go. The direct-key half no longer reproduces on main: reading pyarrow-written files in Go works for uniform AES-GCM, AES-GCM-CTR, AAD prefix and plaintext footer, and pyarrow reads an arrow-go-written file. Checked against pyarrow 26; I suspect #596/#1390/#1412 fixed this but haven't bisected. Nothing tests these paths in either direction today, so it could regress unnoticed.

What still fails is KMS. pyarrow's CryptoFactory writes PKMT1 key material with a wrapped DEK, arrow-go has no parser or unwrapping for it, so footer decryption fails without a custom retriever. That gap stays tracked in #566; per review the skip-only KMS test was left out of this PR and belongs with the PKMT1 follow-up.

What changes are included in this PR?

encryption_interop_pyarrow.py generates the encrypted files and reads a Go-written one; it is embedded in the test binary and run through python3. encryption_interop_pyarrow_test.go holds the tests. They skip when python3 or pyarrow isn't installed, so the default test run is unchanged. plainfooter_direct also asserts the footer magic stays PAR1 (an encrypted footer would end in PARE1) so the case checks what its name says.

CI: ci/scripts/cgo_python_test.sh, which runs in the docker-cgo-python job (the only one with pyarrow installed), now runs go test -run Pyarrow on the package with PARQUET_TEST_REQUIRE_PYARROW=1, which turns the "pyarrow not available" skip into a failure so an image change can't silently disable the tests. Since the install is unpinned, the tests follow the current pyarrow release.

Are these changes tested?

The tests are the change. Against pyarrow 26 locally: the four direct-key configs pass and pyarrow reads the Go-written file back correctly. Without python3 or pyarrow everything skips.

Are there any user-facing changes?

No, tests only.

[zeroshade]

Thanks, this is a useful harness. With pyarrow 26 installed, go test -race -run Pyarrow ./parquet/ passes: Go reads all four pyarrow-written configs, and pyarrow reads the Go-written file. Without pyarrow everything skips cleanly.

• CI: please include the one-line addition to ci/scripts/cgo_python_test.sh in this PR. That job is the only one with pyarrow installed, and without it these tests always skip in CI.
• TestPyarrowKmsWrittenConfigs: it only skips, so I'd leave it out and add it with the PKMT1 follow-up. #566 already tracks the gap.
• Optional: once CI runs these, an env var in that job that turns the "pyarrow not available" skip into a failure would keep an image change from silently disabling them.
• Nit: plainfooter_direct would also pass with an encrypted footer; asserting the footer is plaintext would make the case check what its name says.

[EdwardShen125]

All four done: cgo_python_test.sh runs the suite with PARQUET_TEST_REQUIRE_PYARROW=1 (missing pyarrow fails instead of skips), the skip-only KMS test is dropped for the #566 follow-up, and plainfooter_direct asserts the footer magic stays PAR1. Rebased on current main.