Repository navigation
Conversation
- Bump iceberg.version 1.9.2 -> 1.11.0 and parquet.version 1.15.2 -> 1.17.1 (iceberg-parquet 1.11 requires parquet's variant logical type). - Re-sync vendored Iceberg sources (BaseMetastoreTableOperations, HTTPClient, BaseParquetReaders, MessageTypeToType, ParquetConversions) onto 1.11.0 and re-apply ice patches. - Replace removed GenericParquetWriter::buildWriter with ::create. - Build ParquetMetadataConverter from ParquetReadOptions: parquet 1.17 rejects the -1 max message size used by the no-arg constructor. - Pin commons-io/commons-lang3 for dependency convergence.
Iceberg's REST client (as of 1.11/1.12) doesn't implement table encryption (only HiveCatalog does), so add it on the client side: - New ice-encryption module: EncryptingRESTCatalog (RESTCatalog subclass, also usable from Spark via catalog-impl) whose table operations build a StandardEncryptionManager from encryption.key-id + table encryption keys + the KMS configured via encryption.kms-type / encryption.kms-impl, wrap io() in EncryptingFileIO, and send newly generated keys to the catalog as add-encryption-key updates on commit. The catalog server never needs KMS access. - ice CLI uses EncryptingRESTCatalog (no-op for unencrypted tables); KMS settings go to icebergProperties in .ice.yaml. - ice insert encrypts data files (Parquet modular encryption) and records their key metadata; --no-copy/--s3-copy-object are rejected for encrypted tables. - ice-rest-catalog maintenance skips encrypted tables. - New ice-test-support module with an insecure StaticKeyManagementClient for tests (not bundled into ice).
…egative test REST catalog + Spark 4.1 (iceberg-spark-runtime 1.11.0, EncryptingRESTCatalog via catalog-impl) + ClickHouse sharing file:///warehouse: - Spark creates a v3 table with encryption.key-id through the catalog. - ice insert without a KMS is rejected; with StaticKeyManagementClient it writes encrypted files (PARE parquet footers, AGS1 manifests/manifest lists, encryption-keys in metadata.json). - ClickHouse fails to read the table (encrypted manifest list). - Spark appends a row and reads back all rows written by ice and Spark. The Spark image is built from src/test/docker/spark-iceberg. The IT is excluded from default failsafe runs.
- security.yaml: add -am so ice-encryption/ice-test-support are built (fixes "Could not find artifact ice-encryption" on clean runners). - verify.yaml: add DockerEncryptionIT to the Docker IT list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.