Dependency upgrades and minor fixes, fixes #8966

[hansva]

Dependency upgrades and lib/core cleanup

Upgrades third-party dependencies to their latest patch/minor releases, and removes duplicate and conflicting jars from the assemblies.

Upgrades

• Patch/minor upgrades across lib/pom.xml, lib-jdbc and the plugin poms, among them the AWS SDK (2.20 → 2.56, now one version through the AWS BOM), Netty, OpenTelemetry, Jetty, Jersey, SWT/RAP, the JDBC drivers, Parquet, Tika 3.3.3 and OpenLineage 1.54.
• Driver download defaults in the database meta classes, the bundled-library versions in LICENSE, and the Snowflake docs page now match the bundled drivers.
• Spark, and the libraries Beam/Spark depend on through lib/core (Jackson, protobuf/gRPC, Hadoop, Kafka), are unchanged. ZooKeeper is the exception: it moves to 3.9.6, replacing the end-of-life 3.6.3 that Beam's Spark runner brought.

One version per library in lib/core

lib/core is a single classloader, so two versions of a jar there means whichever is listed first wins.

• lib/pom.xml now manages the libraries that used to land in lib/core in several versions: google-auth, api-common, proto-google-common-protos, animal-sniffer, failureaccess, httpcore5, jaxb-api, jcl-over-slf4j, snappy-java, Parquet, Log4j (through log4j-bom) and more.
• Plugins no longer ship jars that lib/core already has. They declare them provided (JMS, Kafka, Lakehouse, OpenLineage, LanguageModelChat, Cassandra, Stanford NLP and others), or, for Spark, exclude them in the assembly. Spark keeps its own Spark 4 / Scala 2.13 stack.
• Native Spark uses Log4j and lz4 from lib/core. Spark's old org.lz4:lz4-java is replaced by the maintained at.yawk.lz4 fork.
• Test jars no longer reach lib/core: JUnit 4.10 through json-simple, reload4j through Hadoop. Hamcrest stays: Beam's PipelineOptionsFactory needs it at startup.
• Library versions are aligned across plugins: jgit, antlr-runtime, libphonenumber, commons-pool2, jts-core.

Fixes found by the integration tests

• Beam on Spark: OpenLineage 1.54 brought jackson-module-blackbird 2.18. The Beam fat jar bundles it, and it breaks on Spark 3.5 executors (jackson-databind 2.15), which shows up as a serialVersionUID mismatch. It is now pinned to 2.15.3.
• Gremlin: TinkerPop stays on the 3.7 line (3.7.7), because there is no stable JanusGraph release for 3.8 yet.
• Tika: golden-apache-tika.csv is updated in mdi and transforms. Tika 3.3.3 extracts three Office documents slightly differently: it keeps a soft hyphen, drops a trailing newline, and puts a space between hyperlinks.
• GCP / BigQuery: protobuf-javalite is excluded from grpc-protobuf-lite in hop-core. It clashed with protobuf-java in lib/core and caused NoSuchMethodError: DurationProto.getDescriptor().
• Mongo ITs on arm64: the no-AVX mongod wrapper now only uses qemu on x86_64 hosts.

New: GCS upload buffer size for Dataflow

Beam stages the fat jar in 1 MB chunks by default, one request per chunk, which caps staging at about 1 MB/s from a laptop. The Dataflow run configuration has a new GCS upload buffer size (MB) field. When it's empty, 128 MB is used. It is documented in the Dataflow run configuration page.

Testing

• Unit tests: core, engine, Beam, Spark and all touched plugins.
• Integration tests that pass: spark, spark-native, mongo, both Tika tests, beam_directrunner 0006, samples 0001, and transforms 0102/0109/0111/0112.
• Not yet verified: the protobuf-javalite exclusion and the GCS upload buffer field. Both still need a GCP integration test run. The Google Sheets tests were skipped because there was no service-account key.

🤖 Generated with Claude Code

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.
• Mention the appropriate issue in your description (for example: addresses #123), if applicable.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

• I hereby declare this contribution to be licensed under the Apache License Version 2.0, January 2004
• In any other case, please file an Apache Individual Contributor License Agreement.