Skip to content

[To rc/2.5.0] Add multi-platform native releases and installed artifact tests - #1012

Open
ColinLeeo wants to merge 5 commits into
apache:rc/2.5.0from
ColinLeeo:develop
Open

ColinLeeo wants to merge 5 commits into
apache:rc/2.5.0from
ColinLeeo:develop

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Native SDKs, platform packages, and Python wheels need a single release workflow that verifies the files users actually download and install. This PR adds that workflow for 2.5.0 and replaces the separate Python wheel workflow.

Changes

  • Set the C++ and Python release versions to 2.5.0, with an explicit option to preserve those versions during Maven builds.
  • Export an installed CMake package and pkg-config metadata so consumers and C++ regression tests can use the installed SDK without building the library or CLI from source.
  • Use BUNDLED third-party dependencies for every native release package and as the CMake/Maven source-build default. Homebrew Formulae no longer require separate codec libraries. Validate Mach-O load commands in built bottles, staged SDKs and downloaded SDKs, rejecting external third-party dylibs even when present on the runner.
  • Build native SDKs and packages for Linux, macOS, and Windows on x86_64 and ARM64. Produce DEB runtime/development/tools packages, RPM runtime/development/tools packages, Windows ZIP packages, and Homebrew bottles with a merged Formula.
  • Build six CPython ABIs (3.9–3.14) on the five Python platforms: Linux and macOS on both architectures, plus Windows x86_64. Build the four Unix compatibility SDKs once per platform and reuse the native Windows SDK for its wheels.
  • Download and install the outputs in separate jobs. Run the existing C++ API tests through find_package, CLI operations against the four existing python/tests/resources TsFile files, Go's own vet/tests and supported race checks, and the full Python pytest suite against installed wheels. Also relocate and test every downloadable SDK archive.
  • Assemble all outputs only after the installation tests pass. Include source/version metadata, manifest.json, and SHA256SUMS, and require persistent JUnit reports for every Python wheel test.
  • Export share/tsfile/tsfile-build-config.json from the SDK and record the resolved CRT, compiler, configuration and runtime filename. Python reads it automatically, without a separate runtime environment variable.
  • Ship one native library per wheel, exclude SDK headers/import libraries/Cython build files, strip Unix wheel binaries before platform repair, retain license texts, and validate the content of every built and downloaded wheel. Preserve the build helper and Cython inputs in the sdist.
  • Add bounded retries for transient CMake/Maven dependency downloads.

The workflow is manually dispatched and uploads build artifacts; package-registry publication is a separate step. Python retains the existing five-platform scope, without Windows ARM64 wheels. CLI golden files and the existing CLI E2E fixtures are unchanged.

Commit structure

  1. b2ca340e0 — Bump C++ and Python release versions to 2.5.0.
  2. 3f45b0822 — Fix portable C++ builds and installed SDK consumption.
  3. 0ed880344 — Add the complete multi-platform native release workflow.
  4. d3a40dbb9 — Bundle third-party dependencies across all native packages and reject external macOS dylib references.

Validation

The current head, d3a40dbb9ef88a991fc683d042a5368da44173d5, is undergoing the full native release workflow. Results are pending. Local macOS ARM64 validation built the default bundled SDK and tested it after archive relocation: 222 C++ tests passed with 3 existing disabled cases; CLI checks on all four fixtures and Go vet/tests/race passed. Packaging helpers passed 68 tests with 5 existing opt-in skips. The new Mach-O check rejects both previous Homebrew-derived SDKs and accepts the newly built bundled SDK. Workflow contract, actionlint (with the existing newer runner labels allowed), formatting and dependency-policy checks passed.

The results below describe the previous commit and its artifacts; the current build has not yet replaced them.

The previous commit, 0ed880344590ba575101a1fc64d971f8bb642e8f, passed the complete native release workflow: 60/60 jobs successful. An AlmaLinux mirror metadata outage interrupted the first RPM installation attempt; rerunning that task and dependent assembly succeeded without changing code or test coverage.

Validation Verified result
SDK configuration All 12 downloaded SDK variants contain versioned build metadata, the resolved CRT and the runtime filename; Windows SDKs record MultiThreaded
Native installation and SDK relocation C++, CLI and Go passed in all 24 package/SDK scenarios; Go race passed in the 22 supported scenarios
C++ reports 24 downloaded reports; 5,328 passed and 72 existing disabled cases across the reports; zero failures
Python reports All 30 downloaded reports; 18,019 passed and 41 existing platform/Python-version skips; zero failures
Wheel contents All 30 actual downloaded wheels contain one TsFile runtime, no SDK headers/import libraries/Cython build inputs or ELF debug sections; versions, ABIs, binary architectures and Windows PE dependencies verified
Source build compatibility Ordinary Python CI and MSVC Python CI both passed, including default dynamic-CRT builds; compiler inheritance fixed for isolated ANTLR dependency tests
Local packaging checks 68 tests: 63 passed and 5 existing opt-in skips; workflow contract, actionlint, formatting and Apache RAT passed
Source distribution Built an sdist and then a runtime-only wheel from that sdist; repaired/installed macOS ARM64 CPython 3.12 wheel passed 601 tests with 1 existing platform skip
Final bundle Downloaded all 61 manifest entries and verified SHA256, sizes, source commit, versions, SDK metadata, package contents and binary architectures

Actual wheel sizes in the final bundle (decimal MB):

Platform CPython 3.9–3.14 wheel size
manylinux_2_28 x86_64 2.21–2.23 MB
manylinux_2_28 aarch64 2.07–2.10 MB
macOS 12 x86_64 1.88–1.89 MB
macOS 12 arm64 1.62–1.63 MB
Windows x86_64 2.11–2.18 MB

The manylinux x86_64 CPython 3.9 wheel decreased from 12,374,013 bytes to 2,230,136 bytes, about 82% smaller, with 42 files instead of 841. License/notice texts are retained.

The verified final bundle contains 12 SDK archives, 30 wheels, 6 DEBs, 6 RPMs, 2 Windows ZIPs, 2 Homebrew bottles, 2 bottle metadata files and the merged Formula, plus manifest.json and SHA256SUMS. Its compressed size is 117,890,935 bytes. The CI artifact expires on October 28, 2026.

@ColinLeeo
ColinLeeo force-pushed the develop branch 2 times, most recently from ce7ebe6 to 095b086 Compare October 14, 2026 01:59
Fix MSVC language feature detection, runtime selection, native ARM64 builds, dependency downloads and relocatable library paths. Export headers, CMake and pkg-config metadata, static dependencies and native package components. Record the resolved CRT and runtime filename in build-tree and installed SDK metadata so independent consumers can select matching compiler options. Allow existing API tests to consume an installed SDK and include dependency-download regression tests. Inherit the selected compiler in isolated ANTLR4 dependency test configurations.
Build native SDKs, DEB/RPM packages, Windows ZIPs, Homebrew bottles and Python 3.9-3.14 wheels. Reuse SDKs and automatically read their MSVC runtime configuration. Ship runtime-only wheels with one native library, stripped Unix binaries and license texts; reject build-only files, duplicate libraries and ELF debug sections. Test every installed package, wheel and relocated SDK in fresh jobs using existing C++ and Go tests, CLI operations on Python resources and the full pytest suite. Gate checksummed bundle assembly on all test matrices and retire the separate wheel workflow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant