Repository navigation
Conversation
The ThinLTO link of `clickhouse` and `clickhouse-keeper` dominates the release build: ~82 min of a ~117 min `amd_release` build on a 16-core, 62 GB builder. lld can cache the optimized native object of every module (`--thinlto-cache-dir`), keyed by the module's bitcode hash, its import set and the LTO configuration, and reuse it on the next link when nothing in that key changed. Only the parallel backend phase is cached; the single-threaded thin-link and the final native link still run in full. Measured on the same builder class with the production `-g` configuration (sha e3f85bc, same commit built twice): 114.5 min cold, 65.5 min warm. Forty-six commits of churn on top of a warm cache cost about 3 min, since a header change only invalidates modules whose emitted IR changed. The cache lives on an attached volume at `/mnt/clickhouse_thinlto`. praktika gains `--volume-if-exists=HOST:CONTAINER` in a job's docker settings, which mounts the host path only on runners that have it rather than creating an empty directory everywhere as `--volume` does, so the build job enables the cache by the directory's presence and builders without the volume run unchanged. lld prunes inside the linker after each link's backend has run, removing the least recently used entries until the cache fits the size target, so the policy is size-only (`prune_after=0s`, lld's own default is 7 days) at 90% of the volume. The volume must be mounted `relatime` so that hits refresh access times. One `amd_release` `-g` population is ~15 GB. The build job also prints the output of its `sccache --show-stats` and cache stats commands, which `Shell.check` otherwise discards. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
Each ThinLTO link spends most of its time in a single-threaded thin-link phase, so two links overlap almost entirely. Measured on a 16-core, 62 GB builder with zram: 114.5 -> 83 min cold and 65.5 -> 48.5 min warm (with the ThinLTO cache), with a combined memory peak of ~48 GB. `limit_jobs.cmake` caps ThinLTO links from a budget of ~100 GB per `-g` link and applied that cap even to an explicit `PARALLEL_LINK_JOBS`, so a 62 GB builder could never link both binaries at once. The cap now applies only to the auto-calculated value, and the build job sets two link jobs for every ThinLTO flavour. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
Cache does not belong in meta label since builder instances are also used for some tests
strtgbb
added
cicd
On the builder volume the root-owned `lost+found` directory is unreadable for the build user, so `du` failed and the `&&` chain stopped before the entry count and `df`. Exclude it from `du`, count only lld's own `llvmcache-*` entries, and run the three commands independently. https://github.com/Altinity/ClickHouse/actions/runs/40545556921/job/122623854938 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
amd and arm cache entries never share keys, so a build only benefits from a volume where a build for the same target ran recently. With one volume name, the scaler hands each build whichever detached volume it finds first, and a master push runs two amd and two arm ThinLTO builds at once. Give each target its own meta label, `altinity-thinlto-cache-amd` and `altinity-thinlto-cache-arm`, which the runner controller expands into the volume (`clickhouse_thinlto_amd`, `clickhouse_thinlto_arm`) and its location, so neither size nor location is fixed in the repository. The build job mounts whichever of the per-target volumes is present, or the single `/mnt/clickhouse_thinlto` directory that static builders keep on their own disk for both targets. The 180 GB cap in the cache policy stays, since those builders hold both targets' entries. `arm_release` now runs on `BUILDER_ARM`, and `amd_cfi` and the amd cache warmup build on `BUILDER_AMD`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
strtgbb
force-pushed
the
ci/antalya-26.8/thinlto-link-cache
branch
from
October 28, 2026 13:18
511923f to
0d56da2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Significantly reduce release build time by optimizing the link phase.
Changelog category (leave one):
CI/CD Options
Exclude tests:
Regression jobs to run: