Repository navigation
feat(ci): prebuilt Caddy/CrowdSec toolchain image to fix Docker-build timeouts - #1300
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
Contributor
✅ Supply Chain Verification Results✅ PASSED 📦 SBOM Summary
🔍 Vulnerability Scan
📎 Artifacts
Generated by Supply Chain Verification workflow • View Details |
Wikid82
force-pushed
the
feat/prebuilt-toolchain-image
branch
3 times, most recently
from
September 8, 2026 00:34
7372a12 to
5c7ddd8
Compare
Wikid82
force-pushed
the
feat/prebuilt-toolchain-image
branch
from
September 8, 2026 00:47
807dde7 to
5c7ddd8
Compare
Extract the caddy-builder and crowdsec-builder Dockerfile stages into a digest-pinned, multi-arch ghcr.io/wikid82/charon-toolchain image so the ~14-min xcaddy compile runs once per security-relevant change instead of on every CI image build (currently forced via --no-cache-filter in 7 places, the root cause of the PR #1298 build-job timeout cancellations). Rev 2 incorporates the supervisor review: alternatives decision record, corrected security-refresh baseline, withdrawn overstated recurrence claim, pinned xcaddy plugins, guard-never-inert commit slicing, fork-reachable timeout preservation, and a failure-closed freshness guard. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Extract the custom Caddy + CrowdSec build recipe into a separately-versioned
prebuilt image so ordinary app builds stop recompiling xcaddy / CrowdSec on the
hot path (spec docs/plans/current_spec.md §12 Commit 1).
- scripts/lib/dockerfile-stage.sh — shared `extract_stage` (N9), sourced by both
key scripts.
- scripts/toolchain-key.sh — deterministic content-addressed tag
(caddy-crowdsec-<hex>) over the two inline stage bodies + consumed version ARGs
(incl. the two now-pinned xcaddy plugins) + the xx / digest-pinned golang base
lines + .trivyignore + a SCHEMA_VERSION.
- scripts/verify-toolchain-pin.sh — failure-closed freshness guard (B7): on a
trusted same-repo run a missing regctl / missing GHCR_READ_TOKEN /
unresolvable :$KEY / digest mismatch each exit 1; only a fork degrades to
tag-only with a ::warning::.
- scripts/tests/*.bats — determinism + sensitivity + failure-closed matrix (§7);
wired into quality-checks.yml as a non-blocking job for now.
- .github/workflows/toolchain-image.yml — daily schedule + workflow_dispatch +
pull_request(paths) + workflow_call; multi-arch $BUILDPLATFORM cross-compile
(no QEMU) of --target toolchain-runtime, GHCR publish on same-repo, Trivy
CRITICAL/HIGH gate (blocking off-PR, report-only on PR) + SARIF. The
sync-pin-on-pr / open-bump-pr jobs land in Commit 3.
- Dockerfile:
* rename caddy-builder -> caddy-inline, crowdsec-builder -> crowdsec-inline;
* digest-pin the golang:${GO_VERSION}-alpine base of both (N4) and feed the
digest line to the key;
* pin the two loose xcaddy plugins — CADDY_GEOIP2_VERSION (full pseudo-version,
caddy-geoip2 has no semver tags) and CADDY_RATELIMIT_VERSION=0.1.0 (B4) —
and feed both to the key;
* delete the dead crowdsec-fallback stage and the now-unused
CROWDSEC_RELEASE_SHA256 ARG (N1);
* add the toolchain-runtime assembly stage (binaries at identical paths) with
an io.charon.toolchain.key provenance LABEL;
* add temporary aliases `FROM caddy-inline AS caddy-builder` /
`FROM crowdsec-inline AS crowdsec-builder` so the app build is byte-identical
this commit (selector + prebuilt-image consumption land in Commit 2).
- Retarget every --no-cache-filter / no-cache-filters value from
caddy-builder,crowdsec-builder to caddy-inline,crowdsec-inline across
docker-build.yml, security-pr.yml, supply-chain-pr.yml, e2e-tests-split.yml,
nightly-build.yml, and the build-charon-image composite doc comment — so the
CVE-recurrence guard keeps invalidating the real RUN layers through the rename
(B5). Full removal is Commit 4, after the freshness guard is live (Commit 3).
Plugin/base versions are best-effort here; the exact baked CADDY_GEOIP2_VERSION
is confirmed from the first toolchain-image CI run's xcaddy log and re-pinned.
Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
The default app-build path now COPY --from's the digest-pinned prebuilt toolchain
image instead of compiling xcaddy / CrowdSec; the inline stages become the
selectable fork / bootstrap / offline fallback (spec §12 Commit 2).
- Dockerfile:
* CHARON_TOOLCHAIN_DIGEST set to the first published manifest-list (OCI index)
digest sha256:56ea3568… for ghcr.io/wikid82/charon-toolchain
:caddy-crowdsec-1efe7f19fa52a512 (multi-arch: linux/amd64 + linux/arm64
confirmed). Baked plugin versions confirmed from the toolchain build's xcaddy
log: caddy-geoip2 v0.0.0-20260623062220-3675c6e7e63d, caddy-ratelimit v0.1.0
— unchanged from Commit 1's pins.
* add CADDY_BUILDER_SRC / CROWDSEC_BUILDER_SRC selector ARGs (default
toolchain-prebuilt).
* add the toolchain-prebuilt stage
(FROM ${CHARON_TOOLCHAIN_IMAGE}@${CHARON_TOOLCHAIN_DIGEST}) and replace the
temp aliases with FROM ${CADDY_BUILDER_SRC} AS caddy-builder /
FROM ${CROWDSEC_BUILDER_SRC} AS crowdsec-builder. The unreferenced
alternative is pruned — a fork build never pulls the image.
* add org.opencontainers.image.source to the toolchain-runtime LABEL so the
GHCR package links to the repo.
- .github/actions/build-charon-image: new `builder-src` (prebuilt|inline) and
`ghcr-token` inputs; a srcsel resolver step maps them to the two build-args;
a GHCR login step (skipped for inline) so BuildKit can pull the private image.
- Every app-image build path passes the selector + a fork-detection expression
(`head.repo.full_name != '' && != github.repository` -> inline):
* docker-build.yml build-amd64 / build-arm64 (raw buildx, via job env);
* nightly-build.yml (hardcoded prebuilt — same-repo only);
* e2e-tests-split.yml build job (+ GHCR login + packages: read);
* security-pr.yml / supply-chain-pr.yml / the 4 *-integration.yml workflows
(via the composite `builder-src` + `ghcr-token`, + packages: read).
The retargeted `--no-cache-filter caddy-inline,crowdsec-inline` stays for now:
a no-op on the prebuilt path (stages pruned), live on the inline fork path
(B5). Full removal is Commit 4, after the freshness guard is required (Commit 3).
- Makefile: `build-offline` target (--build-arg *_BUILDER_SRC=*-inline).
Local gate: `docker buildx build --target caddy-builder` on the default path
pulls the toolchain image (40 MB) with NO xcaddy / go build step; the pulled
caddy lists http.handlers.{rate_limit,crowdsec,geoip2} + geoip2 and reports
v2.11.4. `docker build --check` clean; toolchain-key unchanged.
Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Spec §12 Commit 3 (+ Rev 2.1 determinism amendment, see docs/plans/current_spec.md
§3.4.3 — flagged for supervisor re-review).
- quality-checks.yml:
* `toolchain-key-tests` (bats) is now BLOCKING (dropped continue-on-error).
* new REQUIRED `verify-toolchain-pin` job — installs regctl
(iarekylew00t/regctl-installer, SHA-pinned) and runs
scripts/verify-toolchain-pin.sh with GHCR_READ_TOKEN=secrets.GITHUB_TOKEN
and head.repo.full_name mapped into env (B7). Failure-closed on same-repo
runs.
- toolchain-image.yml:
* DETERMINISTIC build (fixes a self-perpetuating sync-pin bot-commit loop):
`--provenance=false --sbom=false`, a fixed `SOURCE_DATE_EPOCH=1700000000`,
and `--output type=image,"name=…",push=true,rewrite-timestamp=true`. Two
independent builds now produce an identical manifest-list digest. The
toolchain image is an internal build input; the app image's provenance/SBOM
(docker-build.yml) is separate and unaffected.
* SKIP-IF-ALREADY-PUBLISHED: on any non-forced event, if `:$KEY` resolves in
GHCR the job reuses that digest and skips the build/push entirely. Only
schedule / workflow_dispatch force_rebuild / workflow_call rebuild+repush.
* `sync-pin-on-pr` — idempotent + self-trigger-safe: guarded
`github.actor != 'github-actions[bot]'`, and no-ops unless
`git diff --quiet Dockerfile` shows a real change. Pushes the two-line
TAG/DIGEST bump onto the PR head branch (::notice:: to re-run the check).
* `open-bump-pr` — off the PR path only (schedule / workflow_dispatch /
workflow_call); opens `bot/bump-toolchain-image` against base `development`
(peter-evans/create-pull-request, SHA-pinned) titled `feat(security): refresh
bundled proxy toolchain image`; on failure opens a tracked issue.
- Dockerfile N5 — after the final-stage COPY --from=caddy-builder / crowdsec-builder:
* assert the toolchain caddy binary exposes http.handlers.{rate_limit,crowdsec,
geoip2,waf} (the 4 recipe plugins; the WAF module is `http.handlers.waf`);
* assert cscli runs and emits its recognisable version block (CrowdSec 1.8.x
prints an empty `version:` field regardless of the -X ldflag).
A wrong-arch / rolled-back digest fails the app build here, not only at
toolchain-build time.
- docker-build.yml merge-and-publish: new "Verify pinned toolchain image matches
the recipe (N5)" step — pulls the digest-pinned toolchain image and asserts
its io.charon.toolchain.key LABEL == scripts/toolchain-key.sh.
- renovate.json: packageRules entry — the charon-toolchain digest is bot-owned,
not Renovate-tracked (N7), disabled there.
- docs/plans/current_spec.md §3.4.3: Rev 2.1 amendment documenting the three
determinism / loop-prevention fixes above.
Local: full `docker build .` (default prebuilt path) passes both N5 assertions;
two independent deterministic multi-arch builds produce identical manifest-list
digest sha256:6575f4c6a9f76074870c64df9dd4c9ebee812342f37f52ae5ef8f511ba9f8f00
(pinned in Commit 2); `docker build --check` clean; toolchain key unchanged
(caddy-crowdsec-1efe7f19fa52a512).
Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Spec §12 Commit 4. Now safe: (a) the default app-build path never compiles the inline stages, (b) verify-toolchain-pin is a live required check enforcing pin<->digest freshness per PR (Commit 3), (c) the daily toolchain rebuild + blocking Trivy gate covers base-image drift, (d) the N5 assertions catch a wrong digest in the app build. - docker-build.yml: remove the four `--no-cache-filter caddy-inline` / `crowdsec-inline` lines from build-amd64 / build-arm64; reword the retry-block comment. - security-pr.yml / supply-chain-pr.yml: drop the `no-cache-filters:` override (and its justification comment) from the build-charon-image `with:` block. - e2e-tests-split.yml / nightly-build.yml: drop the `no-cache-filters:` input. - build-charon-image composite: delete the `no-cache-filters` input declaration and its `docker/build-push-action` passthrough; rewrite `description:` to state the binaries come from the digest-pinned toolchain image and every stage is layer-cached. `grep -rn no-cache-filter .github/` is clean. actionlint + lefthook green. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Spec §12 Commit 5 / §3.4.4. security-weekly-rebuild.yml no longer builds a throwaway `charon:security-scan-YYYYMMDD` app image that nothing consumes. Instead: - `toolchain-rebuild` job — `uses: ./.github/workflows/toolchain-image.yml` (workflow_call, force_rebuild: true). This drives the same deterministic `--no-cache --pull` rebuild + publish + BLOCKING Trivy CRITICAL/HIGH gate + `:trivy-toolchain` SARIF + (if the digest moved) `bot/bump-toolchain-image` PR as the daily schedule. N6: the job grants `contents: write`, `pull-requests: write`, `issues: write` (plus packages / security-events) so the reusable workflow's open-bump-pr job has the perms it needs — a workflow_call'ed workflow cannot request perms the caller did not grant. - `extended-report` job — pulls the rebuilt toolchain digest and produces the fuller weekly artefacts: a MEDIUM/LOW Trivy JSON artifact (90-day retention) and an installed-package table in the step summary. No second SARIF upload (the reusable workflow already owns the `:trivy-toolchain` category). - toolchain-image.yml gains `workflow_call` `outputs.key` / `outputs.digest` so the caller can reference the rebuilt digest. actionlint + lefthook green. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
…ale comments Spec §12 Commit 6 / §3.9 / §9. Timeouts (§3.9): - docker-build.yml build-amd64: 15 -> 20 (job + nested nick-fields/retry). No compile on the hot path; 20 gives headroom for a cold GHA cache miss on the fast stages + cache export + push. build-arm64 stays 25; merge-and-publish stays 10; e2e build stays 60. - security-pr / supply-chain-pr / the 4 *-integration jobs KEEP timeout-minutes: 20 (B6 — fork-reachable, they run the ~14-min inline compile). Comments only. Stale-comment sweep: - docker-build.yml: reworded the build-split comment; dropped the dangling `docs/plans/current_spec.md §1.1` cross-ref (pointed at the retired uptime spec) and the QEMU-emulated-compile framing (both builder stages were always $BUILDPLATFORM cross-compiled). - e2e-tests-split.yml: fixed the malformed `# v4uses: docker/setup-buildx-action@…` comment mash on the Set up Docker Buildx step. - Integration + CVE-gate timeout comments reworded to describe the prebuilt vs fork-inline paths. Docs (§9): - ARCHITECTURE.md: Infrastructure table row; new "Prebuilt toolchain image" subsection under Deployment Architecture / Multi-Stage Dockerfile; Directory Structure note (new scripts/workflow, crowdsec-fallback removed, stage renames); Security Architecture / Layer 2 supply-chain-hardening note with the precise §3.8.3 scope; Local Development Setup build note + make build-offline. - SECURITY.md: new "Build Integrity — Bundled Caddy / CrowdSec Toolchain" section (daily deterministic rebuild + blocking Trivy + failure-closed verify-toolchain-pin; unpinned-transitive-MVS gap stated verbatim, no overclaim). - CONTRIBUTING.md: fork PRs compile the toolchain from source; make build-offline. - new docs/ci/toolchain-image.md: operator/maintainer runbook (how the key works, determinism, triggers, force-rebuild, respond to the failure issue / verify-toolchain-pin failure, rollback, one-time bootstrap notes). Ignore-file check (CLAUDE.md): `Dockerfile: COPY scripts/` does copy the new shell scripts into the runtime image (a few KB), consistent with the dozens of scripts/*.sh already shipped — no .dockerignore change. .gitignore: source files that must be committed, no glob match. .codecov.yml: shell/bats/YAML carry no Go/TS coverage — no change. Recorded in docs/ci/toolchain-image.md and the PR. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
…ge context `Dockerfile` `COPY scripts/ /app/scripts/` copies the whole scripts directory into the runtime image. The toolchain helpers added by this PR are used only by .github/workflows/toolchain-image.yml and the quality-checks.yml verify-toolchain-pin / bats jobs from a plain checkout — never from inside a built container — so `.dockerignore` now excludes: scripts/tests/ scripts/toolchain-key.sh scripts/verify-toolchain-pin.sh scripts/lib/dockerfile-stage.sh Verified via a scratch `COPY scripts/` probe: those paths are absent from the build context while runtime scripts (e.g. db-recovery.sh) remain. `.dockerignore` uses blacklist semantics with no `!scripts/…` re-includes, so the plain excludes take effect. Also corrects the stale premise in docs/plans/current_spec.md §2.5 that claimed `scripts/` is not copied into the image. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
…ntext exclusions - docs/ci/toolchain-image.md: the "one-time bootstrap notes" paragraph predated the `.dockerignore` follow-up and still said the new build-only scripts ship in the runtime image. Corrected: `scripts/toolchain-key.sh`, `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh` and `scripts/tests/` are excluded from the image build context via `.dockerignore`. - docs/security.md: added a one-line pointer from the novice-facing security doc to the new "Build Integrity" section in SECURITY.md. Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Wikid82
force-pushed
the
feat/prebuilt-toolchain-image
branch
from
September 8, 2026 06:41
22e9c72 to
e3b76d0
Compare
Wikid82
marked this pull request as ready for review
September 8, 2026 07:02
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every CI workflow that builds the Charon container image recompiles a custom Caddy v2 binary (via
xcaddy, with in-place transitive-dependency patching) and a custom CrowdSec agent from source, from scratch, on every run. Thecaddy-builder/crowdsec-builderDockerfile stages are excluded from all layer caching by--no-cache-filter/no-cache-filtersin six workflows plus the shared composite action. The ~14-minute cold compile blowsbuild-amd64's 15-minute budget and starves the 20-minute integration jobs — PR #1298's four "failed" checks were all CI job-timeout cancellations on the image build.Approach
Extract the two expensive stages' outputs into a separately-versioned, independently-scanned multi-arch prebuilt image —
ghcr.io/wikid82/charon-toolchain— so thexcaddy/ CrowdSec compile runs on a daily schedule and whenever a tracked pin moves, not once per app build. The build recipe stays in the mainDockerfile(stages renamedcaddy-inline/crowdsec-inline); a digest pin + failure-closed freshness guard (scripts/verify-toolchain-pin.sh) keep the CVE-2026-84304-class recurrence guarantee. Fork PRs / bootstrap / offline builds fall back to compiling the inline stages.The toolchain image is built deterministically (
--provenance=false --sbom=false, fixedSOURCE_DATE_EPOCH,rewrite-timestamp) so an unchanged recipe always reproduces the same manifest-list digest — verified across independent local + CI builds.Full technical spec:
docs/plans/current_spec.md(see §3.4.3 Rev 2.1 amendment for the determinism / loop-prevention fixes).Result
build-amd64CACHED, zeroxcaddy--no-cache-filtersitesBase branch
Targets
maindirectly — these are CI/build-infra files; ondevelopmentthey would wait for the weeklynightly → mainpromotion (~a week) while the timeouts keep biting.propagate-changes.ymlback-fillsdevelopment/nightlyafter merge. The digest-bump bot (open-bump-pr) still targetsbase: developmentper spec — only this PR's own base moved.Commits
ci: add toolchain image workflow, key tooling, and split builder stagesrefactor(docker): build the app image from the prebuilt toolchain imagefeat(security): harden build-pipeline integrity verificationperf(ci): stop forcing from-source rebuilds of the bundled toolchainci: route the weekly security rebuild through the toolchain imageci: document the toolchain image, right-size build timeouts, sweep stale commentschore(docker): exclude build-only scripts and bats tests from the image contextdocs(ci): reconcile toolchain-image bootstrap notes with the image-context exclusionsReviews
docs/plans/current_spec.md): supervisor APPROVE (Rev 2, after one revision).docs/reports/qa_report.md): PASS with non-blocking follow-ups. bats 17/17, Trivy green on both images (no new.trivyignoresuppressions), freshness guard failure-closed with no bypass found.Follow-ups (non-blocking, tracked separately)
verify-toolchain-pin+ the bats job as required status checks inmainbranch protection.caddy-geoip2to a real tag once upstream cuts one (currently a Go pseudo-version, Renovate-untracked).tonistiigi/xx+ALPINE_IMAGE+ unresolvable-:$KEY+ empty-digest cases to the bats matrix.regctlrelease version installed byregctl-installer(currentlylatest).make build-offline(the fork/inline whole-app path).Residual supply-chain risk (accept knowingly)
:$KEY-tag window under a maintainer-credential compromise, bounded by fork-path exclusion, the daily deterministic--no-cache --pullrebuild + self-healing bump PR, and the app-image scan/sign gates.Notes
charon-toolchainGHCR package visibility to public later — not required; CI pulls it viadocker login ghcr.io+packages: readin every app-build path.https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS