Skip to content

feat(ci): prebuilt Caddy/CrowdSec toolchain image to fix Docker-build timeouts - #1300

Merged
Wikid82 merged 10 commits into
mainfrom
feat/prebuilt-toolchain-image
Sep 8, 2026
Merged

Wikid82 merged 10 commits into
mainfrom
feat/prebuilt-toolchain-image

Conversation

@Wikid82

@Wikid82 Wikid82 commented Sep 7, 2026 •

Copy link
Copy Markdown
Owner

Problem

Every CI workflow that builds the Charon container image recompiles a custom Caddy v2 binary (via xcaddy, with in-place transitive-dependency patching) and a custom CrowdSec agent from source, from scratch, on every run. The caddy-builder / crowdsec-builder Dockerfile stages are excluded from all layer caching by --no-cache-filter / no-cache-filters in six workflows plus the shared composite action. The ~14-minute cold compile blows build-amd64's 15-minute budget and starves the 20-minute integration jobs — PR #1298's four "failed" checks were all CI job-timeout cancellations on the image build.

Approach

Extract the two expensive stages' outputs into a separately-versioned, independently-scanned multi-arch prebuilt image — ghcr.io/wikid82/charon-toolchain — so the xcaddy / CrowdSec compile runs on a daily schedule and whenever a tracked pin moves, not once per app build. The build recipe stays in the main Dockerfile (stages renamed caddy-inline / crowdsec-inline); a digest pin + failure-closed freshness guard (scripts/verify-toolchain-pin.sh) keep the CVE-2026-84304-class recurrence guarantee. Fork PRs / bootstrap / offline builds fall back to compiling the inline stages.

The toolchain image is built deterministically (--provenance=false --sbom=false, fixed SOURCE_DATE_EPOCH, rewrite-timestamp) so an unchanged recipe always reproduces the same manifest-list digest — verified across independent local + CI builds.

Full technical spec: docs/plans/current_spec.md (see §3.4.3 Rev 2.1 amendment for the determinism / loop-prevention fixes).

Result

Before After
build-amd64 ~15 min → timeout/cancel ~2m40s, all stages CACHED, zero xcaddy
Cerberus / integration jobs 20 min → cancelled ~2 min
--no-cache-filter sites 7 (+ composite input) 0
Toolchain digest for an unchanged recipe new sha every build (bot-commit loop) stable across local + CI + weekly rebuilds

Base branch

Targets main directly — these are CI/build-infra files; on development they would wait for the weekly nightly → main promotion (~a week) while the timeouts keep biting. propagate-changes.yml back-fills development / nightly after merge. The digest-bump bot (open-bump-pr) still targets base: development per spec — only this PR's own base moved.

Commits

  1. ci: add toolchain image workflow, key tooling, and split builder stages
  2. refactor(docker): build the app image from the prebuilt toolchain image
  3. feat(security): harden build-pipeline integrity verification
  4. perf(ci): stop forcing from-source rebuilds of the bundled toolchain
  5. ci: route the weekly security rebuild through the toolchain image
  6. ci: document the toolchain image, right-size build timeouts, sweep stale comments
  7. chore(docker): exclude build-only scripts and bats tests from the image context
  8. docs(ci): reconcile toolchain-image bootstrap notes with the image-context exclusions

Reviews

  • Plan (docs/plans/current_spec.md): supervisor APPROVE (Rev 2, after one revision).
  • Implementation: supervisor APPROVE — all B1–B7 guarantees re-verified as held.
  • QA / security (docs/reports/qa_report.md): PASS with non-blocking follow-ups. bats 17/17, Trivy green on both images (no new .trivyignore suppressions), freshness guard failure-closed with no bypass found.

Follow-ups (non-blocking, tracked separately)

  • Enroll verify-toolchain-pin + the bats job as required status checks in main branch protection.
  • Pin caddy-geoip2 to a real tag once upstream cuts one (currently a Go pseudo-version, Renovate-untracked).
  • Add tonistiigi/xx + ALPINE_IMAGE + unresolvable-:$KEY + empty-digest cases to the bats matrix.
  • Pin the regctl release version installed by regctl-installer (currently latest).
  • Periodic CI coverage of make build-offline (the fork/inline whole-app path).

Residual supply-chain risk (accept knowingly)

  • ~24h poisoned-:$KEY-tag window under a maintainer-credential compromise, bounded by fork-path exclusion, the daily deterministic --no-cache --pull rebuild + self-healing bump PR, and the app-image scan/sign gates.
  • The toolchain image is digest-pinned and recipe-bound (LABEL ↔ key check) but not Cosign-signed.

Notes

  • A maintainer may optionally set the charon-toolchain GHCR package visibility to public later — not required; CI pulls it via docker login ghcr.io + packages: read in every app-build path.

https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS

@codecov

codecov Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-advanced-security

Copy link
Copy Markdown
Contributor

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Supply Chain Verification Results

✅ PASSED

📦 SBOM Summary

  • Components: 1769

🔍 Vulnerability Scan

Severity Count
🔴 Critical 0
🟠 High 0
🟡 Medium 0
🟢 Low 0
Total 0

📎 Artifacts

  • SBOM (CycloneDX JSON) and Grype results available in workflow artifacts

Generated by Supply Chain Verification workflow • View Details

@Wikid82
Wikid82 force-pushed the feat/prebuilt-toolchain-image branch 3 times, most recently from 7372a12 to 5c7ddd8 Compare September 8, 2026 00:34
@Wikid82
Wikid82 changed the base branch from development to main September 8, 2026 00:34
@Wikid82
Wikid82 force-pushed the feat/prebuilt-toolchain-image branch from 807dde7 to 5c7ddd8 Compare September 8, 2026 00:47
Extract the caddy-builder and crowdsec-builder Dockerfile stages into a
digest-pinned, multi-arch ghcr.io/wikid82/charon-toolchain image so the
~14-min xcaddy compile runs once per security-relevant change instead of
on every CI image build (currently forced via --no-cache-filter in 7
places, the root cause of the PR #1298 build-job timeout cancellations).

Rev 2 incorporates the supervisor review: alternatives decision record,
corrected security-refresh baseline, withdrawn overstated recurrence
claim, pinned xcaddy plugins, guard-never-inert commit slicing,
fork-reachable timeout preservation, and a failure-closed freshness guard.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Extract the custom Caddy + CrowdSec build recipe into a separately-versioned
prebuilt image so ordinary app builds stop recompiling xcaddy / CrowdSec on the
hot path (spec docs/plans/current_spec.md §12 Commit 1).

- scripts/lib/dockerfile-stage.sh — shared `extract_stage` (N9), sourced by both
  key scripts.
- scripts/toolchain-key.sh — deterministic content-addressed tag
  (caddy-crowdsec-<hex>) over the two inline stage bodies + consumed version ARGs
  (incl. the two now-pinned xcaddy plugins) + the xx / digest-pinned golang base
  lines + .trivyignore + a SCHEMA_VERSION.
- scripts/verify-toolchain-pin.sh — failure-closed freshness guard (B7): on a
  trusted same-repo run a missing regctl / missing GHCR_READ_TOKEN /
  unresolvable :$KEY / digest mismatch each exit 1; only a fork degrades to
  tag-only with a ::warning::.
- scripts/tests/*.bats — determinism + sensitivity + failure-closed matrix (§7);
  wired into quality-checks.yml as a non-blocking job for now.
- .github/workflows/toolchain-image.yml — daily schedule + workflow_dispatch +
  pull_request(paths) + workflow_call; multi-arch $BUILDPLATFORM cross-compile
  (no QEMU) of --target toolchain-runtime, GHCR publish on same-repo, Trivy
  CRITICAL/HIGH gate (blocking off-PR, report-only on PR) + SARIF. The
  sync-pin-on-pr / open-bump-pr jobs land in Commit 3.
- Dockerfile:
  * rename caddy-builder -> caddy-inline, crowdsec-builder -> crowdsec-inline;
  * digest-pin the golang:${GO_VERSION}-alpine base of both (N4) and feed the
    digest line to the key;
  * pin the two loose xcaddy plugins — CADDY_GEOIP2_VERSION (full pseudo-version,
    caddy-geoip2 has no semver tags) and CADDY_RATELIMIT_VERSION=0.1.0 (B4) —
    and feed both to the key;
  * delete the dead crowdsec-fallback stage and the now-unused
    CROWDSEC_RELEASE_SHA256 ARG (N1);
  * add the toolchain-runtime assembly stage (binaries at identical paths) with
    an io.charon.toolchain.key provenance LABEL;
  * add temporary aliases `FROM caddy-inline AS caddy-builder` /
    `FROM crowdsec-inline AS crowdsec-builder` so the app build is byte-identical
    this commit (selector + prebuilt-image consumption land in Commit 2).
- Retarget every --no-cache-filter / no-cache-filters value from
  caddy-builder,crowdsec-builder to caddy-inline,crowdsec-inline across
  docker-build.yml, security-pr.yml, supply-chain-pr.yml, e2e-tests-split.yml,
  nightly-build.yml, and the build-charon-image composite doc comment — so the
  CVE-recurrence guard keeps invalidating the real RUN layers through the rename
  (B5). Full removal is Commit 4, after the freshness guard is live (Commit 3).

Plugin/base versions are best-effort here; the exact baked CADDY_GEOIP2_VERSION
is confirmed from the first toolchain-image CI run's xcaddy log and re-pinned.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
The default app-build path now COPY --from's the digest-pinned prebuilt toolchain
image instead of compiling xcaddy / CrowdSec; the inline stages become the
selectable fork / bootstrap / offline fallback (spec §12 Commit 2).

- Dockerfile:
  * CHARON_TOOLCHAIN_DIGEST set to the first published manifest-list (OCI index)
    digest sha256:56ea3568… for ghcr.io/wikid82/charon-toolchain
    :caddy-crowdsec-1efe7f19fa52a512 (multi-arch: linux/amd64 + linux/arm64
    confirmed). Baked plugin versions confirmed from the toolchain build's xcaddy
    log: caddy-geoip2 v0.0.0-20260623062220-3675c6e7e63d, caddy-ratelimit v0.1.0
    — unchanged from Commit 1's pins.
  * add CADDY_BUILDER_SRC / CROWDSEC_BUILDER_SRC selector ARGs (default
    toolchain-prebuilt).
  * add the toolchain-prebuilt stage
    (FROM ${CHARON_TOOLCHAIN_IMAGE}@${CHARON_TOOLCHAIN_DIGEST}) and replace the
    temp aliases with FROM ${CADDY_BUILDER_SRC} AS caddy-builder /
    FROM ${CROWDSEC_BUILDER_SRC} AS crowdsec-builder. The unreferenced
    alternative is pruned — a fork build never pulls the image.
  * add org.opencontainers.image.source to the toolchain-runtime LABEL so the
    GHCR package links to the repo.
- .github/actions/build-charon-image: new `builder-src` (prebuilt|inline) and
  `ghcr-token` inputs; a srcsel resolver step maps them to the two build-args;
  a GHCR login step (skipped for inline) so BuildKit can pull the private image.
- Every app-image build path passes the selector + a fork-detection expression
  (`head.repo.full_name != '' && != github.repository` -> inline):
  * docker-build.yml build-amd64 / build-arm64 (raw buildx, via job env);
  * nightly-build.yml (hardcoded prebuilt — same-repo only);
  * e2e-tests-split.yml build job (+ GHCR login + packages: read);
  * security-pr.yml / supply-chain-pr.yml / the 4 *-integration.yml workflows
    (via the composite `builder-src` + `ghcr-token`, + packages: read).
  The retargeted `--no-cache-filter caddy-inline,crowdsec-inline` stays for now:
  a no-op on the prebuilt path (stages pruned), live on the inline fork path
  (B5). Full removal is Commit 4, after the freshness guard is required (Commit 3).
- Makefile: `build-offline` target (--build-arg *_BUILDER_SRC=*-inline).

Local gate: `docker buildx build --target caddy-builder` on the default path
pulls the toolchain image (40 MB) with NO xcaddy / go build step; the pulled
caddy lists http.handlers.{rate_limit,crowdsec,geoip2} + geoip2 and reports
v2.11.4. `docker build --check` clean; toolchain-key unchanged.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Spec §12 Commit 3 (+ Rev 2.1 determinism amendment, see docs/plans/current_spec.md
§3.4.3 — flagged for supervisor re-review).

- quality-checks.yml:
  * `toolchain-key-tests` (bats) is now BLOCKING (dropped continue-on-error).
  * new REQUIRED `verify-toolchain-pin` job — installs regctl
    (iarekylew00t/regctl-installer, SHA-pinned) and runs
    scripts/verify-toolchain-pin.sh with GHCR_READ_TOKEN=secrets.GITHUB_TOKEN
    and head.repo.full_name mapped into env (B7). Failure-closed on same-repo
    runs.
- toolchain-image.yml:
  * DETERMINISTIC build (fixes a self-perpetuating sync-pin bot-commit loop):
    `--provenance=false --sbom=false`, a fixed `SOURCE_DATE_EPOCH=1700000000`,
    and `--output type=image,"name=…",push=true,rewrite-timestamp=true`. Two
    independent builds now produce an identical manifest-list digest. The
    toolchain image is an internal build input; the app image's provenance/SBOM
    (docker-build.yml) is separate and unaffected.
  * SKIP-IF-ALREADY-PUBLISHED: on any non-forced event, if `:$KEY` resolves in
    GHCR the job reuses that digest and skips the build/push entirely. Only
    schedule / workflow_dispatch force_rebuild / workflow_call rebuild+repush.
  * `sync-pin-on-pr` — idempotent + self-trigger-safe: guarded
    `github.actor != 'github-actions[bot]'`, and no-ops unless
    `git diff --quiet Dockerfile` shows a real change. Pushes the two-line
    TAG/DIGEST bump onto the PR head branch (::notice:: to re-run the check).
  * `open-bump-pr` — off the PR path only (schedule / workflow_dispatch /
    workflow_call); opens `bot/bump-toolchain-image` against base `development`
    (peter-evans/create-pull-request, SHA-pinned) titled `feat(security): refresh
    bundled proxy toolchain image`; on failure opens a tracked issue.
- Dockerfile N5 — after the final-stage COPY --from=caddy-builder / crowdsec-builder:
  * assert the toolchain caddy binary exposes http.handlers.{rate_limit,crowdsec,
    geoip2,waf} (the 4 recipe plugins; the WAF module is `http.handlers.waf`);
  * assert cscli runs and emits its recognisable version block (CrowdSec 1.8.x
    prints an empty `version:` field regardless of the -X ldflag).
  A wrong-arch / rolled-back digest fails the app build here, not only at
  toolchain-build time.
- docker-build.yml merge-and-publish: new "Verify pinned toolchain image matches
  the recipe (N5)" step — pulls the digest-pinned toolchain image and asserts
  its io.charon.toolchain.key LABEL == scripts/toolchain-key.sh.
- renovate.json: packageRules entry — the charon-toolchain digest is bot-owned,
  not Renovate-tracked (N7), disabled there.
- docs/plans/current_spec.md §3.4.3: Rev 2.1 amendment documenting the three
  determinism / loop-prevention fixes above.

Local: full `docker build .` (default prebuilt path) passes both N5 assertions;
two independent deterministic multi-arch builds produce identical manifest-list
digest sha256:6575f4c6a9f76074870c64df9dd4c9ebee812342f37f52ae5ef8f511ba9f8f00
(pinned in Commit 2); `docker build --check` clean; toolchain key unchanged
(caddy-crowdsec-1efe7f19fa52a512).

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Spec §12 Commit 4. Now safe: (a) the default app-build path never compiles the
inline stages, (b) verify-toolchain-pin is a live required check enforcing
pin<->digest freshness per PR (Commit 3), (c) the daily toolchain rebuild +
blocking Trivy gate covers base-image drift, (d) the N5 assertions catch a
wrong digest in the app build.

- docker-build.yml: remove the four `--no-cache-filter caddy-inline` /
  `crowdsec-inline` lines from build-amd64 / build-arm64; reword the retry-block
  comment.
- security-pr.yml / supply-chain-pr.yml: drop the `no-cache-filters:` override
  (and its justification comment) from the build-charon-image `with:` block.
- e2e-tests-split.yml / nightly-build.yml: drop the `no-cache-filters:` input.
- build-charon-image composite: delete the `no-cache-filters` input declaration
  and its `docker/build-push-action` passthrough; rewrite `description:` to state
  the binaries come from the digest-pinned toolchain image and every stage is
  layer-cached.

`grep -rn no-cache-filter .github/` is clean. actionlint + lefthook green.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
Spec §12 Commit 5 / §3.4.4.

security-weekly-rebuild.yml no longer builds a throwaway
`charon:security-scan-YYYYMMDD` app image that nothing consumes. Instead:

- `toolchain-rebuild` job — `uses: ./.github/workflows/toolchain-image.yml`
  (workflow_call, force_rebuild: true). This drives the same deterministic
  `--no-cache --pull` rebuild + publish + BLOCKING Trivy CRITICAL/HIGH gate +
  `:trivy-toolchain` SARIF + (if the digest moved) `bot/bump-toolchain-image` PR
  as the daily schedule. N6: the job grants `contents: write`,
  `pull-requests: write`, `issues: write` (plus packages / security-events) so
  the reusable workflow's open-bump-pr job has the perms it needs — a
  workflow_call'ed workflow cannot request perms the caller did not grant.
- `extended-report` job — pulls the rebuilt toolchain digest and produces the
  fuller weekly artefacts: a MEDIUM/LOW Trivy JSON artifact (90-day retention)
  and an installed-package table in the step summary. No second SARIF upload
  (the reusable workflow already owns the `:trivy-toolchain` category).
- toolchain-image.yml gains `workflow_call` `outputs.key` / `outputs.digest` so
  the caller can reference the rebuilt digest.

actionlint + lefthook green.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
…ale comments

Spec §12 Commit 6 / §3.9 / §9.

Timeouts (§3.9):
- docker-build.yml build-amd64: 15 -> 20 (job + nested nick-fields/retry). No
  compile on the hot path; 20 gives headroom for a cold GHA cache miss on the
  fast stages + cache export + push. build-arm64 stays 25; merge-and-publish
  stays 10; e2e build stays 60.
- security-pr / supply-chain-pr / the 4 *-integration jobs KEEP timeout-minutes:
  20 (B6 — fork-reachable, they run the ~14-min inline compile). Comments only.

Stale-comment sweep:
- docker-build.yml: reworded the build-split comment; dropped the dangling
  `docs/plans/current_spec.md §1.1` cross-ref (pointed at the retired uptime
  spec) and the QEMU-emulated-compile framing (both builder stages were always
  $BUILDPLATFORM cross-compiled).
- e2e-tests-split.yml: fixed the malformed `# v4uses: docker/setup-buildx-action@…`
  comment mash on the Set up Docker Buildx step.
- Integration + CVE-gate timeout comments reworded to describe the prebuilt vs
  fork-inline paths.

Docs (§9):
- ARCHITECTURE.md: Infrastructure table row; new "Prebuilt toolchain image"
  subsection under Deployment Architecture / Multi-Stage Dockerfile; Directory
  Structure note (new scripts/workflow, crowdsec-fallback removed, stage
  renames); Security Architecture / Layer 2 supply-chain-hardening note with the
  precise §3.8.3 scope; Local Development Setup build note + make build-offline.
- SECURITY.md: new "Build Integrity — Bundled Caddy / CrowdSec Toolchain"
  section (daily deterministic rebuild + blocking Trivy + failure-closed
  verify-toolchain-pin; unpinned-transitive-MVS gap stated verbatim, no
  overclaim).
- CONTRIBUTING.md: fork PRs compile the toolchain from source; make build-offline.
- new docs/ci/toolchain-image.md: operator/maintainer runbook (how the key
  works, determinism, triggers, force-rebuild, respond to the failure issue /
  verify-toolchain-pin failure, rollback, one-time bootstrap notes).

Ignore-file check (CLAUDE.md): `Dockerfile: COPY scripts/` does copy the new
shell scripts into the runtime image (a few KB), consistent with the dozens of
scripts/*.sh already shipped — no .dockerignore change. .gitignore: source files
that must be committed, no glob match. .codecov.yml: shell/bats/YAML carry no
Go/TS coverage — no change. Recorded in docs/ci/toolchain-image.md and the PR.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
…ge context

`Dockerfile` `COPY scripts/ /app/scripts/` copies the whole scripts directory
into the runtime image. The toolchain helpers added by this PR are used only by
.github/workflows/toolchain-image.yml and the quality-checks.yml
verify-toolchain-pin / bats jobs from a plain checkout — never from inside a
built container — so `.dockerignore` now excludes:

  scripts/tests/
  scripts/toolchain-key.sh
  scripts/verify-toolchain-pin.sh
  scripts/lib/dockerfile-stage.sh

Verified via a scratch `COPY scripts/` probe: those paths are absent from the
build context while runtime scripts (e.g. db-recovery.sh) remain. `.dockerignore`
uses blacklist semantics with no `!scripts/…` re-includes, so the plain excludes
take effect.

Also corrects the stale premise in docs/plans/current_spec.md §2.5 that claimed
`scripts/` is not copied into the image.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
…ntext exclusions

- docs/ci/toolchain-image.md: the "one-time bootstrap notes" paragraph predated
  the `.dockerignore` follow-up and still said the new build-only scripts ship in
  the runtime image. Corrected: `scripts/toolchain-key.sh`,
  `scripts/verify-toolchain-pin.sh`, `scripts/lib/dockerfile-stage.sh` and
  `scripts/tests/` are excluded from the image build context via `.dockerignore`.
- docs/security.md: added a one-line pointer from the novice-facing security doc
  to the new "Build Integrity" section in SECURITY.md.

Claude-Session: https://claude.ai/code/session_01KXA4x9LrA2AsnLrvdHMZbS
@Wikid82
Wikid82 force-pushed the feat/prebuilt-toolchain-image branch from 22e9c72 to e3b76d0 Compare September 8, 2026 06:41
@Wikid82
Wikid82 marked this pull request as ready for review September 8, 2026 07:02
@Wikid82
Wikid82 merged commit 999ca5e into main Sep 8, 2026
54 checks passed
@Wikid82
Wikid82 deleted the feat/prebuilt-toolchain-image branch September 18, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants