feat(promotion): gate promotion out of an org on a passing check

[scott-lowe-vapi]

Value

V.A.L.U.E. tier: project — PR 10 of 10 for inline simulation PR checks (TEST-141), the "check before deploy" step in promotion.

Stacked on #65 (the promotion partial-failure fix), now that #57–#64 have merged. This PR is the single gate commit.

• Problem: promotion copies staging's reviewed files into production, but nothing checks that staging's agents still behave before they move on. Teams promoting dev → staging → prod need a behaviour gate between orgs, without new infrastructure.
• Who it affects: multi-org gitops users (the promotion pipeline), who get a "check before deploy" step with one line of promotion.yml. Single-org users are unaffected.
• What changes:
  • promotion.yml accepts orgs.<slug>.check: <name> (a slug), naming a vapi-checks.yml check.
  • New src/promotion-gate.ts:
    • Validation before any transition: the check must exist, and its org and runOrg must be the gated org, otherwise the run errors. vapi-checks.yml is required once any org is gated.
    • Plan line: what the gate would run, built offline.
    • Live gate: the check runs live and reduces to the worst target result.
  • src/promote-cmd.ts: in each transition, after the plan is built:
    • no changes skips the gate;
    • plan-only prints check would run <name> in <org> (<n> simulations × <t> targets);
    • --apply runs the check (after the bindings refresh, before promotionPlanApply writes anything). Any non-pass throws Promotion out of <org> blocked: check <name> <outcome> (<run url>).
    • A pass is cached per source org and dropped once a transition applies into that org.
    • promotionCommandRun(args, overrides) now takes Partial<PromotionDeps> (childRun, checkRun).
  • .github/workflows/promotion.yml: timeout-minutes: 90 on the "Reconcile configured promotions" step, not the job, so the if: always() commit step (fixed in fix(promotion): commit the files of transitions that applied when a later one fails #65) still runs after a blocked or slow gate.
  • Docs: promotion.example.yml (a commented check:), a README "Check before promoting" section, and a pointer from "PR Checks".

Evidence of value

The real gate, run live in the owner's test org on the TEST-141 parity squad.

• Setup: a scratch repo whose promotion.yml gates parity on check core, with pipeline parity → parity-prod.
• The run: promote --pipeline release --from parity --to parity-prod --apply.
• The fake: the child runner was faked, so bindings pulls were no-ops and the downstream apply.ts was recorded but not run. No second org was needed or touched.

Variant	Gate run	Result	Downstream apply	resources/parity-prod/
Degraded scheduler prompt	7ed19587: 2 of 3 failed	Promotion out of parity blocked: check core failed (https://dashboard.vapi.ai/simulations/run/7ed19587-…)	none	empty (nothing written)
Fixture as-is	95470670: 3 of 3 passed	promoted	["parity-prod"]	written; 20 applied paths recorded

The test org's resource counts were identical before and after both gate runs.

Tests: npm test goes from 484 (#65) to 492 passing, and #68's golden promotion test passes unchanged.

Testing plan

• tests/promotion-gate.test.ts (6 tests, real git fixture, injected childRun / checkRun):
  • a pass applies;
  • failed and incomplete both block with the exact message, with no apply and the target untouched;
  • plan-only prints the line and runs nothing;
  • no changes skips the gate;
  • the three config errors (no vapi-checks.yml, unknown check, check in another org) stop before anything applies;
  • the pass cache: reused for two pipelines out of one org, and re-run after a transition applies into the gated org.
• No gate configured, no change: with no check: in promotion.yml and an invalid vapi-checks.yml present, plan and --apply both succeed, checkRun is never called, and the plan output equals a pinned string. That string is exactly what fix(promotion): commit the files of transitions that applied when a later one fails #65's code (before the gate existed) prints for the same fixture, which I confirmed by running fix(promotion): commit the files of transitions that applied when a later one fails #65's promote-cmd on it. So the gate is invisible unless someone opts in.
• tests/promotion.test.ts: orgs.<slug>.check is parsed, and a non-slug is rejected.
• Not tested:
  • A real two-org promotion: only one test org was available. The downstream apply was faked, so the blocked case shows nothing written, and the pass case shows the apply was called.
  • A GitHub Actions promotion run with a gate, including the step timeout firing.
• Found while testing (pre-existing, out of scope): promotion's dependency check rejects simulations that reference a stock personality by UUID, with "Referenced managed dependency is missing from source: personalities/a0000000-…". So a gated org's tests need local personality files until that's fixed.

Stacked on #65.

Refs TEST-141

After review

The gate now refuses, when the config loads and before anything applies:

• an unknown key under an org in promotion.yml, so a misspelled check: can't silently drop the gate;
• toolMocks: off and stripWebhooks: false, because a gate runs in the real org, never a CI org;
• a check baseUrl that differs from the org's baseUrl in promotion.yml, so the org's key only goes to the host promotion uses (the gate always uses that host);
• a gate on an org that is last in every pipeline, where it would never run;
• gated checks whose combined budget is over 300 minutes.

It also fixes:

• Deadline: each batch of 3 targets gets a full timeoutMinutes, so a check with more than 3 targets is no longer falsely blocked as incomplete.
• Step timeout: raised from 90 to 330 minutes, as a safety net that no longer cuts short long ungated promotions.
• Tests: the deadline and the worst-target rule are pure helpers with their own tests.

The guide changes (blocks stop the whole run, simulation cost, the stock-personality limitation, accurate wording) are in #71. The promote User-Agent for gate runs is in #78. The block-report detail, fetch-stubbed gate test and deduplication are follow-ups.

🤖 Generated with Claude Code

[scott-lowe-vapi]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: identify every gitops API request with a User-Agent #78
• feat(validate): catch broken references and show findings on the PR #77
• ci: validate every org's resources on every pull request #76
• ci: lint workflows and test the PR check's key-sharing decision #75
• test(docs): check that doc links, commands and flags exist #74
• fix(cleanup): never delete resources matched by .vapi-ignore #73
• docs(agents): one compact guide for Claude, Codex and Cursor #72
• docs: edit the guides for new readers and fix inaccuracies #71
• docs: restructure the README into a short landing page and topic guides #70
• docs: prepare for public release: scrub internal names, fix examples, add community files #69
• feat(promotion): gate promotion out of an org on a passing check #66 👈 (View in Graphite)
• fix(promotion): commit the files of transitions that applied when a later one fails #65
• test(promotion): pin the exact files promoting an assistant writes to the target #68
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[scott-lowe-vapi]

Merge activity

• Oct 3, 5:59 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Oct 3, 6:14 AM UTC: Graphite couldn't merge this PR because it had merge conflicts.

[dhruva-vapi]

lgtm, it fails closed on anything short of every target passing and a pass gets dropped once something is promoted into that org

couple of non-blocking things:

• the promotion step timeout is 90 min but a check's timeoutMinutes can go up to 120, and two gated orgs in one run add up, so a long check just gets killed by the step instead of reporting a result. can we check the gated timeouts against that budget in promotionChecksLoad
• this runs the sims again at promotion time on top of the PR check, so every change gets simulated twice. fine as a choice but can we call out the cost in the README so customers aren't surprised

[chris-garber-vapi]

The core design holds up: the gate fails closed, the pass cache is invalidated correctly (I checked that pull --bindings-only only writes state, so a cached pass can't go stale between transitions), and an ungated promotion.yml doesn't change behavior. Tests and tsc --noEmit pass on 84daba8.

Most important, highest first:

• 🟠 A typo'd check: key turns the gate off with no error (confirmed on this branch).
• 🟠 toolMocks: off checks run real tools in the gated org, because runOrg can't be a CI org.
• 🟠 Checks with more than 3 targets can be falsely blocked as incomplete, because timeoutMinutes is also the whole check's budget.
• 🟠 The 90-minute step timeout now applies to ungated promotions too.

Severity: 🔴 blocker · 🟠 should fix before merge · 🟡 should fix, can be stacked · 🟢 nit.

[scott-lowe-vapi]

@dhruva-vapi both done:

• Timeouts: gated checks' combined budget (timeoutMinutes per batch of 3 targets, across every gated org) is checked against 300 minutes when the config loads, and the step cap is now 330 minutes, as a safety net. A long check is rejected up front instead of being killed.
• Cost: the promotion guide's "Check before promoting" section (in docs: edit the guides for new readers and fix inaccuracies #71) says each gated promotion runs its simulations again, on top of the PR check, and uses simulation minutes.

🤖 Generated with Claude Code