Require review, not failure, for narrowly matched data-limited UK replay regressions

changelog.d/653.changed.md

@@ -0,0 +1 @@
+The UK replay qualification gate records narrowly scoped data-limited exceptions under policy d1344-replay-regressions-v2. A matching regression requires coherent zero candidate effects, approved input lineage, and digest-bound Budget-input and positive-input probe evidence from the same candidate-pair attempt. Named acceptance must restate the publication designation; unmatched failures still fail.

changelog.d/654.fixed.md

@@ -0,0 +1 @@
+Replay exceptions reject nonzero candidate effects at every reported fiscal level and retain cumulative usable-input history across certified pairs. Budget-input evidence is bound to the exact pair and attempt, status maps must agree with missing-input claims, and probe evidence must match the installed model and recorded baseline/reform arithmetic.

scripts/check_uk_candidate.py

@@ -66,6 +66,7 @@
     ResolvedCandidate,
     WheelFile,
     accept_review,
+    attempt_evidence_id,
     attempt_is_recoverable,
     attempt_is_waiting_for_replay,
     attempt_key,
@@ -76,6 +77,8 @@
     candidate_pr_body,
     expected_release_bundle,
     finalize_attempt,
+    load_attempt_budget_inputs,
+    load_attempt_replay_probe,
     packaged_manifest_sha256,
     plan_promotion,
     prepare_candidate_commit,
@@ -369,6 +372,16 @@ def cmd_gates(args: argparse.Namespace) -> int:
         ledger_root=ledger_root,
     )
     write_gate_results(attempt, results, evidence)
+    from policyengine.provenance.replay_probe import run_replay_probes
+
+    probe_result, probe_evidence = run_replay_probes(
+        attempt_id=attempt_evidence_id(attempt),
+        candidate_model_version=payload["pair"]["model"]["version"],
+    )
+    write_gate_results(attempt, [probe_result], {"replay_probe": probe_evidence})
+    print(
+        f"{probe_result.gate_id:20s} {probe_result.status:13s} {probe_result.summary}"
+    )
     for result in results:
         print(f"{result.gate_id:20s} {result.status:13s} {result.summary}")
     return 0
@@ -417,10 +430,19 @@ def cmd_poll_replay(args: argparse.Namespace) -> int:
         read_json(attempt / "replay" / "request.json")
     )
     fetch = scorecard_file(settings["repository"])
+    budget_result, budget_evidence = load_attempt_budget_inputs(attempt)
+    probe_result, probe_evidence = load_attempt_replay_probe(attempt)
     deadline = time.monotonic() + 60 * args.wait_minutes
     while True:
         gate, status, receipt = read_replay(
-            request, fetch=fetch, state_ref=settings["state_ref"]
+            request,
+            fetch=fetch,
+            state_ref=settings["state_ref"],
+            budget_inputs_result=budget_result,
+            budget_inputs_evidence=budget_evidence,
+            probe_result=probe_result,
+            probe_evidence=probe_evidence,
+            attempt_id=attempt_evidence_id(attempt),
         )
         if (status is not None and status.terminal) or time.monotonic() >= deadline:
             break

src/policyengine/data/qualification/replay_policy.md

@@ -1,4 +1,4 @@
-# Replay gate policy d1344-replay-regressions-v1
+# Replay gate policy d1344-replay-regressions-v2
 
 Decided under cos d1344 by workstream W4 on 15 October 2026, after two
 independent reviews. The pull request that added this file records the
@@ -7,6 +7,11 @@ October 2026, before first use, after the third to fifth reviews of
 policyengine.py#637: scorecard's own record of a mixed known gap no longer
 blocks, flag counters must equal the flags the rows carry, and any
 candidate value other than £0 at a known gap is a contradiction.
+The data-limited exception amendment below was adopted by the Budget
+coordinator following independent review agreement under cos d1344:
+RULE AGREE (GPT-6.1 Sol, round 2, 17 October 2026, head 7f518ff61) and
+CODE AGREE WITH NITS (round 3, head dbc1806df), with the P3 nit fixed in
+this commit. The registry records this adopted decision and review status.
 `evaluate_replay_receipt` in `policyengine.provenance.qualification`
 applies this text. Each qualification receipt records the policy id and
 this file's SHA-256 in its `replay` gate details.
@@ -20,23 +25,26 @@ every other flagged move to a named reviewer.
 
 - A rejected request, `no_supported_years`, integrity `failed` (a crash or
   missing outputs, persistent or not), or a receipt not bound to the
-  request.
+  request. Only the narrow data-limited exceptions below can make a failed
+  integrity reviewable.
 - A mapping flag.
 - A lost result: a coverage row with a base value and no candidate value,
   unless it is the record of a known gap (below).
 - A lost response:
-  - a measure-year the candidate lists as inert, unless it is a known gap;
+  - a measure-year the candidate lists as inert, unless it is a known gap
+    or a data-limited exception applies;
   - a measure-year total at exactly £0 where the base's absolute value is
     at least the diff's bound, max(£1m, 1% of the base's absolute value),
-    whether the base was positive or negative. A known gap never excuses
-    this: a base whose record lists the measure-year as a gap cannot also
-    hold a material value for it.
+    whether the base was positive or negative, unless a data-limited
+    exception applies. A known gap never excuses this: a base whose record
+    lists the measure-year as a gap cannot also hold a material value for
+    it, and a data-limited exception never excuses that contradiction.
 - An event whose supported grid is incomplete, unless every inert or
-  missing and supported measure-year in it is a known gap. An incomplete
-  event whose receipt does not list its missing measure-years fails, and so
-  does one with no inert and no missing and supported measure-year at all.
-  A receipt whose integrity is `incomplete` with no incomplete event also
-  fails.
+  missing and supported measure-year in it is a known gap or is accounted
+  for by a data-limited exception. An incomplete event whose receipt does
+  not list its missing measure-years fails, and so does one with no inert
+  and no missing and supported measure-year at all. A receipt whose
+  integrity is `incomplete` with no incomplete event also fails.
 - A receipt that contradicts itself, including a complete event that lists
   an inert, or missing and supported, measure-year; flag counters that
   differ, either way, from the flags its rows carry; and a candidate value
@@ -66,6 +74,80 @@ even £0, is still a lost result, and any candidate value other than £0 at a
 known gap contradicts the candidate's own record, which lists the
 measure-year as inert (exactly £0) or missing (no value).
 
+## Data-limited exceptions
+
+`uk_replay_exceptions.json` lists narrowly scoped exceptions for regressions
+caused by inputs absent from the candidate's data. Missingness alone does
+not establish cause. The registry is reviewed like code. Each entry records
+an id, failure codes, event, measure key, years, required missing inputs,
+cause, tracking issue, approved lineage, positive-input probe specification,
+publication designation, governing decision and review status.
+
+An exception applies only when its code, event, measure key and year all
+match, and the same qualification attempt's `budget_inputs` gate result
+and its evidence report every required input as missing from the
+candidate's data. A degenerate but present input cannot satisfy that
+condition. The same evidence must carry the cumulative `ever_usable_inputs`
+set from the previous certified pair, adding inputs usable in that pair and
+in the candidate. Removing an input or making it formula-backed never erases
+this history. An exception requires that the input has never been usable in
+this chain and is not usable now. For evidence predating the cumulative field,
+recompute the exact direct predecessor's usable inputs and record
+`ever_usable_inputs_basis: "seeded from direct predecessor"`; the registry's
+approved lineage records the human-verified history before that base.
+A previous pair with both the variable
+and column present requires a fresh reviewed registry decision and fails,
+including when the previous column was degenerate. The approved
+SDLT lineage is that no certified uk-data release through 1.58.2 supplied
+the input. When the data gains an input, or the measure-year falls outside
+the entry's years, the exception stops applying automatically.
+
+Every reported fiscal effect for an excepted measure-year must be exactly £0,
+including fiscal heads, measure totals, source rows and any other reported
+level. Any nonzero candidate effect is a contradiction and fails,
+independently of any base known gap. This checks reported rows only.
+
+The gates job also runs the registry's positive-input probe directly on the
+exact installed candidate model, using one synthetic household and no
+survey data. For the SDLT exception, an England household buys an additional
+residential property for £300,000 in 2026. The reform adds 0.02 to every
+band of `gov.hmrc.stamp_duty.residential.purchase.additional.rate`. Its
+`sdlt_on_residential_property_transactions` output must increase by £6,000
+within £0.10, and must be exactly unchanged when the purchase input is £0.
+An import error, missing variable or parameter, or any other inability to
+run the probe records failure. An exception requires a passing probe from
+the same attempt (identified by both pair id and attempt key), bound to the
+candidate model version and probe specification by digests. Finalization
+re-reads the evidence and verifies its digest, as it does for `budget_inputs`.
+The consumer also checks the installed model version and verifies that the
+recorded baseline/reform outputs yield the claimed positive and zero deltas.
+Budget-input evidence embeds and checks both pair and attempt identity; its
+candidate status map must agree with the missing list, and its previous
+status flags and cumulative history must be internally consistent.
+
+Matching `construction_inert` integrity failures and `lost_response`
+regressions (candidate inertness or a measure-year total of £0 against a
+material base value) become review items. Each names the exception, cause
+and required missing inputs. A failed receipt proceeds to the ordinary
+corpus and diff checks only if every integrity failure matches an
+exception. Every unmatched failure, mapping flag, lost result and
+contradiction retains its ordinary treatment. An exception accounts only
+for its matched inert measure-years when checking grid completeness; it
+does not excuse a missing result.
+
+An applied exception always requires review and can never qualify a
+candidate automatically. Promotion requires a named reviewer to accept
+the exact qualification receipt. Applied-exception review items carry the
+registry's publication designation, and the named reviewer's acceptance
+explanation must restate it. For SDLT the designation is: not computable,
+in PolicyEngine/uk-autumn-budget-dashboard-2026 (data-limited handling,
+PR #51; Budget-day runbook, PR #14).
+
+The gate records every applied exception's id, event, measure key, year
+and matched codes, the registry's SHA-256, and the digests of the same
+attempt's `budget_inputs` and probe results it relied on. Every result
+that reads the registry records its digest.
+
 ## Review
 
 A needs-review verdict blocks automatic promotion. It stays blocked until a

src/policyengine/data/qualification/uk_replay_exceptions.json

@@ -0,0 +1,42 @@
+{
+  "_comment": "Narrow data-limited exceptions for the UK replay qualification gate, reviewed like code under cos d1344. An entry requires coherent zero candidate effects, never-usable input lineage from budget_inputs, and a passing positive-input probe from the same attempt. It always requires a named reviewer's acceptance. The 17 October 2026 decision is adopted by the Budget coordinator following independent review agreement under cos d1344: RULE AGREE (GPT-6.1 Sol, round 2, 17 October 2026, head 7f518ff61) and CODE AGREE WITH NITS (round 3, head dbc1806df), with the P3 nit fixed in this commit.",
+  "exceptions": [
+    {
+      "id": "ab2024-sdlt-hrad-purchase-input",
+      "codes": [
+        "construction_inert",
+        "lost_response"
+      ],
+      "event": "autumn_budget_2024",
+      "measure_key": "autumn_budget_2024__sdlt_additional_dwelling_surcharge_2pp",
+      "years": [
+        2025,
+        2026,
+        2027,
+        2028,
+        2029
+      ],
+      "requires_missing_inputs": [
+        "additional_residential_property_purchased"
+      ],
+      "cause": "policyengine-uk#2241 charges SDLT higher rates on purchases, which uk-data does not yet impute.",
+      "issue": "PolicyEngine/policyengine-uk-data#573 (cos d1354)",
+      "approved_lineage": "never supplied by any certified uk-data release up to and including 1.58.2.",
+      "publication": "not computable, in PolicyEngine/uk-autumn-budget-dashboard-2026 (data-limited handling, PR #51; Budget-day runbook, PR #14).",
+      "probe": {
+        "input_variable": "additional_residential_property_purchased",
+        "input_entity": "household",
+        "year": 2026,
+        "positive_input_gbp": 300000,
+        "output_variable": "sdlt_on_residential_property_transactions",
+        "parameter": "gov.hmrc.stamp_duty.residential.purchase.additional.rate",
+        "rate_increment": 0.02,
+        "expected_change_gbp": 6000,
+        "absolute_tolerance_gbp": 0.1,
+        "region": "LONDON"
+      },
+      "decided_under": "cos d1344; Budget coordinator's data-limited exception proposal of 17 October 2026",
+      "review": "Adopted by the Budget coordinator following independent review agreement under cos d1344: RULE AGREE (GPT-6.1 Sol, round 2, 17 October 2026, head 7f518ff61) and CODE AGREE WITH NITS (round 3, head dbc1806df), with the P3 nit fixed in this commit."
+    }
+  ]
+}