Skip to content

Require review, not failure, for narrowly matched data-limited UK replay regressions - #654

Draft
MaxGhenis wants to merge 6 commits into
mainfrom
fix/replay-data-limited-exception
Draft

MaxGhenis wants to merge 6 commits into
mainfrom
fix/replay-data-limited-exception

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Fixes #653

Proposed, not yet adopted. Under cos d1344's delegation, this rule change takes effect only after an independent review agrees with both the decision record and the code. That review is running. Max can veto before merge.

Why

#2241 correctly charges SDLT higher rates on purchases, but no uk-data release has the purchase input yet (policyengine-uk-data#573). So every candidate since then replays the AB2024 additional-dwelling surcharge as £0 for 2025–2029, against £2.50bn under the old stocks-as-purchases bug. The gate fails any receipt whose integrity failed, and d1344 bars accepting it on review. So no pe-uk after #2241 can be certified, which would leave the 28 October Budget on 2.130.1, without #2240, #2242, #2428, #2436, #2437 and the pension fixes in review.

Change

  • Registry: src/policyengine/data/qualification/uk_replay_exceptions.json, with one entry: ab2024-sdlt-hrad-purchase-input, covering construction_inert and lost_response for autumn_budget_2024__sdlt_additional_dwelling_surcharge_2pp, 2025–2029, while additional_residential_property_purchased is missing.
  • Narrow match: code, event, measure key and year must all match, and the same attempt's digest-bound budget_inputs evidence must report every required input missing. A present-but-degenerate input doesn't authorise it.
  • Failed receipts: evaluation continues only when every integrity failure is a matched construction_inert and the execution identity, corpus and sealed results exist. No vendored schema or validator is weakened.
  • Verdict: matched regressions become needs_review items, never qualified; promotion still needs a named reviewer's acceptance of the exact receipt.
  • Self-expiry: once the data has the input, the measure must move, or it fails as before.
  • Recording: the applied measure-years and codes, the registry SHA-256 and the input-result digest are kept in the gate evidence. Finalization re-evaluates the saved replay bytes against the current attempt inputs.
  • Policy: replay_policy.md gains a "Data-limited exceptions" section, and the id moves to d1344-replay-regressions-v2. No scorecard-facing request embeds the id.

Invariants (each held by tests in tests/test_replay_exceptions.py)

  1. An applied exception never produces qualified (Hypothesis).
  2. An unmatched code, event, key, year or input status, or an extra integrity failure, still fails (Hypothesis, plus partial-subset tests).
  3. With an empty registry, results equal main's. This is a frozen-main differential over every replay evaluation in the 314-test contracts suite; only the new v2 metadata and digest fields are normalised.
  4. Adding a regression never improves a verdict (Hypothesis).
  5. Determinism of verdicts and recorded digests (Hypothesis).
  6. The real 2.134.7 receipt gives needs_review with exactly 5 applied measure-years and 195 other flagged moves left for review; without the missing-input evidence, or with an extra failure, it fails.

Mutation: 9 of 9 executed mutants are killed, including dropping the missing-input condition, matching any year, accepting any rather than all failures, and letting the exception pass.

Verification

1,037 tests pass (CPython 3.13, frozen UK and dev extras, serial). Ruff check and format pass on the changed files; repo-wide lint already fails on main with 68 UP038 findings.

Built on a Subfleet lane (GPT-6.1 Sol). Decision record: ~/reviews/budget-coordinator/replay-data-limited-exception/decision.md on the coordinator's machine; its rule is reproduced in replay_policy.md.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 17, 2026 07:42
Allow the registered SDLT purchase-input losses to reach named review only
when the same attempt's digest-bound budget-input evidence reports all
required inputs missing. Preserve unmatched failures and contradictions,
reevaluate cached replay results at finalization, and record the registry,
matched measure-years and input-result digest.

Move the decision policy to d1344-replay-regressions-v2: the scorecard
request contract does not embed the policy id, and gate details are open.
Add real-receipt, property, frozen-main differential and mutation coverage,
retaining the proposal's pending review status in registry metadata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Independent review round 1 (GPT-6.1 Sol, head 27b42d4): RULE: AGREE WITH CHANGES, CODE: DISAGREE. Two P2 findings, both accepted:

  1. Contradictory receipts became reviewable: an excepted SDLT measure-year reporting £1bn while claimed inert returned needs_review. The fix requires a £0 candidate effect for every excepted measure-year.
  2. Missing data isn't causal proof, and expiry could reverse: a dataset that later loses a supplied input would reactivate the exception, and a model bug could hide behind missingness. The fix adds a lineage rule (the previous certified pair must never have had the input usable), a positive-input probe on the exact candidate model (synthetic household; the reform must move SDLT with the input set and leave it unchanged at zero), and a publication designation (not computable).

The decision record now has these as items 7–10, and round 2 is building. This stays a draft until a review agrees with both the rule and the code.

MaxGhenis and others added 4 commits October 17, 2026 09:48
Reject contradictory effects and previously supplied input loss. Require exact-attempt candidate probes, record publication handling, and revalidate evidence at finalization. Cover the independent review reproducers, properties, real-model probes and mutation guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject nonzero effects at every reported fiscal level before applying an
exception. Persist cumulative usable-input history through certified pairs,
seed legacy evidence explicitly, and fail closed when recorded history is
unavailable or inconsistent.

Bind budget-input evidence to its pair and attempt, validate status maps,
and cross-check installed probe versions and raw baseline/reform deltas.
Convert the round-2 reproducers and add lineage, boundary, property and
mutation regression coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject rehashed cumulative histories that contradict the budget-input
gate-result marker. Preserve both reviewer reproducers as rejection tests
and kill a mutant that skips the marker check. Record Budget coordinator
adoption and the independent RULE/CODE review agreement under cos d1344.

Validation: requested suites passed 937 tests with three model-version skips;
all three real-model checks then passed on policyengine-uk 2.134.7. Eight
new cases pass, including the marker-check mutant kill. Ruff lint and format
checks pass on all five changed Python files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Rounds 3–4 and merge status:

This merges with a merge commit once CI is green at 0ef4a12.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UK replay gate: a data-limited inert measure blocks certification of every pe-uk release after #2241

1 participant