Skip to content

Read scorecard#183's base-side grid record and apply known gaps - #637

Merged
MaxGhenis merged 28 commits into
mainfrom
w4/replay-known-gaps
Oct 16, 2026
Merged

MaxGhenis merged 28 commits into
mainfrom
w4/replay-known-gaps

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Merge hold: cos d1613 (security). Until 16 Oct 2026 this branch's history carried a pre-suppression aggregate-impact report (tests/fixtures/uk_metrics/budget_metrics_report.json, blobs c79f1a06 from 3ea0b19 and 6b599fb5 from b424ffb). That report published Disabled Students' Allowance, which comes from fewer than 10 records, unsuppressed. The tip now carries W3's suppressed 1.58.1 report at tool commit 228a05a3e. Max is deciding whether to rewrite the history. Don't merge, force-push or delete anything for it until d1613 is ruled.

Fixes #636

Stacked on #635, which is stacked on #574. It merges after both.

policyengine-scorecard#183 merged as 71cad306. Every replay receipt now carries:

  • diff.base_events.<event>, the comparison base's own grid record, taken from the base's verified staging and run manifests;
  • the candidate's missing_measure_years and unsupported_measure_years lists.

Our vendored schema and mirrors were closed, so until this lands every new receipt is refused and nothing can qualify.

Change

  • Vendored schema: replay_receipt.v1.json from scorecard main at 71cad306. The README records the commit; the request and status schemas are unchanged.
  • Mirrors:
    • MeasureYear replaces InertMeasureYear, which remains as an alias, and now requires a non-empty measure_key.
    • The measure-year lists and supported years must have unique items, as the schema says. The supported-years rule was already in the schema but missing from the mirror.
    • EventResult gains the optional grid lists, and Diff gains the optional base_events, of the new type BaseEvent.
  • The known-gap test from the d1344 decision record, exactly:
    • A candidate measure-year that is inert, or missing and supported, is a known gap only if base_events.<event> exists and lists the same pair as inert, or as missing and supported.
    • Each known gap is recorded with the base's staging manifest SHA-256.
    • With no entry (a legacy base, or an event the base did not replay), no gap is known.
  • Stricter around it:
    • An incomplete event fails unless every gap in it is known. It also fails if its receipt doesn't list its missing measure-years, or lists none.
    • A complete event that lists a gap contradicts itself.
    • A known gap never excuses a £0 total against a material base value, since a base with an inert record can't hold one.
  • Policy text: replay_policy.md states the rule. Its digest changes; the policy id stays d1344-replay-regressions-v1, since this implements the agreed rule.

Invariants and tests

  • Schema/mirror differential: seven new rejection cases, such as an empty key, a duplicate measure-year, a duplicate year, a base event missing a field, a short digest, or an extra field. One acceptance case shows a valid record round-trips with its digest.
  • Known-gap table: inert in both, missing in both, inert here but missing in the base, the base computed it, missing but unsupported in the base, a legacy base, another event's record, another year, and nothing listed.
  • Two edge tests: a receipt from before Enhance entity mapping with flexible aggregation methods #183 fails even with base evidence, and a known gap never excuses a material base value.
  • The existing property test still holds: the gate fails exactly when a regression is recorded, no flagged row passes, and adding a row never improves the verdict.
  • The first real receipt (6.4.1) still fails on its four inert UC measure-years. It predates Enhance entity mapping with flexible aggregation methods #183, so it has no base record.
  • Mutation: eight mutants are killed.
  • Qualification suites: 444 passed, 52 skipped.

axiom: n/a: qualification CI infrastructure, no policy encoded.

🤖 Generated with Claude Code

MaxGhenis and others added 27 commits October 15, 2026 23:19
scorecard#183 merged as 71cad306: every replay receipt now carries the
base's own grid record (diff.base_events) and the candidate's missing
and unsupported measure-years. The vendored schema and the mirrors are
closed, so until now every such receipt was refused. They now match
71cad306, including its tightened measure-year lists (a non-empty
measure_key and unique items; supported years were already unique in
the schema but not in the mirror).

The replay gate applies the decision record's exact test: a candidate
measure-year that is inert, or missing and supported, is a known gap
only if the base's record for that event lists the same measure and
year as inert, or missing and supported. Known gaps are recorded with
the base's staging manifest digest. An incomplete event fails unless
every gap in it is known, and also fails when its receipt does not
list its missing measure-years or lists none. A complete event that
lists a gap contradicts itself. A known gap never excuses a GBP 0 total
against a material base value, which the base's inert record could not
hold. The policy text states the rule; its digest changes accordingly.

Eight mutants are killed, among them ignoring the base's unsupported
list, matching another event's record, ignoring the year, and letting a
known gap excuse a material base value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 1 of the review found the mirrors accepted documents the schema
rejects: null for the new grid lists or for diff.base_events, and a
string year coerced to an integer, which could establish a known gap.
A received receipt (ReplayReceipt.from_document) and status (read_replay)
must now satisfy the vendored scorecard schema as well as the mirror;
a refusal is a ValueError, so the poll step fails closed and the
attempt stays retriable rather than being finalized on bad evidence.
Five such documents join the schema/mirror differential cases.

The round's eight probes, which killed mutants the table missed
(unsupported filtering, measure identity, mixed known and unknown gaps,
cross-kind matching, negative and exact-bound zero totals), are now
tests, and each kills its mutant. The policy text says an incomplete
event fails when it lists no inert and no missing-and-supported
measure-year at all, as the code does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions

Round 2 of the review found that a measure-year the candidate lists as
inert excused its whole flagged row: with a known inert gap, a row
flagged inertness and construction passed with no review. The inert
list now decides only the inertness flag; the row's other flags are
judged as usual.

The round's seven probes are now tests, each killing its mutant: a known
gap never excuses a complete event listing a gap, a counted mapping flag
or a lost result; base evidence never crosses into another event, for
inert or missing measure-years; a known missing gap is recorded as
missing and never hides another missing measure-year. A schema-invalid
replay status (an explicit null the mirror accepts) is refused when
read, killing the mutant that drops the status check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 1.58.0 report predates the tool's small-cell rule and published
Disabled Students' Allowance's total and caseload, which come from
fewer than 10 records, unsuppressed; totals containing it gave it back.
The 1.58.1 report at tool commit 228a05a3e publishes those figures as
suppressed and rebuilds every total without them.

The gate and tests that read the report move to the suppression format
in the next commit. History is left as is pending a security decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3e pin

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… rows

Round 3 of #637 found that a known gap where the candidate is missing
and the base inert always failed: scorecard records it as a coverage
row with the base's £0 and no candidate value (a "lost result") plus a
base-only inertness flag, and records the inert/missing mirror as a £0
coverage gain. The d1344 decision says a known gap does not block.

- At a known gap, a coverage row with exactly £0 on the inert side and
  nothing on the other, and (candidate missing, base inert) the
  base-only inertness flag, are the gap itself: neither fail nor
  review. The gate records them as known_gap_rows. Any other row at a
  known gap is judged as usual: a material base value is still a lost
  result, and a candidate value contradicts the candidate's record.
- flag_counts must equal the flags the rows carry, either way (equal on
  all three real receipts); a counter with no row is a contradiction,
  which retires the counted_flag kind.
- replay_policy.md says both, with the revision dated.
- Tests: the four producer-shaped known-gap kinds, other coverage rows
  at a gap, the reviewer's 18 round-3 probes (one adapted to the
  revised rule), a Hypothesis property against an independent restating
  of the policy with known gaps, and the real 6.5.0 receipt (first with
  base_events): needs_review, 8 flagged moves, all AB2024 CGT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… evidence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rt values, signed exclusions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 4 of #637:

- The policy says any candidate value at a known gap contradicts the
  candidate's record, but the code checked only coverage rows; at a
  missing/inert gap, an inertness row carrying a £500m candidate value
  passed. Any row at a known gap with a candidate value other than £0
  now fails as a contradiction (scorecard cannot write one: an inert
  artifact's effects are exactly £0 and a missing one has none).
- The missing/inert base-only inertness strip applies only to a row
  with no base value, the shape scorecard writes.
- replay_policy.md: the inert side's flag on a valueless row, rows
  accounted for in whole or in part, and "any base value other than
  exactly £0" for a lost result.
- Tests: the reviewer's round-4 probes (the gap's event, measure and
  year; exactly £0; partly excused rows; three gaps at once), six
  candidate-value cases, and the reference property updated to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ew records

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 5 of #637:

- The missing/inert base-only inertness strip also excused rows with £0
  on either side, though the policy text and scorecard (which flags
  inertness only on a row with no value either side) say a valueless
  row. It now needs no value on either side.
- replay_policy.md said a base of exactly £0 with no candidate value is
  not a lost result; the code fails it at every known gap except the
  missing/inert gap's own record. The text now says so. The fail list
  says "a candidate value other than £0", the revision note covers
  rounds 3 to 5, and the gate's docstring matches.
- Tests: the reviewer's round-5 probes (valued inertness rows, the
  coverage record carrying inertness, partly excused and partly stripped
  rows, negative candidate values) and the text probes, with the
  Hypothesis reference updated to the valueless rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hape

Round 6 of #637 (approved; lows):

- The docstring said any base value with no candidate value is a lost
  result; only a coverage row is, as the code and policy say. It also
  read as if a known gap excused mapping flags and the £0-total lost
  response, which it never does. Both corrected.
- replay_policy.md's summary now says the gate passes the rows that
  record a known gap.
- Tests (the reviewer's probes): a valueless inertness row at a
  missing/inert gap is the gap's record at every level; a coverage row
  with no value on either side at a mixed gap is not its record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis changed the base branch from w4/d1344-automatic-promotion to main October 16, 2026 15:45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 16, 2026 19:18
@MaxGhenis
MaxGhenis merged commit 4a3a078 into main Oct 16, 2026
12 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged at 00627f3 with a merge commit. Gates: gh pr checks exit 0 (12/12 pass, run against main after #635), mergeable, no change requests. Independent review (GPT-6.1 Sol): round 6 APPROVE at 3decffb and confirmation APPROVE at 656eede of this PR's own commits. Every later commit is a merge of #635 (reviewed separately, 13 rounds) or of main; the merged tree passed 1,724 tests locally before the last push. Replay gate policy: coordinator's call under cos d1344, with independent review agreement.

🤖 Generated with Claude Code

@MaxGhenis
MaxGhenis deleted the w4/replay-known-gaps branch October 16, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vendor scorecard#183's base-side grid record and apply known gaps in the replay gate

1 participant