Repository navigation
Read scorecard#183's base-side grid record and apply known gaps - #637
Merged
Merged
Conversation
scorecard#183 merged as 71cad306: every replay receipt now carries the base's own grid record (diff.base_events) and the candidate's missing and unsupported measure-years. The vendored schema and the mirrors are closed, so until now every such receipt was refused. They now match 71cad306, including its tightened measure-year lists (a non-empty measure_key and unique items; supported years were already unique in the schema but not in the mirror). The replay gate applies the decision record's exact test: a candidate measure-year that is inert, or missing and supported, is a known gap only if the base's record for that event lists the same measure and year as inert, or missing and supported. Known gaps are recorded with the base's staging manifest digest. An incomplete event fails unless every gap in it is known, and also fails when its receipt does not list its missing measure-years or lists none. A complete event that lists a gap contradicts itself. A known gap never excuses a GBP 0 total against a material base value, which the base's inert record could not hold. The policy text states the rule; its digest changes accordingly. Eight mutants are killed, among them ignoring the base's unsupported list, matching another event's record, ignoring the year, and letting a known gap excuse a material base value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 1 of the review found the mirrors accepted documents the schema rejects: null for the new grid lists or for diff.base_events, and a string year coerced to an integer, which could establish a known gap. A received receipt (ReplayReceipt.from_document) and status (read_replay) must now satisfy the vendored scorecard schema as well as the mirror; a refusal is a ValueError, so the poll step fails closed and the attempt stays retriable rather than being finalized on bad evidence. Five such documents join the schema/mirror differential cases. The round's eight probes, which killed mutants the table missed (unsupported filtering, measure identity, mixed known and unknown gaps, cross-kind matching, negative and exact-bound zero totals), are now tests, and each kills its mutant. The policy text says an incomplete event fails when it lists no inert and no missing-and-supported measure-year at all, as the code does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to w4/replay-known-gaps
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions Round 2 of the review found that a measure-year the candidate lists as inert excused its whole flagged row: with a known inert gap, a row flagged inertness and construction passed with no review. The inert list now decides only the inertness flag; the row's other flags are judged as usual. The round's seven probes are now tests, each killing its mutant: a known gap never excuses a complete event listing a gap, a counted mapping flag or a lost result; base evidence never crosses into another event, for inert or missing measure-years; a known missing gap is recorded as missing and never hides another missing measure-year. A schema-invalid replay status (an explicit null the mirror accepts) is refused when read, killing the mutant that drops the status check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to w4/replay-known-gaps
The 1.58.0 report predates the tool's small-cell rule and published Disabled Students' Allowance's total and caseload, which come from fewer than 10 records, unsuppressed; totals containing it gave it back. The 1.58.1 report at tool commit 228a05a3e publishes those figures as suppressed and rebuilds every total without them. The gate and tests that read the report move to the suppression format in the next commit. History is left as is pending a security decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3e pin Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… rows Round 3 of #637 found that a known gap where the candidate is missing and the base inert always failed: scorecard records it as a coverage row with the base's £0 and no candidate value (a "lost result") plus a base-only inertness flag, and records the inert/missing mirror as a £0 coverage gain. The d1344 decision says a known gap does not block. - At a known gap, a coverage row with exactly £0 on the inert side and nothing on the other, and (candidate missing, base inert) the base-only inertness flag, are the gap itself: neither fail nor review. The gate records them as known_gap_rows. Any other row at a known gap is judged as usual: a material base value is still a lost result, and a candidate value contradicts the candidate's record. - flag_counts must equal the flags the rows carry, either way (equal on all three real receipts); a counter with no row is a contradiction, which retires the counted_flag kind. - replay_policy.md says both, with the revision dated. - Tests: the four producer-shaped known-gap kinds, other coverage rows at a gap, the reviewer's 18 round-3 probes (one adapted to the revised rule), a Hypothesis property against an independent restating of the policy with known gaps, and the real 6.5.0 receipt (first with base_events): needs_review, 8 flagged moves, all AB2024 CGT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… evidence Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rt values, signed exclusions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 4 of #637: - The policy says any candidate value at a known gap contradicts the candidate's record, but the code checked only coverage rows; at a missing/inert gap, an inertness row carrying a £500m candidate value passed. Any row at a known gap with a candidate value other than £0 now fails as a contradiction (scorecard cannot write one: an inert artifact's effects are exactly £0 and a missing one has none). - The missing/inert base-only inertness strip applies only to a row with no base value, the shape scorecard writes. - replay_policy.md: the inert side's flag on a valueless row, rows accounted for in whole or in part, and "any base value other than exactly £0" for a lost result. - Tests: the reviewer's round-4 probes (the gap's event, measure and year; exactly £0; partly excused rows; three gaps at once), six candidate-value cases, and the reference property updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ew records Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 5 of #637: - The missing/inert base-only inertness strip also excused rows with £0 on either side, though the policy text and scorecard (which flags inertness only on a row with no value either side) say a valueless row. It now needs no value on either side. - replay_policy.md said a base of exactly £0 with no candidate value is not a lost result; the code fails it at every known gap except the missing/inert gap's own record. The text now says so. The fail list says "a candidate value other than £0", the revision note covers rounds 3 to 5, and the gate's docstring matches. - Tests: the reviewer's round-5 probes (valued inertness rows, the coverage record carrying inertness, partly excused and partly stripped rows, negative candidate values) and the text probes, with the Hypothesis reference updated to the valueless rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hape Round 6 of #637 (approved; lows): - The docstring said any base value with no candidate value is a lost result; only a coverage row is, as the code and policy say. It also read as if a known gap excused mapping flags and the £0-total lost response, which it never does. Both corrected. - replay_policy.md's summary now says the gate passes the rows that record a known gap. - Tests (the reviewer's probes): a valueless inertness row at a missing/inert gap is the gap's record at every level; a coverage row with no value on either side at a mixed gap is not its record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
October 16, 2026 19:18
Contributor
Author
|
Merged at 00627f3 with a merge commit. Gates: 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #636
Stacked on #635, which is stacked on #574. It merges after both.
policyengine-scorecard#183 merged as 71cad306. Every replay receipt now carries:
diff.base_events.<event>, the comparison base's own grid record, taken from the base's verified staging and run manifests;missing_measure_yearsandunsupported_measure_yearslists.Our vendored schema and mirrors were closed, so until this lands every new receipt is refused and nothing can qualify.
Change
replay_receipt.v1.jsonfrom scorecard main at 71cad306. The README records the commit; the request and status schemas are unchanged.MeasureYearreplacesInertMeasureYear, which remains as an alias, and now requires a non-emptymeasure_key.EventResultgains the optional grid lists, andDiffgains the optionalbase_events, of the new typeBaseEvent.base_events.<event>exists and lists the same pair as inert, or as missing and supported.replay_policy.mdstates the rule. Its digest changes; the policy id staysd1344-replay-regressions-v1, since this implements the agreed rule.Invariants and tests
axiom: n/a: qualification CI infrastructure, no policy encoded.
🤖 Generated with Claude Code