[JianhaoYu (JianhaoYu-AI)]

Summary

Add an offline checker for split manifests, saved per-item scores, and evaluation summaries. It checks the supplied files without model calls or evaluator reruns.

Related Issue

Related to #158 and the offline calculation requested in #326. This leaves the historical-artifact questions open and does not change #176's LiveMath metadata fix.

Motivation

A split size alone does not establish how a reported score was calculated. Hard scores can be continuous, and filtering or averaging across runs can change the denominator. The checker compares saved results only when the input files and aggregation protocol provide enough evidence.

Changes

• Report the checkout SHA, declared dataset/run revisions, and file hashes; check split counts, duplicate IDs, overlap, and result coverage.
• Reuse compute_score for hard/soft means, including failed result rows. Compare summaries with explicit scale and rounding precision.
• Separate RECORDED historical/new runs, RECONSTRUCTED examples, and MISSING evidence. Unknown protocols withhold the numerical verdict; available hash bindings are checked independently.
• Add text/JSON output, a small synthetic fixture, 60 regression cases, and usage documentation. No dependencies, released data, or training/evaluation defaults change.

Validation

Windows, pytest 9.1.1. Focused tests: 75 passed on both Python 3.11.9 and 3.14.5.

Full suite	This branch	Clean upstream baseline
Python 3.11.9	1826 passed, 20 failed, 61 skipped	1766 passed, 20 failed, 61 skipped
Python 3.14.5	1817 passed, 29 failed, 61 skipped	1757 passed, 29 failed, 61 skipped

Baselines use 1a44050fc53aa4938a7a3927b6cca550491bfe51. Failed test identifiers match in each environment; no added failures were observed. The full suites are not green locally. Each full run also reports 363 passing subtests. One additional harness failure in the first Python 3.11 baseline run passed both its isolated rerun and the serial full-baseline confirmation.

Ruff, git diff --check, and strict MkDocs builds passed. All six released manifests have consistent counts and no duplicate or overlapping IDs; their historical run evidence remains MISSING. Opt-in live backend tests were disabled. Python 3.10/3.12 and Linux CI have not been run locally.

Reproduction Instructions

From the repository root:

python -m scripts.check_reproducibility tests/fixtures/reproducibility/split_manifest.json --results tests/fixtures/reproducibility/results.jsonl --summary tests/fixtures/reproducibility/eval_summary.json --evidence tests/fixtures/reproducibility/evidence.json
python -m pytest -q tests/test_reproducibility.py tests/test_scoring.py tests/test_data_manifests.py

The fixture gives hard=0.5 and soft approximately 0.6. Its aggregate comparison passes, while the overall status is MISSING (exit 2) because the records are explicitly RECONSTRUCTED. Use --format json for structured output. The input schema and precision options are documented in docs/guide/offline-reproducibility.md.

Limitations

This checks saved-score aggregation, not model-answer correctness, raw dataset payloads, or original runtime settings. Hashes bind declarations to files; they do not authenticate historical origin. Multi-run, weighted, filtered, grouped, and subset protocols are not inferred. The checker cannot prove missing historical results or supply the original records, source snapshots, skills, and effective configurations requested in #326.

[JianhaoYu (JianhaoYu-AI)]

@microsoft-github-policy-service agree

[Yifan Yang (Yif-Yang)]

Reviewed 0014cf408a87051fe9dfa2542b67a0050ba6b032. This is useful, well-scoped offline tooling: it preserves error rows, supports continuous hard scores, requires explicit aggregation/precision, and correctly distinguishes declarations from authentication. The synthetic example and new runs do not establish historical results. Linux validation passed all 75 focused tests and the full suite (1889 passed, 13 skipped); the review baseline 1a44050fc53a passed 1829 tests with the same 13 skips.

Two additional partial-input cases still need correction. If the manifest is missing, the early return skips the evidence and otherwise available files, hiding a known supplied digest mismatch as MISSING. If results are missing, an available, correctly hash-bound but malformed summary is never schema-checked and also remains MISSING. These are diagnostic omissions, not false PASS results, but they undermine the promised independent integrity checks and malformed-input reporting. Please validate each available artifact independently, preserve known failures, and withhold only the aggregate comparison when its inputs are incomplete.

The separate 26-case independent matrix passed 24 cases and reproduced those two failures. No supplied input bytes were modified. Please keep #158 and #326 open: this checker does not provide the original evaluation records, source/Skill snapshots, or effective runtime configurations.

Review baseline: 1a44050fc53a. The PR remains open; no merge was performed.