LiveMath GPT-5.5 66.9%: request original per-item results, code snapshot, and effective run configuration #326

The 66.9% result remains unexplained after the split correction

The [paper, arXiv:2605.23904v2](https://arxiv.org/html/2605.23904v2), reports 37.6% → 66.9% for GPT-5.5 on LiveMathematicianBench in direct-chat mode.

In [#158](#158), the authors clarified that the paper used 125 test items, and corrected the released split from 35/18/124 to 35/17/125.

However, 66.9% cannot result from a single, unweighted mean of 125 binary outcomes, rounded to one decimal place:

• 83 / 125 × 100 = 66.4%
• 84 / 125 × 100 = 67.2%

This is consistent with the released [binary evaluator](https://github.com/microsoft/SkillOpt/blob/ae2f766ea02cb41855b6f8f10e38e10264ce0a1b/skillopt/envs/livemathematicianbench/evaluator.py), [mean aggregation](https://github.com/microsoft/SkillOpt/blob/ae2f766ea02cb41855b6f8f10e38e10264ce0a1b/skillopt/utils/scoring.py), and [125-item manifest](https://github.com/microsoft/SkillOpt/blob/ae2f766ea02cb41855b6f8f10e38e10264ce0a1b/data/livemathematicianbench_id_split/split_manifest.json) at commit ae2f766ea02cb41855b6f8f10e38e10264ce0a1b.

For completeness, 83/124 does round to 66.9%, but that would conflict with the stated 125-item protocol. Switching both results to 124 items does not reconcile them either: 37.6% is attainable with 125 binary items, but not with 124.

This is therefore not simply a repetition of the already-corrected split-metadata issue.

Requested artifacts for this specific result

1. Original per-item evaluation records and exact calculation

Please provide machine-readable records, preferably CSV or JSONL, containing:

• Test-item ID and original run ID.
• Model output, extracted answer, and per-item score.
• Completion/error status, including failed or excluded items and reasons.
• The exact numerator, effective denominator, aggregation formula, and rounding procedure producing 66.9%.

Please include every scheduled item, not only successful requests.

If this cell used averaging across runs, categories, or other groups, please provide the component results, denominators, and weights. The response to the baseline-protocol questions in [#108](#108) described a single fixed-seed run; please clarify whether this cell followed that protocol or used a different one.

2. The original code and Skill artifact

Please identify the actual training and evaluation versions used for this result, including:

• Commit IDs or archived source snapshots, including any uncommitted changes.
• The evaluated Skill file and its digest.
• The run/checkpoint selected for the reported cell.
• The effective slow-update gating setting.

[#22](#22) states that the exact gated snapshot used for the paper was not included in the initial release. A subsequently added configuration switch does not, by itself, establish which original code produced this result.

Please distinguish original archived artifacts from later reconstructions or reruns.

3. Actual runtime configuration, including reasoning/thinking

Please provide the resolved configuration and a sanitized example of the actual request parameters, rather than only current repository defaults:

• Target and optimizer model versions/snapshots, separately.
• GPT-5.5 reasoning settings, including reasoning_effort, whether it was explicitly sent, and relevant provider defaults or overrides.
• Output/reasoning token limits, sampling parameters, seeds where applicable, retries, and response-cache behavior.
• Dataset revision, test-item list, and scorer version.

Please also provide the corresponding settings for the 37.6% no-skill baseline, identifying any differences beyond Skill injection.

This is a request to verify matched conditions, not an assumption that the two runs used different thinking settings.

4. An offline reproduction of the reported number

Please provide a script or command that reads the original records and reproduces the reported aggregate without making new model calls, with the counted and excluded items explicitly reported.

What would resolve this issue?

A general statement that “the split was corrected,” “the paper used the gated version,” or “thinking was disabled” does not explain this numerical inconsistency.

Please provide an artifact-backed calculation of 66.9%. If the number is a reporting error, please publish the corrected value and supporting records. If the original records or code snapshot are unavailable, please state that explicitly rather than presenting a new run as verification of the historical result.