Repository navigation
Several reported scores are incompatible with the released test split sizes #158
Description
Hi, thanks for releasing SkillOpt. I found a possible mismatch between the paper results and the currently released dataset splits.
The released code computes the hard score as the mean of per-example scores. For LiveMath, the evaluator is binary exact match, and the released manifest contains 124 test items:
data/livemathematicianbench_id_split/split_manifest.jsonskillopt/envs/livemathematicianbench/evaluator.pyskillopt/utils/scoring.py
However, several LiveMath results in Table 1 cannot be obtained from 124 binary examples after rounding to one decimal:
| Score | Implied result |
|---|---|
| 22.4 | 28/125 |
| 27.2 | 34/125 |
| 52.0 | 65/125 |
| 31.2 | 39/125 |
| 29.6 | 37/125 |
| 41.6 | 52/125 |
These values all fall exactly on a 125-item grid, suggesting that some experiments may have used a 125-item test split rather than the released 124-item split.
I found a similar issue for DocVQA. Its released test split contains 374 items, but the reported score 87.6 is not attainable in a single binary run:
- 327/374 = 87.4%
- 328/374 = 87.7%
There are also some denominator-incompatible cells for OfficeQA and SpreadsheetBench under the released hard-score implementation.
Could you please clarify:
- Did the paper use an earlier version of the dataset splits?
- Were the reported values averaged across multiple runs?
- Were any failed or invalid examples removed before aggregation?
- Could the exact paper splits and aggregation protocol be released?
Publishing per-example results or the exact result manifests would make the reported table much easier to reproduce.
Related: #108
Activity
raytrun commentedon Jul 26, 2026
Thanks for catching this and for the careful analysis!
We checked the issue and found that one sample from the original test set was mistakenly included in the validation set during the data split, resulting in the released 35/18/124 split instead of the intended 35/17/125 split used for the reported results in Table 1.
We have now fixed the manifest and updated the repository accordingly. The corrected split matches the evaluation used in the paper.
Thanks again for pointing this out—we've updated the repository accordingly.
fix: sync LiveMath split metadata
Yif-Yang commentedon Jul 26, 2026
Reopening because #176 resolves the LiveMath split-metadata portion of this report, while the DocVQA, OfficeQA, SpreadsheetBench, and aggregation-protocol questions still need separate clarification. We will keep tracking those remaining reproducibility questions here. Thanks again for the careful analysis.
JianhaoYu-AI commentedon Oct 11, 2026
Hi SkillOpt team,
I'm interested in contributing to the remaining reproducibility work in #158.
I noticed that #176 resolved the LiveMath split metadata mismatch, while the questions concerning DocVQA, OfficeQA, SpreadsheetBench, and score aggregation remain open.
I'd like to investigate the evaluation pipeline, verify the distinction between hard and soft metrics, and build a small, offline reproducibility checker for split IDs, per-example results, and aggregate scores.
Would a focused PR with regression tests and a short audit report be useful?
Thanks!
Yif-Yang commentedon Oct 17, 2026
A focused offline checker would be useful if it stays tied to the remaining reproducibility questions: pin the source/data revision, validate split IDs and overlap, show hard versus soft per-example aggregation, and distinguish recorded results from reconstructed examples. Please include a small synthetic fixture that demonstrates the checks without model calls. #176's resolved LiveMath metadata portion should remain separate, and missing original run artifacts or aggregation details should be reported as missing rather than inferred. This issue stays open for the remaining benchmark and protocol questions.
JianhaoYu-AI commentedon Oct 19, 2026
Thanks for the clarification! That scope makes sense.
I'll work on a focused PR covering:
I'll keep the resolved LiveMath portion in fix: sync LiveMath split metadata #176 out of scope and explicitly report any missing run artifacts or aggregation details rather than making assumptions.
Once the implementation and tests are ready, I'll open a draft PR for review.
Thanks for the guidance!
Yif-Yang commentedon Oct 23, 2026
The proposed checker in #329 makes concrete progress on the offline tooling discussed here: it audits split IDs/counts, saved hard/soft means including error rows, and explicit aggregation and provenance declarations. The new Linux review confirms the supported arithmetic paths, while two partial-input diagnostic gaps remain under review.
This is separate from supplying the original evidence. #176 corrected the LiveMath split metadata; #329 does not recover the original per-item records or answer the remaining DocVQA, OfficeQA, SpreadsheetBench and aggregation-protocol questions. This issue should remain open, alongside the specific historical-cell request in #326.