[tonyketcham]

Summary of changes

This is layer 6 of a stack. It sits on #285, which sits on #284, #283, #282 and #281.

This PR adds no code. It records the first paid test of proof find. Haiku answered every benchmark question on the 10x graph (4,980 records) twice with the find task text and twice with no-brief. Each question ran in its own fresh, headless session: 540 sessions per run. The cards were those from #284 (find_hash e5be6532…).

Results on the 245 comparable questions (all but world-077, whose count includes filler), pooled over both runs:

Measure	find	no-brief
Correct, run 1	231.1	210.9
Correct, run 2	229.1	210.7
Runs of a question only this arm got right	48	12
Mean read tokens	13,019	33,839
Mean tool calls	9	21

The sign test gives p < 0.0001. Counting each question once, find did better on 37 and worse on 10.

The plan's pass marks, measured on the plan's comparable set: the 245 less the 13 retraction questions, which leaves 232.

1. Pooled sign test p < 0.05: 46 wins, 9 losses. Pass.
2. Read tokens at most 0.65 of no-brief's: 0.39. Pass.
3. "Unknown" answers fall by at least 10 per run: from 32 to 14, and from 35 to 12. Pass.
4. No category loses more than 2 questions: none loses any. Pass.

Checks on the result:

• Seven sessions hit the 10-minute limit, six of them in the no-brief arm. Without them it is 45 wins to 9.
• The gain holds on the half of the questions the cards were not tuned on: 26 wins to 6 (p = 0.0005).
• On the 24 questions that name no Effort, find got 46 of 48 and no-brief 31 of 48. find did better on 10 questions and worse on none.

Files:

• benchmarks/proof-recall/runs/x10-find-r{1,2}-{find,nobrief}.json: answers and costs.
• benchmarks/proof-recall/README.md: a new section with the tables above, and what the runs do not show. The brief arm was not rerun, only Haiku was tested, and the filler is easier to tell apart than real records. The look-alike-filler run tests that last point.
• .flatbread-proof/findings/fnd-search-cards-raise-accuracy-and-cut-reading-on-t--ydqm2ybyqjedzw9y.md: a measurement Finding derived from the 10x Finding.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• I added doc comments to any new public exports, and inline comments to any hard-to-understand areas (none added)
• My changes generate no new console errors locally
• If applicable, try to include a test that fails without this PR but passes with it (results only, no code)

Does this introduce any non-backwards compatible changes?

• Yes
• No

Does this include any user config changes?

• Yes
  • If so, I have updated the relevant areas of documentation
• No

---

Generated by Claude Code