[tonyketcham]

Summary of changes

This is layer 16 of a stack. It sits on #295.

This PR adds no code. It records the paid rerun of the Phase 2 trial on the scaled test bed from #295. Haiku answered the same 44 code and branch questions twice, using findcode (proof find --path and --refs all) or read-only git alone, with a 20-minute timeout.

Test bed	findcode correct	No-brief correct	Read tokens, findcode / no-brief	Tool calls
Small (524 records)	85 of 88	84 of 88	11,642 / 26,283	7 / 13
Scaled (5,131 records)	85 of 88	85 of 88	13,422 / 22,829	7 / 15

What it shows:

• Accuracy: scale does not separate the arms. They disagree on two question-runs, one each way.
• Cost: findcode reads 41% fewer tokens and makes half the tool calls.
• Session checks: no session timed out. One no-brief session was flagged for reading outside its folder. It was reading its own background task's output, nothing from the benchmark.

This differs from search over record text, which did gain accuracy on the 10x graph. Git is already a precise index for code. git log -- <path> and git diff main...origin/<branch> stay small however large the graph grows, and the filler never touches a real file. So --path and --refs earn their place by cost, not accuracy. A test that could show more would ask about code that no single path names, or about many branches at once.

Files:

• four run files;
• a README section, "Phase 2 at scale";
• a measurement Finding in .flatbread-proof/.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• I added doc comments to any new public exports, and inline comments to any hard-to-understand areas (none added)
• My changes generate no new console errors locally
• If applicable, try to include a test that fails without this PR but passes with it (results only, no code)

Does this introduce any non-backwards compatible changes?

• Yes
• No

Does this include any user config changes?

• Yes
  • If so, I have updated the relevant areas of documentation
• No

---

Generated by Claude Code