Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
id: fnd-at-scale-code-and-branch-reads-still-match-git-o--e2ws7dke8rr7qbej
effort: eff-effort-graph-memory-and-agent-wedge--szeqvmgqjqnhd002
title: 'At scale, code and branch reads still match git on accuracy'
kind: measurement
created_at: '2026-10-20T08:19:29.482Z'
derives_from:
- fnd-code-and-branch-reads-halve-the-reading-on-a-sma--4ffp9s93s9swpctv
---

The Phase 2 trial was rerun on a test bed ten times larger: the 10x graph (5,131 records), about 470 code files with filler that grep cannot tell apart, and ten branches with filler conflicts. Haiku answered the same 44 code and branch questions twice, with `proof find --path` and `--refs all` or with read-only git alone.

- Accuracy matched again: 85 against 85 of 88 question-runs (1 to 1), as on the small bed (85 against 84).
- The new reads read 41 percent fewer tokens (13,422 against 22,829) and made 7 tool calls instead of 15.

Unlike search over record text, these reads do not gain accuracy at scale, because git is already a precise index for code: `git log -- <path>` and `git diff main...origin/<branch>` stay small as the graph grows. `--path` and `--refs` are worth keeping for cost. A test that could show more would ask about code no single path names, or about many branches at once.
25 changes: 25 additions & 0 deletions benchmarks/proof-recall/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1428,3 +1428,28 @@ on both sets. The 10x test showed that accuracy separates only once
crawling stops scaling; the same test is needed here, on a test bed with
the 10x graph, a larger codebase and more branches, before claiming more
than the saving.

### Phase 2 at scale

The same trial on the scaled test bed (`build-repo.mjs --scale 10`: the 10x
graph, about 470 code files, ten branches), Haiku, twice, with
`--timeout 1200`. Run files: `runs/p2x10-r{1,2}-*.json`.

| Test bed | `findcode` correct | no-brief correct | Read tokens, `findcode` / no-brief | Tool calls |
| ------------------- | ------------------ | ---------------- | ---------------------------------- | ---------- |
| small (524 records) | 85 of 88 | 84 of 88 | 11,642 / 26,283 | 7 / 13 |
| scaled (5,131) | 85 of 88 | 85 of 88 | 13,422 / 22,829 | 7 / 15 |

On the scaled bed the arms disagree on two question-runs, one each way.
`findcode` read 41% fewer tokens (code set 11,384 against 24,642; branch
set 15,460 against 21,016). No session timed out; one no-brief session
was flagged for reading outside its folder, and it was reading its own
background task's output, not anything from the benchmark.

So scale does not separate the arms here, unlike search on the 10x graph.
Git is already a precise index for code: `git log -- <path>` and
`git diff main...origin/<branch>` stay small however large the graph and
codebase grow, and the filler never touches a real file. `--path` and
`--refs` earn their place by cost, about half the reading and half the
tool calls, not by accuracy. A test that could show more would ask about
code no single path names, or about many branches at once.
Loading
Loading