Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
id: fnd-code-and-branch-reads-halve-the-reading-on-a-sma--4ffp9s93s9swpctv
effort: eff-effort-graph-memory-and-agent-wedge--szeqvmgqjqnhd002
title: Code and branch reads halve the reading on a small test bed
kind: measurement
created_at: '2026-10-20T05:22:58.158Z'
derives_from:
- fnd-search-cards-raise-accuracy-and-cut-reading-on-t--ydqm2ybyqjedzw9y
---

Phase 2 added two reads for teams with many branches: `proof find --path` lists the records that govern a piece of code, and `proof find --refs all` overlays what other branches propose, with a CONFLICTS block. Haiku answered 22 code-context and 22 branch questions twice on a git test bed, each session in its own clone, with the new reads or with read-only git alone.

- Accuracy matched: 85 against 84 of 88 question-runs (2 to 1).
- The new reads cut read tokens by 56 percent (11,642 against 26,283) and tool calls from 13 to 7, on both question sets.

The test bed is small (524 records, about 40 source files), so git and grep reach every answer and both arms sit near the ceiling. The saving is real; an accuracy gain needs a test bed large enough that crawling stops working, as the 10x graph showed for search.
30 changes: 30 additions & 0 deletions benchmarks/proof-recall/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1334,3 +1334,33 @@ The accuracy gain is all on the tune questions this round was chosen from
before and had little room. The saving in reading carries over: 15% fewer
tokens on holdout. So this round is kept for clearer, cheaper reads; it has
not shown that it raises accuracy on questions it was not built from.

### Phase 2 trial: code context and branches

Haiku, twice, on the 22 code-context and 22 branch questions of the git
test bed (`build-repo.mjs`), each session in its own clone:
`run.mjs --repo <bed> --strategies findcode,nobrief --only code,branch`.
In this mode every arm may use read-only git, so no-brief can run
`git log`, `git show <ref>:<path>` and `git diff main...origin/<branch>`;
`findcode` adds `proof find --path` and `--refs all`. Run files:
`runs/p2-r{1,2}-*.json`.

Pooled over both runs (88 question-runs per arm):

| Measure | `findcode` | no-brief |
| ------------------------------- | ----------- | ----------- |
| Correct | 85 | 84 |
| Only this arm right | 2 | 1 |
| Mean read tokens | 11,642 | 26,283 |
| Mean tool calls | 7 | 13 |
| Code set: correct / read tok. | 42 / 10,186 | 41 / 29,912 |
| Branch set: correct / read tok. | 43 / 13,097 | 43 / 22,654 |

Both arms are near the ceiling, so this trial cannot show an accuracy
gain: with git on a 524-record graph and about 40 source files, an agent
finds these answers by grepping and reading branches directly. What the
new reads change is cost: 56% fewer read tokens and half the tool calls,
on both sets. The 10x test showed that accuracy separates only once
crawling stops scaling; the same test is needed here, on a test bed with
the 10x graph, a larger codebase and more branches, before claiming more
than the saving.
Loading
Loading