Skip to content

docs(bench): record the Phase 2 trial of code and branch reads - #294

Draft
tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-code-benchfrom
claude/agent-memory-context-potency-6sniud-p2-results
Draft

tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-code-benchfrom
claude/agent-memory-context-potency-6sniud-p2-results

Conversation

@tonyketcham

Copy link
Copy Markdown
Collaborator

Summary of changes

This is layer 14 of a stack. It sits on #293.

This PR adds no code. It records the paid trial of the Phase 2 reads from #292: proof find --path and proof find --refs all.

Haiku answered the 22 code-context and 22 branch questions twice on the git test bed from #293. Each session worked in its own clone and could use read-only git. The no-brief arm could therefore run git log, git show <ref>:<path> and git diff main...origin/<branch>. The findcode arm also had --path and --refs.

Measure findcode No-brief
Correct, of 88 question-runs 85 84
Only this arm right 2 1
Mean read tokens 11,642 26,283
Mean tool calls 7 13
Code set: correct, read tokens 42, 10,186 41, 29,912
Branch set: correct, read tokens 43, 13,097 43, 22,654

What it shows. Both arms are near the ceiling. On a graph of 524 records and about 40 source files, git and grep reach every answer. The trial cannot show an accuracy gain. What the new reads change is cost: 56% fewer read tokens and half the tool calls, on both question sets.

The 10x test showed that accuracy separates only once crawling stops scaling. The same test is needed here before claiming more than the saving: a test bed with the 10x graph, a larger codebase and more branches.

Files:

  • four run files;
  • a README section, "Phase 2 trial: code context and branches";
  • a measurement Finding in .flatbread-proof/.

The compare.mjs --repo fix needed to score these runs correctly is in #293.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • I added doc comments to any new public exports, and inline comments to any hard-to-understand areas (none added)
  • My changes generate no new console errors locally
  • If applicable, try to include a test that fails without this PR but passes with it (results only, no code)

Does this introduce any non-backwards compatible changes?

  • Yes
  • No

Does this include any user config changes?

  • Yes
    • If so, I have updated the relevant areas of documentation
  • No

Generated by Claude Code

Haiku answered the 22 code-context and 22 branch questions twice on
the git test bed, with `findcode` (proof find --path and --refs all) or
with read-only git alone:

- correct: 85 against 84 of 88 question-runs (2 to 1);
- read tokens: 11,642 against 26,283 (56% fewer); tool calls 7 against 13.

Both arms are near the ceiling on a test bed this small, so the trial
shows the saving, not an accuracy gain. The README and a measurement
Finding say so, and name the larger test bed needed next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants