Repository navigation
docs(bench): record the Phase 2 trial of code and branch reads - #294
Draft
tonyketcham wants to merge 1 commit into
Conversation
Haiku answered the 22 code-context and 22 branch questions twice on the git test bed, with `findcode` (proof find --path and --refs all) or with read-only git alone: - correct: 85 against 84 of 88 question-runs (2 to 1); - read tokens: 11,642 against 26,283 (56% fewer); tool calls 7 against 13. Both arms are near the ceiling on a test bed this small, so the trial shows the saving, not an accuracy gain. The README and a measurement Finding say so, and name the larger test bed needed next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
4 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
This is layer 14 of a stack. It sits on #293.
This PR adds no code. It records the paid trial of the Phase 2 reads from #292:
proof find --pathandproof find --refs all.Haiku answered the 22 code-context and 22 branch questions twice on the git test bed from #293. Each session worked in its own clone and could use read-only git. The no-brief arm could therefore run
git log,git show <ref>:<path>andgit diff main...origin/<branch>. Thefindcodearm also had--pathand--refs.findcodeWhat it shows. Both arms are near the ceiling. On a graph of 524 records and about 40 source files, git and grep reach every answer. The trial cannot show an accuracy gain. What the new reads change is cost: 56% fewer read tokens and half the tool calls, on both question sets.
The 10x test showed that accuracy separates only once crawling stops scaling. The same test is needed here before claiming more than the saving: a test bed with the 10x graph, a larger codebase and more branches.
Files:
.flatbread-proof/.The
compare.mjs --repofix needed to score these runs correctly is in #293.Please don't delete this checklist! Before submitting the PR, please make sure you do the following:
Does this introduce any non-backwards compatible changes?
Does this include any user config changes?
Generated by Claude Code