Repository navigation
docs(bench): record Phase 3 on the refactor bed - #299
Draft
tonyketcham wants to merge 1 commit into
Draft
tonyketcham wants to merge 1 commit into
tonyketcham wants to merge 1 commit into
Conversation
Haiku answered 74 questions twice on the 10x refactor bed: today's `findcode` against read-only git, then `findcode` with the rename-aware reads from the previous commit. - today's reads: 133 against 129 of 148 (p = 0.48), 27% fewer tokens; behaviour no single path names read a third less; refactor set slightly worse than git (34 against 35). - rename-aware reads: refactor-set reading down a fifth (a third on holdout), tool calls 14 to 10, accuracy unchanged (129 against 133, p = 0.34). The README and a measurement Finding say stale anchors cost less than expected because git still shows the moves, and name a stale-anchor audit read as the next change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #298.
Records the paid Phase 3 runs: Haiku, 74 questions, twice, on the 10x refactor bed. It adds a README section, the six run files, and a measurement Finding.
findcode, current readsfindcode, #298's readsNone of the accuracy gaps is larger than chance. The token savings are consistent.
None of the sessions timed out. Ten sessions wrote scratch files outside their folder. None of them read the questions, the keymap or the refactor map.
Generated by Claude Code