Repository navigation
feat(bench): scale the git test bed with --scale - #295
Draft
tonyketcham wants to merge 1 commit into
Draft
tonyketcham wants to merge 1 commit into
tonyketcham wants to merge 1 commit into
Conversation
The Phase 2 trial matched no-brief on accuracy because its test bed was small enough to crawl. `build-repo.mjs --scale 10` builds the same bed from the 10x graph, with: - about 470 code files, 437 of them filler, in their own folders with code that differs by module kind, so a grep cannot tell them apart; - filler code Citations, record bodies naming filler paths, and filler commits whose `Proof:` trailers name Decisions in force at the time, some since superseded; - six more engineer branches with filler work and draft PRs, including three conflicts between filler branches. The 44 code and branch questions keep their wording and answers, and `check.mjs --repo` proves all of them on both beds. `branch-011` now checks that no branch changes the Decision's standing, since its old check listed every branch by name. `--scale 1` builds today's bed byte for byte, and two `--scale 10` builds are identical. The bed is packed with `git gc`. `collect.mjs` now marks sessions that time out or touch paths outside their folder, and `grade.mjs` lists them. Filler never touches a real file, so routes that follow one file's history are no harder at scale; the README says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
4 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
This is layer 15 of a stack. It sits on #294.
The Phase 2 trial in #294 found that
proof find --pathand--refs allmatched no-brief on accuracy, at 85 against 84 of 88, while reading 56% less. The test bed was small enough to crawl with git: 524 records, about 40 files and 4 branches. This PR builds the same bed at scale, so the comparison can be rerun once crawling costs more.build-repo.mjs --scale 10builds the bed from the 10x graph. It adds:Proof:trailers name Decisions that were in force at the commit's date, some of them since superseded.Kept fixed:
check.mjs --repoproves all of them on both beds.branch-011needed a different check, because its old check listed every branch by name. The new check asserts that no branch changes the Decision's standing, which means the same thing.--scale 1builds today's bed byte for byte, and two--scale 10builds are identical. The bed is packed withgit gc.Harness:
collect.mjsnow marks sessions that time out or touch paths outside their folder, andgrade.mjslists them.Offline check on the scaled bed: 38 of 44 questions have the evidence in the top five, the same as on the small bed. One branch question,
branch-010, loses its evidence to filler. A word search withgit grepreturns about 5 times as many files.Limit. Filler never touches a real file, so routes that follow one file's history are no harder at scale. Changing that would change the answer key. The README says so.
Please don't delete this checklist! Before submitting the PR, please make sure you do the following:
check.mjsproves all 44 answers on both beds, and builds are byte-identical at--scale 1and repeatable at--scale 10.Does this introduce any non-backwards compatible changes?
Does this include any user config changes?
Generated by Claude Code