Skip to content

feat(bench): scale the git test bed with --scale - #295

Draft
tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-p2-resultsfrom
claude/agent-memory-context-potency-6sniud-p2-scale
Draft

tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-p2-resultsfrom
claude/agent-memory-context-potency-6sniud-p2-scale

Conversation

@tonyketcham

Copy link
Copy Markdown
Collaborator

Summary of changes

This is layer 15 of a stack. It sits on #294.

The Phase 2 trial in #294 found that proof find --path and --refs all matched no-brief on accuracy, at 85 against 84 of 88, while reading 56% less. The test bed was small enough to crawl with git: 524 records, about 40 files and 4 branches. This PR builds the same bed at scale, so the comparison can be rerun once crawling costs more.

build-repo.mjs --scale 10 builds the bed from the 10x graph. It adds:

  1. About 470 code files, 437 of them filler. The filler sits in its own folders, and its code differs by module kind, so a grep cannot tell filler from real.
  2. Noise anchors around the real ones:
    • filler code Citations;
    • record bodies that name filler paths;
    • filler commits whose Proof: trailers name Decisions that were in force at the commit's date, some of them since superseded.
  3. Six more engineer branches with filler work and draft PRs, ten in all. They include three conflicts between filler branches, so a CONFLICTS block does not list only the answers.

Kept fixed:

  • The 44 code and branch questions keep their wording and answers. check.mjs --repo proves all of them on both beds.
  • branch-011 needed a different check, because its old check listed every branch by name. The new check asserts that no branch changes the Decision's standing, which means the same thing.
  • --scale 1 builds today's bed byte for byte, and two --scale 10 builds are identical. The bed is packed with git gc.

Harness: collect.mjs now marks sessions that time out or touch paths outside their folder, and grade.mjs lists them.

Offline check on the scaled bed: 38 of 44 questions have the evidence in the top five, the same as on the small bed. One branch question, branch-010, loses its evidence to filler. A word search with git grep returns about 5 times as many files.

Limit. Filler never touches a real file, so routes that follow one file's history are no harder at scale. Changing that would change the answer key. The README says so.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • I added doc comments to any new public exports, and inline comments to any hard-to-understand areas
  • My changes generate no new console errors locally
  • If applicable, try to include a test that fails without this PR but passes with it. These are benchmark scripts. check.mjs proves all 44 answers on both beds, and builds are byte-identical at --scale 1 and repeatable at --scale 10.

Does this introduce any non-backwards compatible changes?

  • Yes
  • No

Does this include any user config changes?

  • Yes
    • If so, I have updated the relevant areas of documentation
  • No

Generated by Claude Code

The Phase 2 trial matched no-brief on accuracy because its test bed was
small enough to crawl. `build-repo.mjs --scale 10` builds the same bed
from the 10x graph, with:

- about 470 code files, 437 of them filler, in their own folders with
  code that differs by module kind, so a grep cannot tell them apart;
- filler code Citations, record bodies naming filler paths, and filler
  commits whose `Proof:` trailers name Decisions in force at the time,
  some since superseded;
- six more engineer branches with filler work and draft PRs, including
  three conflicts between filler branches.

The 44 code and branch questions keep their wording and answers, and
`check.mjs --repo` proves all of them on both beds. `branch-011` now
checks that no branch changes the Decision's standing, since its old
check listed every branch by name. `--scale 1` builds today's bed byte
for byte, and two `--scale 10` builds are identical. The bed is packed
with `git gc`.

`collect.mjs` now marks sessions that time out or touch paths outside
their folder, and `grade.mjs` lists them.

Filler never touches a real file, so routes that follow one file's
history are no harder at scale; the README says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants