Skip to content

feat(bench): refactor bed with behaviour, many-branch and refactor question sets - #297

Draft
tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-p2-scale-resultsfrom
claude/agent-memory-context-potency-6sniud-p3-bench
Draft

tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-p2-scale-resultsfrom
claude/agent-memory-context-potency-6sniud-p3-bench

Conversation

@tonyketcham

Copy link
Copy Markdown
Collaborator

Stacked on #296.

What

Phase 3 of the code-context benchmark. Two new tests:

  1. Questions no single path names. behaviour.json (16) asks about behaviour spread across folders, with spread and control groups. multibranch.json (12) asks about many branches at once.

  2. Code anchors a refactor left stale. build-repo.mjs --refactor adds 10 (1x) or 11 (10x) refactor commits on main without updating Proof:

    • two folder moves;
    • file renames, and symbol renames (evict_pages → reclaim_pages, warm_district → prewarm_district, upgrade_steps → migration_steps, hash_blocks → hash_state_blocks);
    • one module split and one merge;
    • one behaviour change: the hash interval goes from 30 to 60 frames, while the accepted Decision still says 30.

    refactor.json (19) asks about the moved code. Its groups are follow, no-follow, drift, location and control. The refactor map is written beside the keymap, outside the repository.

Details

  • New test-bed checks: renamed, staleAnchor, codeAt, anchorDrift, goneText and refAll.
  • check.mjs refuses the new sets on a bed without a 10x refactor map.
  • run.mjs leaves out the 17 code and branch questions the refactor makes stale.
  • Without --refactor, both existing beds rebuild byte for byte.

Checks

  • check.mjs: ok, 270 questions; ok, 91 questions on the 10x refactor bed; ok, 44 questions on p2-bed-x10.
  • Prettier is clean.

Offline baseline with today's reads

  • Every --path query on moved code returns no governing records: 10 of 10.
  • Four cards cite moved code, and none of them says the code moved.

The paid baseline comes next.


Generated by Claude Code

…or question sets

build-repo.mjs --refactor adds 10 (1x) or 11 (10x) refactor commits on
main that move folders, rename files and symbols, split one module, merge
two, and change one behaviour, without touching Proof. Its map is written
beside the keymap, outside the repository.

New test-bed checks: renamed, staleAnchor, codeAt, anchorDrift, goneText
and refAll. New sets: behaviour (16), multibranch (12) and refactor (19).
check.mjs refuses these sets on a bed without a refactor map, and run.mjs
leaves out the 17 code and branch questions the refactor makes stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants