[tonyketcham]

Stacked on #299.

What

benchmarks/proof-build measures whether having Proof in context helps or hurts an agent that plans or builds a feature from a product-sense request: one that describes behaviour in a user's words and names no file, symbol or record.

Bed. A deterministic generator builds Tessera, a zero-dependency TypeScript design-editor core:

• Code: 79 source files and 132 node:test tests, with 84 commits by five authors.
• Proof: a 7-Effort graph with 149 live records, 37 commits with Proof: trailers, and 31 code citations.
• Rules: 6 are recorded only in Proof, and 4 are also visible in code.
• History: five drifts that the code moved past without updating Proof, and two proper supersessions.
• Variants: none (the same code with no Proof folder and no trailers) and x10 (1,531 records).

Tasks. 18 build tasks (6 proof-only, 5 code-visible, 4 stale, 3 control) and 8 plan tasks with rubrics, phrased as product requests.

Arms. Each arm sees identical code and an identical request. Only Proof access differs:

Arm	Proof access
none	no Proof at all
skill	Proof installed; the agent decides whether to use it
brief	every Effort's brief pasted in
find	told to run proof find first
findcode	find, plus --path and --refs
dump	every live record pasted in
dump10x	the same, on the 10x graph

Grading.

• grade.mjs applies each session's diff to a clean tree and runs hidden acceptance, constraint and regression tests with no network.
• judge.mjs scores plans after blinding them, so the judge can't tell which arm wrote a plan.
• compare.mjs pairs each arm with none, using sign tests with Holm correction.

Results: Haiku, full run

There were 446 sessions plus a 30-session pilot. No session timed out or hit an API error. 23 sessions left their folder harmlessly (scratch files, cd /tmp), and none read the hidden tests.

Builds, all-green of 54:

Arm	Result	Tokens processed vs none
none	47	1.0×
skill	41	1.3×
find	40	1.1×
findcode	41	1.3×
brief	37	1.5×
dump	36	2.0×

• One stale Decision caused most of the loss. It says shortcuts are registered in a file that a refactor turned into the frozen plugin table. Every task that adds a shortcut then broke plugin API v1 in the Proof arms; on B-S3, none went 3 of 3 and every Proof arm 0 of 3.
• The stale-anchor note from feat(proof): follow git renames in --path and flag stale code anchors #298 didn't stop it. findcode sessions saw the note and still followed the Decision.
• Proof-only build tasks showed no gain (none passed 17 of 18): the code's own patterns taught the recorded rules.
• Plans: pasted Proof lifted proof-only plans (dump 1.00 against 0.90) and dragged down stale ones (0.76 against 1.00), so overall they're equal.
• Statistics: none of this is significant on the pre-registered tests; the best Holm-corrected p is 0.31. Pooled, dump has p = 0.059, but the harm sits in a few tasks that share one record.

The full tables are in the README. Per-session rows and the compare output are in results/, and a measurement Finding records the result.

Other changes in this PR

• The hidden tests and the reference and naive patches are now committed in hidden/, since the runs are done.
• grade.mjs flags sessions that read the committed copies.
• gate.mjs refuses an --out that would rebuild tasks/ in place.
• A checker fix stops the false "left its folder" flags the pilot found.

Checks

• Offline gate: 127 static checks and 54 graded patches, with 0 failures, both from the committed hidden/ and from the copy outside the repo.
• Beds: they rebuild byte for byte.
• The none clone: it has no trace of Proof.

Next

1. A read that marks a Decision itself as suspect when the code it rests on has moved.
2. Proof-only tasks whose rules the code doesn't hint at.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW