Repository navigation
feat(bench): proof-only-hard tasks whose rules the code never hints at - #302
Draft
tonyketcham wants to merge 2 commits into
Conversation
…ts at v1's proof-only tasks showed no gain because the code's own patterns taught their rules. The new category has 6 build and 2 plan tasks whose rule is a product decision recorded in Proof before the feature existed (a security-review constraint, a deferred issue with agreed behaviour, usability findings) and nothing in the code, tests, docs or history hints at: plugin events redact names unless the plugin may export; font size steps follow the type scale; a wrapping frame has no fill and does not clip; colour replacement skips locked layers; select-same-fill stays in its top-level frame; pixel rounding rounds edges, not sizes. 18 records join the bed before the first drift; the none bed is unchanged. The gate checks that every leak word is in the records and none is in the prompt or the none bed's history: 193 static checks and 72 graded patches pass, and the v1 tasks grade as before. DESIGN.md pre-registers the new category and its Sonnet 5.5 run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Proof-only-hard on Sonnet 5.5 (149 sessions): builds all-green none 3 of 18, skill 15, find 15, findcode 11, brief 11, dump 18; every Proof arm beats none after Holm correction (dump p = 0.0005), and plans score 0.88-1.00 against 0.25. Search arms lost where the query did not reach the record. The suspect warning on Haiku (216 sessions, stale and code-visible tasks): build all-green of 27 rose from 17 to 20 for findcode and 13 to 18 for brief, and findcode passed the stale shortcut task 3 of 3 after 0 of 3; it did not help when sessions saw only find's summary line or when the warning sat inside a full dump. The README gains a v2 results section, results/ the per-session rows and compare output, and a measurement Finding records both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #301.
Why
In #300, the proof-only tasks showed no gain: the no-Proof arm passed 17 of 18. The code's own patterns already taught their rules, so those tasks couldn't show a benefit.
What
This adds a new category, proof-only-hard: 6 build tasks and 2 plan tasks. Each rule is a product decision recorded in Proof before the feature existed. Nothing in the code, tests, docs or history hints at it, and the natural implementation breaks it.
Results (in the README and
results/)Proof-only-hard on Sonnet 5.5 (149 sessions):
noneskillfindfindcodebriefdumpnoneon the pre-registered test, after Holm correction:dumpp = 0.0005,skillandfind0.002,briefandfindcode0.043.skillanddumpwin 6 tasks to 0 (p = 0.031).findcodepassed 0 of 3 on the type-scale task, wherefindpassed 3 of 3.The #301 suspect warning on Haiku (216 sessions, stale and code-visible tasks). Builds all-green out of 27, v1 → v2:
findcodebriefskillfinddumpfindcodepassed the shortcut task 3 of 3, up from 0 of 3.find's summary line, or when the warning sat inside a full dump.A measurement Finding records both runs.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW