Skip to content

feat(bench): add questions that name no Effort, and filler that looks real - #285

Draft
tonyketcham wants to merge 2 commits into
claude/agent-memory-context-potency-6sniud-find-cardsfrom
claude/agent-memory-context-potency-6sniud-find-ambiguous
Draft

tonyketcham wants to merge 2 commits into
claude/agent-memory-context-potency-6sniud-find-cardsfrom
claude/agent-memory-context-potency-6sniud-find-ambiguous

Conversation

@tonyketcham

Copy link
Copy Markdown
Collaborator

Summary of changes

This is layer 5 of a stack. It sits on #284, which sits on #283, #282 and #281.

The recall benchmark's questions almost always name their Effort, Proof's top-level record for one piece of work. Real prompts often don't, and the benchmark's filler records are easy to tell apart from real ones. This PR adds a test for each of those gaps. It changes benchmark files only.

  1. questions/ambiguous.json: 24 questions rewritten from tier 0-2 questions.
    • None names an Effort, an Effort title, or a platform that points at one Effort.
    • Each records its source question in from, and lists in rivals near-miss records from other Efforts.
    • check.mjs proves each answer is unique across the whole graph, using checks scoped to effort: "*". It refuses a question with no rivals, or with a rival in the answer's own Effort. It also rejects a platform word when fewer than two Efforts use it, or when one Effort holds more than two thirds of the records that use it.
    • New category: ambiguous-effort.
  2. Defaults stay put. run.mjs, grade.mjs, compare.mjs and find-recall.mjs still ask only the original 246 questions unless given --only ambiguous, or --only default,ambiguous for both. compare.mjs on the full Haiku runs still prints 221.42 against 226.25, 6 against 12.
  3. build.mjs --scale N --filler-overlap <fraction>: that share of filler records reuse real subsystem names, but link to no real record and state no answer value.
    • Output is seeded, and repeats for a given fraction and set of question files.
    • Default builds are unchanged: rebuilt 1x and --scale 10 graphs match byte for byte (diff -r clean).
    • check.mjs --graph <dir> validates any built graph.

First results. The ambiguous set was frozen before proof find ran on it. The cards are those from #284, on the Effort-stripped set:

Graph Questions Evidence in top 5 Answer on a card Right Effort first
10x 24 ambiguous 96% 87% 88%
10x, 30% look-alike filler 24 ambiguous 96% 87% 92%
10x, 30% look-alike filler 246 original 80% 62% (gate passes) 80%

Search finds the right record without an Effort name, but these questions come from easy tiers. Look-alike filler costs one point of evidence and no answers. The README says what the paid runs still need to show.

On both 10x graphs, check.mjs reports one known error, world-077. The 10x run already set that question aside: its count includes filler.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • I added doc comments to any new public exports, and inline comments to any hard-to-understand areas
  • My changes generate no new console errors locally
  • If applicable, try to include a test that fails without this PR but passes with it. These are benchmark scripts. check.mjs now validates 270 questions, and I broke a question on purpose to confirm both new rivals errors fire.

Does this introduce any non-backwards compatible changes?

  • Yes
  • No

Does this include any user config changes?

  • Yes
    • If so, I have updated the relevant areas of documentation
  • No

Generated by Claude Code

… real

Two new ways to test `proof find` the way agents will use it:

- `questions/ambiguous.json` holds 24 questions rewritten from tier 0-2
  questions. None of them names an Effort, an Effort title, or a
  platform that points at one Effort. Each lists `rivals`, near-miss
  records in other Efforts. `check.mjs` proves each answer is unique
  across the whole graph and rejects a platform word that one Effort
  dominates. The new category is `ambiguous-effort`.
- `build.mjs --scale N --filler-overlap <fraction>` makes that share of
  filler records reuse real subsystem names while linking to no real
  record and stating no answer value. Default builds are unchanged,
  byte for byte, at 1x and at 10x.

Every script still asks the original 246 questions unless given
`--only ambiguous`, so earlier runs stay comparable.

The ambiguous set was frozen before `find` ran on it. On the 10x graph,
search puts the evidence in the top five for 23 of 24 and the answer
on a card for 87%. With 30% look-alike filler, the gate on all 246
questions still passes: evidence 80%, answer on a card 62%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW

Copy link
Copy Markdown
Collaborator Author

integration-nextjs (20.x, macos-latest) failed during pnpm install, before any test ran. sharp's install timed out downloading libvips from GitHub releases, and so did sqlite3's prebuilt download. This PR changes only files under benchmarks/proof-recall/, so the failure is not this PR's, and no fix is needed in the code. I re-ran the failed job once.


Generated by Claude Code

…-find-ambiguous

Brings main's cross-Effort derives_from change (#280) up the stack,
through the merge on the branch below. No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants