Skip to content

feat(proof): put the answer on more proof find cards - #284

Draft
tonyketcham wants to merge 2 commits into
claude/agent-memory-context-potency-6sniud-find-gatefrom
claude/agent-memory-context-potency-6sniud-find-cards
Draft

tonyketcham wants to merge 2 commits into
claude/agent-memory-context-potency-6sniud-find-gatefrom
claude/agent-memory-context-potency-6sniud-find-cards

Conversation

@tonyketcham

Copy link
Copy Markdown
Collaborator

Summary of changes

This is layer 4 of a stack. It sits on #283 (the offline gate), which sits on #282 (proof find) and #281.

proof find returns one short "card" per hit. Before this PR, the free offline gate in #283 failed: the right record was in the top five often enough, but the answer itself was on a card for only 56% of questions, against a 60% bar. This PR makes cards quote more of what each hit says, so an agent can answer from the card more often without reading the record in full.

Changes in packages/flatbread/src/proof/find.ts:

  1. Passage choice. The quoted passage is the window whose query words weigh most, with rare words weighing more (the same IDF weight BM25 ranking uses). Words the title already shows only break ties. The best passage comes first.
  2. Placement. The matches sit in the middle of the roughly 160-character window. The passage runs on to finish a short word after the last match, so a value that follows its label stays on the card.
  3. Second passage. A second passage appears only when another part of the body holds a query word that the first passage and the title lack.
  4. Attached Blob passage. A Blob is a record that holds raw data such as a capture or a table. A new blob <id>: line quotes the best passage of a Blob the hit carries, or cites through a Citation.
    • Only a current, unretracted Citation and Blob count, as of --as-of.
    • With --retracted include, the line is marked (retracted).
    • No Blob is quoted twice in one reply.
  5. Size aim. Passages add their own length to the reply's size aim, so they never push a chain or a link off a card.
  6. Reverse-link order. The in: line lists verdict links (resolves, rejects, mitigates and the like) before derived_by. On a much-used record, derived_by used to fill the line.

FindHit.snippet is replaced by passages[] and an optional attached. The skill docs (reference.md, SKILL.md) describe the new card lines.

Guarding against fitting to the questions. find-recall.mjs gains --split tune|holdout|all. A hash of each question id puts 130 questions in tune and 116 in holdout. Only tune misses were read to choose the changes, and holdout was scored once, after the code was final. Results on the Effort-stripped set:

Graph Split Answer on a card, before After
10x holdout 58% (51/88) 66% (58/88)
10x tune 54% 59%
10x all 56% 62% (gate passes)
1x holdout 57% (50/88) 65% (57/88)

On each graph's holdout half, 7 questions gained the answer and none lost it. The tuned half still falls short of the bar. Evidence ranking did not change. The mean reply grew from about 6,080 to 6,370 characters, and the largest from 6,700 to 7,360. Most misses left are count and status-history questions, which a later proof state read should answer. The README records the diagnosis, a check that the chance set-asides did not create the gain, and the new find_hash.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • I added doc comments to any new public exports, and inline comments to any hard-to-understand areas
  • My changes generate no new console errors locally
  • If applicable, try to include a test that fails without this PR but passes with it. find.test.ts adds ten tests, and the first six fail against the old find.ts. serve.test.ts still shows served output matching local output.

Does this introduce any non-backwards compatible changes?

Does this include any user config changes?

  • Yes
    • If so, I have updated the relevant areas of documentation
  • No

Generated by Claude Code

A `proof find` card now quotes more of what a hit says, so an agent
can answer from the card without a second read more often.

- The passage is the window whose query words weigh most, with rare
  words weighing more. Words the title already shows only break ties.
  The best passage comes first.
- The matches sit in the middle of the window, and the passage runs on
  to finish a short word, so a value after its label stays.
- A second passage appears when another part of the body holds a query
  word the first lacks.
- A `blob <id>:` line quotes the best passage of a Blob the hit carries
  or cites through a Citation. Only a current, unretracted Citation and
  Blob count, as of `--as-of`, and no Blob is quoted twice in a reply.
- Reverse links list verdict links such as `resolves` and `rejects`
  before `derived_by`.

`find-recall.mjs` gains `--split tune|holdout|all`, so changes can be
chosen from one half of the questions and scored on the other. On the
held-out half of the 10x graph the answer is on a card for 58 of 88
questions, up from 51, with none lost. That passes the gate's 60% bar.
The tuned half reaches only 59%. The README records both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
…-find-cards

Brings main's cross-Effort derives_from change (#280) up the stack,
through the merge on the branch below. No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants