Repository navigation
feat(proof): put the answer on more proof find cards - #284
Draft
tonyketcham wants to merge 2 commits into
Draft
tonyketcham wants to merge 2 commits into
tonyketcham wants to merge 2 commits into
Conversation
A `proof find` card now quotes more of what a hit says, so an agent can answer from the card without a second read more often. - The passage is the window whose query words weigh most, with rare words weighing more. Words the title already shows only break ties. The best passage comes first. - The matches sit in the middle of the window, and the passage runs on to finish a short word, so a value after its label stays. - A second passage appears when another part of the body holds a query word the first lacks. - A `blob <id>:` line quotes the best passage of a Blob the hit carries or cites through a Citation. Only a current, unretracted Citation and Blob count, as of `--as-of`, and no Blob is quoted twice in a reply. - Reverse links list verdict links such as `resolves` and `rejects` before `derived_by`. `find-recall.mjs` gains `--split tune|holdout|all`, so changes can be chosen from one half of the questions and scored on the other. On the held-out half of the 10x graph the answer is on a card for 58 of 88 questions, up from 51, with none lost. That passes the gate's 60% bar. The tuned half reaches only 59%. The README records both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
4 of 8 tasks
…-find-cards Brings main's cross-Effort derives_from change (#280) up the stack, through the merge on the branch below. No conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
4 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
This is layer 4 of a stack. It sits on #283 (the offline gate), which sits on #282 (
proof find) and #281.proof findreturns one short "card" per hit. Before this PR, the free offline gate in #283 failed: the right record was in the top five often enough, but the answer itself was on a card for only 56% of questions, against a 60% bar. This PR makes cards quote more of what each hit says, so an agent can answer from the card more often without reading the record in full.Changes in
packages/flatbread/src/proof/find.ts:blob <id>:line quotes the best passage of a Blob the hit carries, or cites through a Citation.--as-of.--retracted include, the line is marked(retracted).in:line lists verdict links (resolves,rejects,mitigatesand the like) beforederived_by. On a much-used record,derived_byused to fill the line.FindHit.snippetis replaced bypassages[]and an optionalattached. The skill docs (reference.md,SKILL.md) describe the new card lines.Guarding against fitting to the questions.
find-recall.mjsgains--split tune|holdout|all. A hash of each question id puts 130 questions intuneand 116 inholdout. Only tune misses were read to choose the changes, and holdout was scored once, after the code was final. Results on the Effort-stripped set:On each graph's holdout half, 7 questions gained the answer and none lost it. The tuned half still falls short of the bar. Evidence ranking did not change. The mean reply grew from about 6,080 to 6,370 characters, and the largest from 6,700 to 7,360. Most misses left are count and status-history questions, which a later
proof stateread should answer. The README records the diagnosis, a check that the chance set-asides did not create the gain, and the newfind_hash.Please don't delete this checklist! Before submitting the PR, please make sure you do the following:
find.test.tsadds ten tests, and the first six fail against the oldfind.ts.serve.test.tsstill shows served output matching local output.Does this introduce any non-backwards compatible changes?
FindHit.snippetbecamepassages[].proof findis new in Find Proof records by what they say withproof find#282 and unreleased, so nothing outside this stack depends on it.Does this include any user config changes?
Generated by Claude Code