feat(proof): put the answer on more proof find cards

.agents/skills/proof/SKILL.md

@@ -175,13 +175,14 @@ Pick the read by what you need, in this order:
    records by their title, full body, and Blob payload, and prints the hit
    cards inline in the JSON, so you need no second Read. Each card shows the
    newest version of the record (or, with `--as-of`, the version current
-   then), its standing, the passage that matched, its supersession chain
-   with dates, and its links in both directions. Leave out `--effort` when
-   you do not know the Effort: hits then group by Effort, at most 3 each
-   unless one Effort clearly leads. The `missing` terms name query words no
-   hit contains. Ordinary words often land there; a missing distinctive
-   term, such as a name, a number, or a code reference, is the cue that the
-   answer may not be recorded.
+   then), its standing, the passages that matched (up to two, plus one
+   from an attached Blob payload such as a capture or table), its
+   supersession chain with dates, and its links in both directions. Leave
+   out `--effort` when you do not know the Effort: hits then group by
+   Effort, at most 3 each unless one Effort clearly leads. The `missing`
+   terms name query words no hit contains. Ordinary words often land there;
+   a missing distinctive term, such as a name, a number, or a code
+   reference, is the cue that the answer may not be recorded.
 2. **Read one record in full: `proof get <id>`.**
 3. **Count or page through records: `proof records`.** `find` shows only
    the top hits, so never count from it.

.agents/skills/proof/reference.md

@@ -348,10 +348,31 @@ flatbread proof serve [--socket .flatbread/proof/serve.sock]
   Effort. Unless the top Effort's sum is at least 1.5 times the next one's,
   at most 3 hits come from one Effort at first; slots still empty then go to
   the next hits by score.
+- **Passages:** a card quotes up to two passages of the matched version's
+  body, each about 160 characters. The first is the best: the window whose
+  query words weigh most, counting rare words more (by the same inverse
+  document frequency that ranks hits) and counting words the card's title
+  already shows only to break ties. The matches sit in the middle of the
+  window, and the passage runs a little past 160 characters to finish a
+  short word right after the last match, so a value after a matched label
+  (`"fuse_ms": 42`) is not cut off. A second passage appears only when
+  another part of the body holds a query word the first passage and the
+  title lack; it may come from earlier in the body, as its offsets show.
+  When the shown version carries a Blob (a Citation's `blob`), or cites a
+  Citation that does, and that payload holds a query word, the card also
+  quotes the payload's best passage on a `blob <id>:` line. A Blob that is a
+  hit itself, or that an earlier card already quotes, is not quoted again.
+  The Citation and the Blob must be current: not superseded, not written
+  after `--as-of`, and not retracted unless `--retracted include` or `only`
+  lets retracted records in; then the line reads `blob <id> (retracted):`.
+  Query words found only in a quoted Blob count as matched, not missing.
 - **Caps:** `--limit` 1 to 10, default 5. Each card holds at most 4 chain
   versions (the head and the shown and matched versions always appear), 3
-  links per relation, and 6 code references. When five cards run past about
-  4,000 characters, the cards shrink in this order: link titles, the
+  links per relation, and 6 code references. The cards aim for 300
+  characters plus 600 per card, plus the length of every passage they
+  quote: about 4,000 characters for five cards with one passage each.
+  Passages are never cut, so quoting more never pushes out a chain or a
+  link. Past the aim, the cards shrink in this order: link titles, the
   `efforts by match` line, the middle of each chain, then link entries. A
   relation with no entry left shows its count, as `cited_by +4`. When more
   records matched than are shown, the envelope has `complete: false` and
@@ -364,10 +385,12 @@ Each card has these lines:
 1. <id> | <kind>[:<kind field>] | <standing> [(head)] | <created date> | score <n>
    "<title>"
    matched: v<n> <id>                       (only when an older version matched)
-   "…<160-character passage with the most query words>…" [chars <from>-<to> of <length>]
+   "…<best passage of the matched version>…" [chars <from>-<to> of <length>]
+   "…<second passage, when the body matches in a second place>…" [chars <from>-<to> of <length>]
+   blob <id> [(retracted)]: "…<passage of an attached Blob payload>…" [chars <from>-<to> of <length>]
    chain: v1 <id> <standing> <date> -> (n more) -> v4 <id> <standing> <date> (head)
    out: <relation> <id> "<title, 60 chars>" (<standing>), … +n more; <relation> +n
-   in: <derived_by|cited_by|resolves|rejects|mitigates|evidence_for|blob_of> <id> "<title>" (<standing>)
+   in: <resolves|rejects|mitigates|evidence_for|cited_by|blob_of|derived_by> <id> "<title>" (<standing>)
    code: <paths, symbol() names, SHAs, #numbers the chain's records name>
 ```
 

benchmarks/proof-recall/README.md

@@ -278,7 +278,9 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`.
    the `find` source. The gate passes when, on the stripped set, the
    evidence is in the top 5 for at least 75% of questions and the answer is
    on a card for at least 60% of those it applies to; if it fails, do not
-   pay for sessions.
+   pay for sessions. `--split tune` or `--split holdout` asks only half the
+   questions, chosen by a hash of each id. Read tune misses to decide what
+   to change in `find`; run holdout once the change is done.
 
 3. Run the benchmark:
 
@@ -611,3 +613,94 @@ Next steps, in order:
 2. Decide whether counting and lifecycle questions need their own read,
    such as the `proof state` view the design review proposed, instead of
    search.
+
+#### Cards that quote more
+
+`find-recall.mjs --split tune|holdout|all` now keeps half the questions
+back. A question is in `tune` when the first byte of the SHA-256 of its id
+is even, and in `holdout` when it is odd: 130 and 116 questions. Only tune
+misses were read to decide what to change in this step; holdout was scored
+once the code was done. Holdout is unseen only question by question, though:
+the plan to quote Blob payloads came from the diagnosis of all 246
+questions in the step before, made before the split existed.
+
+On the 10x graph, 43 tune questions missed the answer on the cards. Why:
+
+| Where the answer was                                        | Misses |
+| ----------------------------------------------------------- | ------ |
+| Not a hit, and not one link from a hit                      | 20     |
+| In a hit's body, outside the passage or cut off at its edge | 3      |
+| In a Blob that is a hit, outside its passage                | 3      |
+| In a Blob attached to a hit, which the card did not quote   | 2      |
+| A record linked from a hit, cut by the per-line link cap    | 3      |
+| A chain version the card left out as `(n more)`             | 3      |
+| A retracted record, which `find` leaves out by default      | 3      |
+| In a linked record's text or author                         | 2      |
+| Not written anywhere; it must be worked out                 | 2      |
+| A hit's author, which cards do not show                     | 1      |
+| On the card, but as a bare number the gate does not count   | 1      |
+
+The 1x graph gave 42 misses with the same causes. The cards changed in
+four ways:
+
+1. The best passage is the window whose query words weigh most, rare words
+   more, and words the title already shows count only to break ties. It
+   comes first on the card.
+2. The matches sit in the middle of the window, and the passage runs on to
+   finish a short word right after the last match, so a value after its
+   label stays.
+3. A second passage appears when another part of the body holds a query
+   word the first lacks, and a card quotes the best passage of a Blob that
+   the hit carries or cites through a Citation. Only a current, unretracted
+   Citation and Blob count, and a Blob quoted on one card is not quoted
+   again on a later one. Passages add their length to the size aim, so they
+   never push out a chain or a link.
+4. Reverse links list `resolves`, `rejects`, and the like before
+   `derived_by`, which on a much-used record fills the line.
+
+Showing each card's author was tried and dropped: it pushed chains off the
+cards and put a name on every card, so name answers matched by chance.
+
+The chance rates now come from every question's cards, even with
+`--split`, so each split's figures below are its part of the `all` run.
+The tables also print the largest reply next to the mean.
+
+Holdout is the clean measure. Stripped set, answer on a card, before and
+after (`find_hash`
+`e5be653299abe9b2329822d98ffe09e90ee7cfe96f8711c813a51dc6500b887f`):
+
+| Graph | Split   | Evidence in top 5 | Answer before | Answer after  | Mean reply chars | Largest reply  |
+| ----- | ------- | ----------------- | ------------- | ------------- | ---------------- | -------------- |
+| 10x   | holdout | 85% (98/116)      | 58% (51/88)   | 66% (58/88)   | 6,084 to 6,365   | 6,509 to 7,112 |
+| 10x   | tune    | 79% (102/130)     | 54% (51/94)   | 59% (53/90)   | 6,080 to 6,369   | 6,700 to 7,360 |
+| 10x   | all     | 81% (200/246)     | 56% (102/182) | 62% (111/178) | 6,082 to 6,368   | 6,700 to 7,360 |
+| 1x    | holdout | 84% (97/116)      | 57% (50/88)   | 65% (57/88)   | 6,093 to 6,411   | 6,502 to 7,056 |
+| 1x    | tune    | 79% (102/130)     | 55% (52/94)   | 59% (54/91)   | 6,085 to 6,403   | 6,698 to 7,358 |
+| 1x    | all     | 81% (199/246)     | 56% (102/182) | 62% (111/179) | 6,089 to 6,407   | 6,698 to 7,358 |
+
+Search ranking did not change, so evidence in the top 5 is the same before
+and after. On the 10x holdout, which the code was not tuned on, the answer
+is on a card for 7 more questions and for none fewer: 66% against the 60%
+bar. On the 1x holdout it is also 7 more and none fewer. The gate on all
+246 questions of the 10x graph passes by two points, but that figure
+includes the 130 tune questions, which four rounds of changes were measured
+against (55%, 56%, 57%, then 59% on the 10x tune split).
+
+The tune gain is small partly because longer cards match more answers by
+chance. On the 10x graph, 3 tune answers that were on a card are now set
+aside as chance matches (ports-001, render-003, world-065), and on the 1x
+graph 2 are. One tune miss, world-052, also left the count the same way,
+which nudges the rate up. To check that the set-aside list did not make
+the gain, one list, the union of both runs' set-asides, was applied to
+before and after: on the 10x graph the answer is on a card for 99/178 (55.6%) before
+and 111/178 (62.4%) after; holdout 51/88 to 58/88; tune 48/90 to 53/90. On
+the 1x graph, all goes from 100/179 (55.9%) to 111/179 (62.0%). One 1x tune
+answer, ports-026, was lost when two equally good passages of a Blob
+swapped places. Most misses left are records no hit reaches, mainly
+counting and lifecycle questions, which a `proof state` read should answer
+instead.
+
+The largest reply grew by about 700 characters, to 7,360. That is close
+to 7,500 characters, the most this step allowed for the mean reply. A later
+change that adds passage lines should watch the largest reply, not only the
+mean.

benchmarks/proof-recall/scripts/find-recall.mjs

@@ -3,7 +3,7 @@
 // whether its top hits hold the evidence, before any paid agent run.
 //
 //   node scripts/find-recall.mjs [--graph <built-graph-dir>] [--ids a,b]
-//        [--out <json>] [--jobs 8]
+//        [--split tune|holdout|all] [--out <json>] [--jobs 8]
 //
 // The gate verdict that counts is the one on the 10x graph, given with
 // `--graph <x10 build>`. The default, the committed 1x graph, is a quick
@@ -37,20 +37,35 @@
 //                       10% of the other questions' cards in the same set
 //                       (`chance_rate`), since a card shows those by
 //                       accident: "2", "6", "accepted". With --ids, the
-//                       rates come from the chosen questions only.
+//                       rates come from the chosen questions only. With
+//                       --split, they still come from every question, so a
+//                       split's rows are the same as its part of an `all`
+//                       run.
 //   right_effort_first  the top Effort in `efforts by match` is the
 //                       question's own (single-Effort questions, --all sets)
 //   reply_chars, cards_chars
+//                       the tables print the mean reply size and the
+//                       largest one
 //
 // The gate passes when, on the stripped set, evidence_top5 holds for at least
 // 75% of questions and answer_on_card for at least 60% of those it applies
 // to. Questions are read only to score; nothing here tunes `find`. The output
 // records `find_hash`, a hash of the `find` source, so a later run file can
 // be checked against the search this gate measured.
 //
+// `--split` keeps half the questions back. A question belongs to `tune` when
+// the first byte of the SHA-256 of its id is even, and to `holdout` when it is
+// odd, so the split never changes and needs no list. Read the misses of the
+// tune split to decide what to change in `find`; run the holdout split only
+// after the change is done, to see whether it holds on questions nobody
+// looked at. The default, `all`, scores every question. Every split asks
+// every question, because the chance rates above need all the cards; the
+// rows outside the split are dropped after that.
+//
 // The graph is copied, without its key map, into a temporary folder, and one
 // `proof serve` there answers every query. The folder is removed at the end.
 import { spawn } from 'node:child_process';
+import { createHash } from 'node:crypto';
 import { mkdtemp, rm, writeFile } from 'node:fs/promises';
 import { request } from 'node:http';
 import { tmpdir } from 'node:os';
@@ -73,22 +88,34 @@ const GATE = { evidence_top5: 0.75, answer_on_card: 0.6 };
 // A value answer that matches more than this share of other questions' cards
 // is set aside: a card shows it by accident.
 const CHANCE_CUTOFF = 0.1;
+const SPLITS = ['tune', 'holdout', 'all'];
+
+/** `tune` or `holdout`, from the first byte of the SHA-256 of the id. */
+export function splitOf(id) {
+  return createHash('sha256').update(id).digest()[0] % 2 === 0
+    ? 'tune'
+    : 'holdout';
+}
 
 function args(argv) {
   const out = {
     graph: join(BENCH, '.flatbread-proof'),
     ids: [],
     out: join(BENCH, 'runs', 'find-recall.json'),
     jobs: 8,
+    split: 'all',
   };
   for (let i = 0; i < argv.length; i++) {
     const flag = argv[i];
     if (flag === '--graph') out.graph = resolve(argv[++i]);
     else if (flag === '--ids') out.ids = argv[++i].split(',');
     else if (flag === '--out') out.out = resolve(argv[++i]);
     else if (flag === '--jobs') out.jobs = Number(argv[++i]);
+    else if (flag === '--split') out.split = argv[++i];
     else throw new Error(`Unknown argument ${flag}`);
   }
+  if (!SPLITS.includes(out.split))
+    throw new Error(`--split must be one of ${SPLITS.join(', ')}`);
   return out;
 }
 
@@ -268,6 +295,7 @@ function tally(rows) {
     mean_reply_chars: Math.round(
       chars.reduce((a, b) => a + b, 0) / (chars.length || 1)
     ),
+    max_reply_chars: Math.max(0, ...chars),
   };
 }
 
@@ -281,7 +309,8 @@ function printTable(label, groups) {
       'ev@5'.padStart(7) +
       'card'.padStart(7) +
       'effort1'.padStart(9) +
-      'chars'.padStart(8)
+      'chars'.padStart(8) +
+      'max'.padStart(7)
   );
   for (const [name, t] of Object.entries(groups))
     console.log(
@@ -290,7 +319,8 @@ function printTable(label, groups) {
         pct(t.evidence_top5).padStart(7) +
         pct(t.answer_on_card).padStart(7) +
         pct(t.right_effort_first).padStart(9) +
-        String(t.mean_reply_chars).padStart(8)
+        String(t.mean_reply_chars).padStart(8) +
+        String(t.max_reply_chars).padStart(7)
     );
 }
 
@@ -365,14 +395,21 @@ async function main() {
   }
 
   setAsideChance(rows);
+  const inSplit = (id) =>
+    options.split === 'all' || splitOf(id) === options.split;
+  rows.splice(0, rows.length, ...rows.filter((r) => inSplit(r.id)));
   const order = new Map(jobs.map((job, i) => [`${job.set} ${job.q.id}`, i]));
   rows.sort(
     (a, b) => order.get(`${a.set} ${a.id}`) - order.get(`${b.set} ${b.id}`)
   );
   const bySet = Object.fromEntries(
     SETS.map((set) => [set, tally(rows.filter((r) => r.set === set))])
   );
-  printTable(`graph ${options.graph}`, bySet);
+  const questions = new Set(rows.map((r) => r.id)).size;
+  printTable(
+    `graph ${options.graph}, split ${options.split} (${questions} questions)`,
+    bySet
+  );
   console.log(
     `Value answers set aside by chance: ${SETS.map(
       (set) => `${set} ${bySet[set].set_aside_by_chance}`
@@ -407,6 +444,8 @@ async function main() {
     `${JSON.stringify(
       {
         graph: options.graph,
+        split: options.split,
+        questions,
         find_hash,
         gate: { ...GATE, chance_cutoff: CHANCE_CUTOFF, pass },
         bySet,