diff --git a/benchmarks/proof-recall/README.md b/benchmarks/proof-recall/README.md index 7c93c600..7f719e34 100644 --- a/benchmarks/proof-recall/README.md +++ b/benchmarks/proof-recall/README.md @@ -23,6 +23,7 @@ Each question has a known answer and a stated path through the graph. | `scripts/run.mjs` | Runs one headless agent session per question | | `scripts/task.mjs` | The task text each session reads | | `scripts/compare.mjs` | Compares two runs question by question | +| `scripts/find-recall.mjs` | Checks offline whether `proof find` reaches the evidence | | `runs/` | Answers and costs from each pilot and full run | | `.flatbread-proof/` | The built graph, committed so runs need no build | | `flatbread.config.js` | Lets `flatbread proof …` read the graph from this folder | @@ -251,17 +252,47 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`. 1. Build the workspace (`pnpm build` at the repository root), so the `flatbread` binary exists. -2. Run the benchmark: +2. Before a paid run that tests `proof find`, run the free offline gate: + `node scripts/find-recall.mjs --graph [--out ]`. + The gate verdict that counts is the one on the 10x graph; with no + `--graph` the script checks the committed 1x graph and says so. It asks + the real `proof find` every question three ways: the raw text with + `--all`; the text with Effort titles, the words that name an Effort right + before "Effort" (as in "the co-op netcode Effort"), the code names + Kestrel and Lantern, and the platform names removed, with `--all`; and + the raw text with `--effort` set to the question's own Effort. Topic + words that an Effort title also uses, such as "build farm", stay in the + stripped text by design. The plan's third set, queries taken from pilot + transcripts, does not exist yet, because no pilot ran `proof find`; the + scoped set stands in for it. + + For each query the script records whether a top-5 hit, or the version it + matched, is an evidence or answer record; whether the answer appears on + the cards; and whether the right Effort ranks first. A value answer + counts only as a whole token, never as its number without its unit + ("61" for "61 MB/s"), and one that matches more than 10% of the + other questions' cards (such as "2" or "accepted", which dates, counts, + and standings print everywhere) is set aside, since a card shows it by + chance. The script prints a table and writes per-question rows to + `--out` (default `runs/find-recall.json`), with `find_hash`, a hash of + the `find` source. The gate passes when, on the stripped set, the + evidence is in the top 5 for at least 75% of questions and the answer is + on a card for at least 60% of those it applies to; if it fails, do not + pay for sessions. + +3. Run the benchmark: ```bash node scripts/run.mjs /tmp/recall --name --model haiku --jobs 12 ``` `run.mjs` gives every question its own folder and its own fresh, - headless `claude -p` session, once per strategy (`brief` first, or - no `brief`). Each folder holds the graph without the key map, the - config, a `./flatbread` wrapper, `TASK.md`, and, afterwards, the - agent's `answer.json` and `transcript.jsonl`. Sessions get only the + headless `claude -p` session, once per strategy. `--strategies` picks + them from `brief` (start from `proof brief`), `nobrief`, and `find` + (start from `proof find`); the default is `brief,nobrief`. Each folder + holds the graph without the key map, the config, a `./flatbread` + wrapper, `TASK.md`, and, afterwards, the agent's `answer.json` and + `transcript.jsonl`. Sessions get only the Bash, Read, and Write tools, no MCP servers, and no settings files. Runs share one read-only copy of each built graph and one `flatbread proof serve` process, both under @@ -274,15 +305,28 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`. `--no-serve` makes reads run locally. Keep the work folder outside the repository. A rerun skips finished sessions. When every session ends, `run.mjs` writes - `runs/-brief.json` and `runs/-nobrief.json`, taking tokens - and tool calls from the transcripts. + `runs/-.json` for each strategy, taking tokens and tool + calls from the transcripts. Each run file records the same `find_hash` + as the gate, so you can check that both used the same search. -3. Score each run with `node scripts/grade.mjs runs/-brief.json`. -4. Compare the two strategies question by question with +4. Score each run with `node scripts/grade.mjs runs/-brief.json`. +5. Compare two strategies question by question with `node scripts/compare.mjs runs/-brief.json runs/-nobrief.json`. It reports both scores, the questions only one got right, a sign test on those, and read tokens, which leave out each session's fixed first turn. +6. To pool repeated runs, give each side a comma-separated list of run + files, in the same order: + `node scripts/compare.mjs runs/r1-find.json,runs/r2-find.json runs/r1-nobrief.json,runs/r2-nobrief.json`. + Run i of one side is paired with run i of the other, and each question + in each pair is one row, named `#` in the lists of one-side wins. + `signTestP` counts every disagreement across all the runs. That treats + repeats as independent, which holds only for separate runs of fresh + sessions, so the script refuses a file listed twice; and it says how + sure we can be about these questions, not about new ones. + `questionSignTest` counts each question once, as a win for the side + with more right answers over the runs, and is the stricter reading. The + script warns when the run files record different `find_hash` values. Three rules shaped this protocol, each learned from a pilot below: @@ -535,3 +579,35 @@ Next steps, in order: graph. Give-ups, not wrong answers, are what grew. 2. Look at the questions `brief` agents gave up on, to see whether the summary steers agents away from records they should read. + +### The first offline gate for `proof find` + +`proof find` now exists, and `scripts/find-recall.mjs` checks it for free +before any paid run. On the 10x graph, with Effort titles and platform +names removed from each question: + +| Check | Result | Bar | +| --------------------------------------- | ---------------- | --- | +| An evidence record is in the top 5 hits | 81% (200 of 246) | 75% | +| The answer itself appears on a card | 56% (102 of 182) | 60% | +| The right Effort is ranked first | 81% | | + +The gate fails on the second check, so no paid run tested `find` yet. The +1x graph gives the same numbers within a point, which fits the worry that +filler is easy to tell apart from real records. + +Where cards miss the answer: + +- **Blob payloads (22%).** `find` reaches the record every time (100%), + but the value sits in the attached payload, which the card does not quote. +- **Aggregates (24%) and lifecycle (46%).** These ask for counts or states + across many records, which a search cannot give. +- **Retraction (40%).** `find` leaves retracted records out by default. + +Next steps, in order: + +1. Quote the matching passage of a linked Blob payload on the card, and + rerun the gate. +2. Decide whether counting and lifecycle questions need their own read, + such as the `proof state` view the design review proposed, instead of + search. diff --git a/benchmarks/proof-recall/scripts/collect.mjs b/benchmarks/proof-recall/scripts/collect.mjs index ab26011e..9b0b7676 100644 --- a/benchmarks/proof-recall/scripts/collect.mjs +++ b/benchmarks/proof-recall/scripts/collect.mjs @@ -9,9 +9,27 @@ // Each folder holds the agent's `answer.json`: { "answer": ... }. Transcripts // are Claude Code agent logs or `claude -p --output-format stream-json` // output (JSON lines); each assistant turn carries usage. +import { createHash } from 'node:crypto'; import { readFile, writeFile } from 'node:fs/promises'; -import { join, resolve } from 'node:path'; -import { pathToFileURL } from 'node:url'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; + +// The `proof find` source: its tokenizer and BM25 weights. +const FIND_SOURCE = resolve( + dirname(fileURLToPath(import.meta.url)), + '../../../packages/flatbread/src/proof/find.ts' +); + +/** + * A sha256 of the `proof find` source, so a run file and the offline gate + * can show they used the same search. It hashes the source, not the build, + * so rebuild after any change to it. + */ +export async function findHash() { + return createHash('sha256') + .update(await readFile(FIND_SOURCE)) + .digest('hex'); +} /** * `context_tokens` counts what entered the context once (new input, cache @@ -59,8 +77,11 @@ export async function sessionCost(transcriptPath) { return { ...cost, models: [...models] }; } -/** Reads each session's answer and cost, and writes one run file. */ -export async function collect(manifest, name, outPath) { +/** + * Reads each session's answer and cost, and writes one run file. `about` + * adds fields to the file's `run` header, such as `find_hash`. + */ +export async function collect(manifest, name, outPath, about = {}) { const answers = {}; const models = new Set(); for (const session of manifest) { @@ -88,6 +109,7 @@ export async function collect(manifest, name, outPath) { name, protocol: 'one fresh session per question', models: [...models], + ...about, }, answers, }; diff --git a/benchmarks/proof-recall/scripts/compare.mjs b/benchmarks/proof-recall/scripts/compare.mjs index ada71d5b..ca7f8e26 100644 --- a/benchmarks/proof-recall/scripts/compare.mjs +++ b/benchmarks/proof-recall/scripts/compare.mjs @@ -2,7 +2,18 @@ // Compares two runs over the questions both answered, pair by pair. // // node scripts/compare.mjs runs/.json runs/.json [--skip id1,id2] +// node scripts/compare.mjs runs/a-r1.json,runs/a-r2.json \ +// runs/b-r1.json,runs/b-r2.json // +// With several files on each side, the runs are pooled: run i of side A is +// paired with run i of side B, and each question in each pair of runs is one +// row, named `#` in the lists of questions only one side got right. +// `signTestP` then counts every disagreement across the runs. That is sound +// only when each file is its own run of fresh sessions, so the script refuses +// a file listed twice, and it speaks only for these questions. To ask whether +// the gap would hold on new questions, `questionSignTest` counts each +// question once: a win for the side with more right answers over the runs. +// If the run files record different `find_hash` values, a warning says so. // Read tokens leave out each session's first turn (`fixed_tokens`), which is // the same task and system prompt in both runs. A sign test on the questions // that only one run got right says how likely the accuracy gap is by chance. @@ -44,35 +55,67 @@ async function main() { const [pathA, pathB] = argv; if (!pathB) throw new Error( - 'usage: compare.mjs [--skip ids]' + 'usage: compare.mjs [,more] [,more] [--skip ids]' ); - const [a, b] = await Promise.all( - [pathA, pathB].map(async (p) => - JSON.parse(await readFile(resolve(p), 'utf8')) - ) + const load = (paths) => + Promise.all( + paths + .split(',') + .map(async (p) => JSON.parse(await readFile(resolve(p), 'utf8'))) + ); + const files = [...pathA.split(','), ...pathB.split(',')].map((p) => + resolve(p) ); + if (new Set(files).size !== files.length) + throw new Error('Each run file may appear only once'); + const [runsA, runsB] = await Promise.all([load(pathA), load(pathB)]); + if (runsA.length !== runsB.length) + throw new Error('Give each side the same number of run files'); + const hashes = new Set( + [...runsA, ...runsB].map((run) => run.run?.find_hash).filter(Boolean) + ); + if (hashes.size > 1) + console.warn( + `Warning: the run files used different proof find code: ${[ + ...hashes, + ].join(', ')}` + ); + const pooled = runsA.length > 1; + const name = (runs, paths) => + pooled + ? runs.map((run, i) => run.run?.name ?? paths.split(',')[i]).join(',') + : runs[0].run?.name ?? paths; const graph = await loadGraph(join(BENCH, '.flatbread-proof')); + const sets = await loadQuestions(); const rows = []; - for (const set of await loadQuestions()) - for (const q of set.questions) { - const x = a.answers?.[q.id]; - const y = b.answers?.[q.id]; - if (!x || !y || skip.has(q.id)) continue; - rows.push({ - id: q.id, - tier: q.tier, - a: score(graph, q, set.effort, x.answer), - b: score(graph, q, set.effort, y.answer), - readA: x.tokens - (x.fixed_tokens ?? 0), - readB: y.tokens - (y.fixed_tokens ?? 0), - toolsA: x.tool_calls, - toolsB: y.tool_calls, - }); - } + for (const [i, a] of runsA.entries()) + for (const set of sets) + for (const q of set.questions) { + const x = a.answers?.[q.id]; + const y = runsB[i].answers?.[q.id]; + if (!x || !y || skip.has(q.id)) continue; + rows.push({ + id: pooled ? `${q.id}#${i + 1}` : q.id, + question: q.id, + tier: q.tier, + a: score(graph, q, set.effort, x.answer), + b: score(graph, q, set.effort, y.answer), + readA: x.tokens - (x.fixed_tokens ?? 0), + readB: y.tokens - (y.fixed_tokens ?? 0), + toolsA: x.tool_calls, + toolsB: y.tool_calls, + }); + } if (!rows.length) throw new Error('The runs share no questions'); const onlyA = rows.filter((r) => r.a === 1 && r.b < 1).map((r) => r.id); const onlyB = rows.filter((r) => r.b === 1 && r.a < 1).map((r) => r.id); const total = (key) => +rows.reduce((n, r) => n + r[key], 0).toFixed(2); + // Per question, the sum over runs of A's score minus B's. + const margin = new Map(); + for (const r of rows) + margin.set(r.question, (margin.get(r.question) ?? 0) + r.a - r.b); + const questionWins = (sign) => + [...margin.values()].filter((m) => Math.sign(m) === sign).length; const byTier = {}; for (const r of rows) { const t = (byTier[`tier-${r.tier}`] ??= { n: 0, a: 0, b: 0 }); @@ -83,14 +126,24 @@ async function main() { console.log( JSON.stringify( { - a: a.run?.name ?? pathA, - b: b.run?.name ?? pathB, + a: name(runsA, pathA), + b: name(runsB, pathB), + ...(pooled ? { runs: runsA.length } : {}), questions: rows.length, correct: { a: total('a'), b: total('b') }, byTier, onlyA, onlyB, signTestP: +signTest(onlyA.length, onlyB.length).toFixed(4), + ...(pooled + ? { + questionSignTest: { + a: questionWins(1), + b: questionWins(-1), + p: +signTest(questionWins(1), questionWins(-1)).toFixed(4), + }, + } + : {}), readTokens: { a: summary(rows.map((r) => r.readA)), b: summary(rows.map((r) => r.readB)), diff --git a/benchmarks/proof-recall/scripts/find-recall.mjs b/benchmarks/proof-recall/scripts/find-recall.mjs new file mode 100644 index 00000000..c8a0c3b9 --- /dev/null +++ b/benchmarks/proof-recall/scripts/find-recall.mjs @@ -0,0 +1,426 @@ +#!/usr/bin/env node +// The free offline gate: asks the real `proof find` every question and checks +// whether its top hits hold the evidence, before any paid agent run. +// +// node scripts/find-recall.mjs [--graph ] [--ids a,b] +// [--out ] [--jobs 8] +// +// The gate verdict that counts is the one on the 10x graph, given with +// `--graph `. The default, the committed 1x graph, is a quick +// check; the script says so when the graph holds fewer than 1000 records. +// +// Three query sets, each one `find` per question: +// raw the question text, with --all +// stripped the question text without Effort titles, the words just before +// "Effort" that name one ("the co-op netcode Effort"), Effort code +// names, or platform names, with --all. Topic words that an +// Effort title also uses, such as "build farm", stay elsewhere. +// scoped the question text, with --effort set to the question's Effort +// (cross-Effort questions are skipped) +// The plan asked for a third set of queries taken from pilot transcripts. +// No pilot ran `proof find`, so none exist; `scoped` stands in until a find +// pilot gives real agent queries. +// +// Per question and set it records: +// evidence_top5 a top-5 hit's id or matched_id is one of the +// question's evidence records or answer records. Only +// the hit itself and the version `find` matched count, +// not older versions in its chain, because a card may +// elide those ("(2 more)"). +// answer_on_card record answers: every answer id appears in the +// cards. Value answers: the gold value or an accepted +// alternative appears as a whole token, ignoring case; +// an alternative that is the gold's number without its +// unit ("61" for "61 MB/s") is dropped. Null for count, +// boolean, text, and "unknown" answers, which no card +// can show, and for value answers that match more than +// 10% of the other questions' cards in the same set +// (`chance_rate`), since a card shows those by +// accident: "2", "6", "accepted". With --ids, the +// rates come from the chosen questions only. +// right_effort_first the top Effort in `efforts by match` is the +// question's own (single-Effort questions, --all sets) +// reply_chars, cards_chars +// +// The gate passes when, on the stripped set, evidence_top5 holds for at least +// 75% of questions and answer_on_card for at least 60% of those it applies +// to. Questions are read only to score; nothing here tunes `find`. The output +// records `find_hash`, a hash of the `find` source, so a later run file can +// be checked against the search this gate measured. +// +// The graph is copied, without its key map, into a temporary folder, and one +// `proof serve` there answers every query. The folder is removed at the end. +import { spawn } from 'node:child_process'; +import { mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { request } from 'node:http'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { loadQuestions } from './check.mjs'; +import { findHash } from './collect.mjs'; +import { loadGraph, qualify } from './graph.mjs'; +import { prepareFolder } from './prepare-run.mjs'; + +const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const SETS = ['raw', 'stripped', 'scoped']; +// Names that point at one Effort or platform without its full title. The +// Effort titles themselves come from the graph. +const NAMES = ['Kestrel', 'Lantern', 'PC', 'Meridian', 'Skiff']; +// Words that name an Effort when they stand right before "Effort", besides +// the words of its title: "the render Effort", "the world Effort". +const EFFORT_WORDS = ['render', 'world', 'ports', 'netcode', 'tools']; +const GATE = { evidence_top5: 0.75, answer_on_card: 0.6 }; +// A value answer that matches more than this share of other questions' cards +// is set aside: a card shows it by accident. +const CHANCE_CUTOFF = 0.1; + +function args(argv) { + const out = { + graph: join(BENCH, '.flatbread-proof'), + ids: [], + out: join(BENCH, 'runs', 'find-recall.json'), + jobs: 8, + }; + for (let i = 0; i < argv.length; i++) { + const flag = argv[i]; + if (flag === '--graph') out.graph = resolve(argv[++i]); + else if (flag === '--ids') out.ids = argv[++i].split(','); + else if (flag === '--out') out.out = resolve(argv[++i]); + else if (flag === '--jobs') out.jobs = Number(argv[++i]); + else throw new Error(`Unknown argument ${flag}`); + } + return out; +} + +const sleep = (ms) => new Promise((done) => setTimeout(done, ms)); + +function serverAnswers(socket) { + return new Promise((answer) => { + const req = request({ socketPath: socket, path: '/' }, (res) => { + res.resume(); + answer(true); + }); + req.on('error', () => answer(false)); + req.end(); + }); +} + +/** Runs `./flatbread` in `folder`; resolves with stdout. */ +function flatbread(folder, argv, env = {}) { + return new Promise((done, fail) => { + const child = spawn(join(folder, 'flatbread'), argv, { + cwd: folder, + env: { ...process.env, ...env }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (chunk) => (stdout += chunk)); + child.stderr.on('data', (chunk) => (stderr += chunk)); + child.on('close', (code) => + code === 0 + ? done(stdout) + : fail(new Error(`flatbread ${argv.join(' ')}: ${stderr}`)) + ); + }); +} + +const escape = (text) => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + +/** Question text with every Effort title and short name taken out. */ +export function stripNames(text, titles) { + const nameWords = new Set([ + ...titles.flatMap((title) => title.toLowerCase().split(/\s+/)), + ...EFFORT_WORDS, + ]); + // "the co-op netcode Effort" becomes "the Effort". + let out = text.replace(/(?:[\w-]+ )+(?=Effort\b)/g, (run) => { + const words = run.trim().split(' '); + while (words.length && nameWords.has(words.at(-1).toLowerCase())) + words.pop(); + return words.length ? `${words.join(' ')} ` : ''; + }); + for (const name of [...titles, ...NAMES]) + out = out.replace(new RegExp(`\\b${escape(name)}(?:'s)?\\b`, 'gi'), ' '); + return out + .replace(/"\s*"/g, ' ') + .replace(/\s+/g, ' ') + .replace(/ ([?.,;:!])/g, '$1') + .trim(); +} + +const list = (value) => + Array.isArray(value) ? value : value === undefined ? [] : [value]; + +/** Record ids that count as evidence, and the answer ids, for a question. */ +function targets(graph, effort, q) { + const id = (key) => graph.keymap[qualify(effort, key)]; + const answers = ['record', 'records'].includes(q.answer.kind) + ? list(q.answer.value).map(id) + : []; + return { + evidence: new Set([...(q.evidence ?? []).map(id), ...answers]), + answers, + }; +} + +const bareNumber = (text) => /^[\d.,]+$/.test(text); +const number = (text) => Number(text.replace(/,/g, '')); + +/** + * Patterns for the forms of a value answer a card may show: the gold and its + * accepted alternatives, each as a whole token, so "6" does not match "16" + * or "6.2". Null when no card can show the answer. + */ +function valueForms(q) { + const { kind, value, accept } = q.answer; + if (kind !== 'value' || String(value).toLowerCase() === 'unknown') + return null; + const gold = String(value); + // "61" for "61 MB/s" is the gold without its unit, and too easy to match; + // "0.52" for "52 percent" is another way to write it, and stays. + const lead = bareNumber(gold) ? null : gold.match(/^[\d.,]+/)?.[0]; + const unitless = (form) => + Boolean(lead) && bareNumber(form) && number(form) === number(lead); + const forms = [gold, ...(accept ?? []).map(String)].filter( + (form) => !unitless(form) + ); + return forms.map( + (form) => new RegExp(`(? cards.includes(id)); + const forms = valueForms(q); + return forms ? forms.some((form) => form.test(cards)) : null; +} + +/** + * For each value answer, the share of the other questions' cards in the same + * set that it matches. Above CHANCE_CUTOFF, its answer_on_card becomes null. + */ +function setAsideChance(rows) { + for (const row of rows) { + const forms = valueForms(row.q); + if (!forms) continue; + const others = rows.filter((r) => r.set === row.set && r.id !== row.id); + if (!others.length) continue; + const hits = others.filter((r) => + forms.some((form) => form.test(r.cards)) + ).length; + row.chance_rate = +(hits / others.length).toFixed(3); + if (row.chance_rate > CHANCE_CUTOFF) row.answer_on_card = null; + } + for (const row of rows) { + delete row.q; + delete row.cards; + } +} + +function scoreReply(graph, effortIds, row, stdout) { + const reply = JSON.parse(stdout); + const top5 = reply.find.hits.slice(0, 5); + const { evidence, answers } = targets(graph, row.effort, row.q); + return { + id: row.q.id, + set: row.set, + effort: row.effort, + categories: row.q.categories, + query: row.query, + evidence_top5: top5.some( + (hit) => evidence.has(hit.id) || evidence.has(hit.matched_id) + ), + answer_on_card: answerOnCard(row.q, answers, reply.cards), + right_effort_first: + row.set === 'scoped' || row.effort === 'cross' + ? null + : reply.find.efforts?.[0]?.id === effortIds[row.effort], + reply_chars: stdout.length, + cards_chars: reply.cards.length, + hits: top5.map((hit) => hit.matched_id ?? hit.id), + chance_rate: null, + // Kept until every reply is in, to measure chance matches; then dropped. + q: row.q, + cards: reply.cards, + }; +} + +function rate(rows, field) { + const known = rows.filter((r) => r[field] !== null); + if (!known.length) return { n: 0, rate: null }; + const yes = known.filter((r) => r[field]).length; + return { n: known.length, yes, rate: +(yes / known.length).toFixed(3) }; +} + +function tally(rows) { + const chars = rows.map((r) => r.reply_chars); + return { + n: rows.length, + evidence_top5: rate(rows, 'evidence_top5'), + answer_on_card: rate(rows, 'answer_on_card'), + set_aside_by_chance: rows.filter((r) => r.chance_rate > CHANCE_CUTOFF) + .length, + right_effort_first: rate(rows, 'right_effort_first'), + mean_reply_chars: Math.round( + chars.reduce((a, b) => a + b, 0) / (chars.length || 1) + ), + }; +} + +const pct = (r) => (r.rate === null ? ' -' : `${Math.round(r.rate * 100)}%`); + +function printTable(label, groups) { + console.log(`\n${label}`); + console.log( + 'group'.padEnd(24) + + 'n'.padStart(5) + + 'ev@5'.padStart(7) + + 'card'.padStart(7) + + 'effort1'.padStart(9) + + 'chars'.padStart(8) + ); + for (const [name, t] of Object.entries(groups)) + console.log( + name.padEnd(24) + + String(t.n).padStart(5) + + pct(t.evidence_top5).padStart(7) + + pct(t.answer_on_card).padStart(7) + + pct(t.right_effort_first).padStart(9) + + String(t.mean_reply_chars).padStart(8) + ); +} + +async function main() { + const options = args(process.argv.slice(2)); + const graph = await loadGraph(options.graph); + const effortIds = Object.fromEntries( + Object.entries(graph.keymap) + .filter(([key]) => key.endsWith('.effort')) + .map(([key, id]) => [key.split('.')[0], id]) + ); + const titles = Object.values(effortIds).map( + (id) => graph.records.get(id).frontmatter.title + ); + if (graph.records.size < 1000) + console.log( + `Note: this graph holds ${graph.records.size} records. The gate verdict ` + + 'that counts is the one on the 10x graph (--graph ).' + ); + const find_hash = await findHash(); + const wanted = new Set(options.ids); + const jobs = []; + for (const set of await loadQuestions()) + for (const q of set.questions) { + if (wanted.size && !wanted.has(q.id)) continue; + const row = { q, effort: set.effort }; + jobs.push({ ...row, set: 'raw', query: q.question, scope: ['--all'] }); + jobs.push({ + ...row, + set: 'stripped', + query: stripNames(q.question, titles), + scope: ['--all'], + }); + if (set.effort !== 'cross') + jobs.push({ + ...row, + set: 'scoped', + query: q.question, + scope: ['--effort', effortIds[set.effort]], + }); + } + + const folder = await mkdtemp(join(tmpdir(), 'find-recall-')); + const socket = join(folder, 'serve.sock'); + await prepareFolder(folder, [], options.graph); + const server = spawn( + join(folder, 'flatbread'), + ['proof', 'serve', '--socket', socket], + { cwd: folder, stdio: 'ignore' } + ); + const rows = []; + try { + for (let i = 0; !(await serverAnswers(socket)); i++) { + if (i > 600) throw new Error('proof serve did not start'); + await sleep(200); + } + const todo = [...jobs]; + const worker = async () => { + for (let job = todo.shift(); job; job = todo.shift()) { + const stdout = await flatbread( + folder, + ['proof', 'find', job.query, ...job.scope], + { FLATBREAD_PROOF_SERVER: socket } + ); + rows.push(scoreReply(graph, effortIds, job, stdout)); + } + }; + await Promise.all(Array.from({ length: options.jobs }, worker)); + } finally { + server.kill(); + await rm(folder, { recursive: true, force: true }); + } + + setAsideChance(rows); + const order = new Map(jobs.map((job, i) => [`${job.set} ${job.q.id}`, i])); + rows.sort( + (a, b) => order.get(`${a.set} ${a.id}`) - order.get(`${b.set} ${b.id}`) + ); + const bySet = Object.fromEntries( + SETS.map((set) => [set, tally(rows.filter((r) => r.set === set))]) + ); + printTable(`graph ${options.graph}`, bySet); + console.log( + `Value answers set aside by chance: ${SETS.map( + (set) => `${set} ${bySet[set].set_aside_by_chance}` + ).join(', ')}` + ); + const byCategory = {}; + for (const set of SETS) { + const groups = {}; + for (const r of rows.filter((row) => row.set === set)) + for (const c of r.categories) (groups[c] ??= []).push(r); + byCategory[set] = Object.fromEntries( + Object.entries(groups) + .sort(([a], [b]) => a.localeCompare(b)) + .map(([c, group]) => [c, tally(group)]) + ); + } + printTable('stripped set, by category', byCategory.stripped); + + const s = bySet.stripped; + const pass = + s.evidence_top5.rate >= GATE.evidence_top5 && + s.answer_on_card.rate >= GATE.answer_on_card; + console.log( + `\nGate ${pass ? 'PASS' : 'FAIL'}: stripped evidence_top5 ${pct( + s.evidence_top5 + )} (needs ${GATE.evidence_top5 * 100}%), answer_on_card ${pct( + s.answer_on_card + )} (needs ${GATE.answer_on_card * 100}%)` + ); + await writeFile( + options.out, + `${JSON.stringify( + { + graph: options.graph, + find_hash, + gate: { ...GATE, chance_cutoff: CHANCE_CUTOFF, pass }, + bySet, + byCategory, + rows, + }, + null, + 2 + )}\n` + ); + console.log(`find_hash ${find_hash}\nWrote ${options.out}`); +} + +main().catch((error) => { + console.error(error.message); + process.exitCode = 1; +}); diff --git a/benchmarks/proof-recall/scripts/run.mjs b/benchmarks/proof-recall/scripts/run.mjs index 428936c2..7afbbed1 100644 --- a/benchmarks/proof-recall/scripts/run.mjs +++ b/benchmarks/proof-recall/scripts/run.mjs @@ -3,17 +3,19 @@ // and per strategy, then collects answers and cost into run files. // // node scripts/run.mjs --name [--model haiku] -// [--strategies brief,nobrief] [--only render] [--ids a,b] +// [--strategies brief,nobrief,find] [--only render] [--ids a,b] // [--jobs 8] [--timeout 600] [--graph ] // [--server ] [--no-serve] // node scripts/run.mjs --stop-server [--server ] // -// Each session works in ///, which holds a link to -// the shared copy of the graph, TASK.md, the agent's answer.json, and -// transcript.jsonl. A rerun skips sessions that already have both an answer -// and a transcript, so an interrupted run picks up where it stopped. Keep -// outside the repository, so no CLAUDE.md or AGENTS.md reaches the -// agent. +// --strategies names the task texts in task.mjs to run; the default is +// brief,nobrief. Each session works in ///, which +// holds a link to the shared copy of the graph, TASK.md, the agent's +// answer.json, and transcript.jsonl. A rerun skips sessions that already +// have both an answer and a transcript, so an interrupted run picks up where +// it stopped. Keep outside the repository, so no CLAUDE.md or +// AGENTS.md reaches the agent. Each run file records `find_hash`, a hash of +// the `proof find` source, to match against the offline gate's output. // // Runs share two things, kept under /flatbread-proof-recall/: // @@ -46,7 +48,7 @@ import { request } from 'node:http'; import { tmpdir } from 'node:os'; import { dirname, join, relative, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; -import { collect } from './collect.mjs'; +import { collect, findHash } from './collect.mjs'; import { copyPublicGraph, prepareFolder, @@ -88,7 +90,7 @@ function args(argv) { dir: null, name: null, model: 'haiku', - strategies: STRATEGIES, + strategies: ['brief', 'nobrief'], only: [], ids: [], jobs: 8, @@ -116,6 +118,11 @@ function args(argv) { else throw new Error(`Unknown argument ${flag}`); } if (out.stopServer) return out; + for (const strategy of out.strategies) + if (!STRATEGIES.includes(strategy)) + throw new Error( + `Unknown strategy ${strategy}; choose from ${STRATEGIES.join(',')}` + ); if (!out.dir || !out.name) throw new Error('usage: run.mjs --name [options]'); if (out.dir.startsWith(resolve(BENCH, '..', '..') + '/')) @@ -311,8 +318,9 @@ async function main() { const options = args(process.argv.slice(2)); if (options.stopServer) return stopServer(options.server); const questions = await publicQuestions(options); + const find_hash = await findHash(); // Alternate strategies question by question, so a change in API speed or - // load over the run falls on both strategies alike. + // load over the run falls on every strategy alike. const sessions = []; for (const q of questions) for (const strategy of options.strategies) @@ -370,7 +378,8 @@ async function main() { transcript: join(s.folder, 'transcript.jsonl'), })), `${options.name}-${strategy}`, - join(BENCH, 'runs', `${options.name}-${strategy}.json`) + join(BENCH, 'runs', `${options.name}-${strategy}.json`), + { find_hash } ); } diff --git a/benchmarks/proof-recall/scripts/task.mjs b/benchmarks/proof-recall/scripts/task.mjs index 41854ca4..abc7bbe4 100644 --- a/benchmarks/proof-recall/scripts/task.mjs +++ b/benchmarks/proof-recall/scripts/task.mjs @@ -2,16 +2,23 @@ // commands its strategy allows, and one question. Never the answer. import { join } from 'node:path'; -export const STRATEGIES = ['brief', 'nobrief']; +export const STRATEGIES = ['brief', 'nobrief', 'find']; const BRIEF_LINE = '- `./flatbread proof brief `: an Effort\'s latest summary (a "compaction") in full, plus every record that changed since. Its envelope\'s `compaction.stale` is true when a record the summary relies on has changed. Start each Effort you need here.'; +const FIND_LINE = + "- `./flatbread proof find [--all | --effort ] [--kinds k1,k2] [--as-of iso] [--retracted include|only] [--limit 5]` ranks records by the words they contain, counting all versions of a record as one hit. Its envelope's `cards` field shows each hit's id, standing, best-matching passage, versions, and links."; +const NO_BRIEF = + '\nDo not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n'; +const FIND_FIRST = + '\nStart with `find`. Pass `--all` when the question names no Effort, and `--effort ` when it does (`list` gives Effort ids). Run `get` on a hit to read its full text. Use `records` only to count records or to page through them. Do not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n'; /** Markdown for TASK.md in `folder`, for one question and one strategy. */ export function taskText({ folder, question, format, strategy }) { if (!STRATEGIES.includes(strategy)) throw new Error(`Unknown strategy ${strategy}`); const brief = strategy === 'brief'; + const find = strategy === 'find'; return `# One-question recall test Answer one question about a project-memory graph, using only its read commands. Every command you run is counted, so be efficient, but a right answer matters most. @@ -22,18 +29,16 @@ Answer one question about a project-memory graph, using only its read commands. ## Read commands Run from the folder. Each prints a JSON envelope; its \`artifact_path\` names a Markdown digest. Read or grep that digest for evidence. -- \`./flatbread proof list --status active,paused,completed,abandoned\` lists Efforts (the default shows only active ones). +${ + find ? FIND_LINE + '\n' : '' +}- \`./flatbread proof list --status active,paused,completed,abandoned\` lists Efforts (the default shows only active ones). ${ brief ? BRIEF_LINE + '\n' : '' }- \`./flatbread proof records [--kinds issue,finding,decision,constraint,risk,citation,blob] [--state s] [--status s] [--kind k] [--retracted include|only] [--limit 25] [--cursor c]\` browses records. Bodies are cut at 600 characters or 12 lines; Blob bodies are left out. Retracted records are left out unless you pass \`--retracted\`, and are marked RETRACTED when shown. - \`./flatbread proof get [--resolve head]\` shows one record in full, Blob payloads included. \`--resolve head\` follows replacements to the newest version. - \`./flatbread proof relations --relations derives_from,supersedes,superseded_by,invalidates,invalidated_by,resolved_by,rejected_by,mitigated_by,evidence,cites\` follows one hop of links. - \`./flatbread proof blocking-decisions \`. -${ - brief - ? '' - : '\nDo not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n' -} +${brief ? '' : find ? FIND_FIRST : NO_BRIEF} Records can be superseded (replaced), invalidated (shown wrong), rejected, reopened, or retracted (noise, hidden from browse reads unless asked; \`get\` still shows them). Titles repeat on purpose. Efforts never link to each other; one may name another's record by title in prose. A summary can be out of date; check its claims against the records. ## The question