From 0133622b72c2ca784a3d7ed7dbf32e40cf42aed4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 15:31:06 +0000 Subject: [PATCH 1/2] feat(proof): report compaction drift on `get`, filter records by standing Two read changes, chosen from the wrong answers on the benchmark's tune questions. The two largest causes were these: 1. Agents could not tell whether a compaction was stale. `brief` worked it out, but `get` and `state` did not. Now `get` on a compaction says whether it is the head, whether it is stale and why, and the exact number of records that changed since it was written, listing up to 25. `state` gives the head compaction its own section, with the same verdict and rule. The rule lives once, in `compactionReport` and `COMPACTION_STALE_RULE` in `packages/proof/src/compaction.ts`, and `brief` uses it too, with byte-identical output. 2. Agents tried `records --state invalidated`, which silently returned nothing, because "invalidated" is decided by links, not stored. `records` now takes `--standing ` (superseded, invalidated, rejected and the stored states) and `--rests-on ` (live records that derive from, cite, or give as evidence a dead record; the same rule `state` uses). An unknown `--state` or `--status` value is refused with the valid values and a pointer to the new flags. On the 10x graph the find and state gates are unchanged on every question, so this changes no search result. `scripts/read-recall.mjs` checks the new reads offline, and run files now record `read_hash`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW --- .agents/skills/proof/SKILL.md | 11 +- .agents/skills/proof/reference.md | 35 +- benchmarks/proof-recall/README.md | 150 +++++- benchmarks/proof-recall/scripts/collect.mjs | 40 +- .../proof-recall/scripts/read-recall.mjs | 497 ++++++++++++++++++ benchmarks/proof-recall/scripts/run.mjs | 12 +- packages/flatbread/src/cli/proof.test.ts | 258 +++++++++ packages/flatbread/src/cli/proof.ts | 14 + packages/flatbread/src/cli/serve.test.ts | 56 ++ packages/flatbread/src/proof/read.ts | 254 ++++++++- packages/flatbread/src/proof/state.test.ts | 162 +++++- packages/flatbread/src/proof/state.ts | 178 ++++--- packages/proof/skills/proof/SKILL.md | 11 +- packages/proof/skills/proof/reference.md | 35 +- packages/proof/src/compaction.ts | 40 ++ 15 files changed, 1646 insertions(+), 107 deletions(-) create mode 100644 benchmarks/proof-recall/scripts/read-recall.mjs diff --git a/.agents/skills/proof/SKILL.md b/.agents/skills/proof/SKILL.md index 18427ce0..18ba10ed 100644 --- a/.agents/skills/proof/SKILL.md +++ b/.agents/skills/proof/SKILL.md @@ -191,9 +191,16 @@ Pick the read by what you need, in this order: live records that rest on superseded, disproved, rejected, or retracted ones; dated lifecycles; and retractions. It holds no facts from record bodies: use `find` for those. -3. **Read one record in full: `proof get `.** +3. **Read one record in full: `proof get `.** On a compaction, `get` + also says whether it is stale, and why, gives the exact number of + records that changed since it was written, and lists up to 25 of them. 4. **Page through records: `proof records`.** `find` shows only the top - hits, so never count from it; count from `state`. + hits, so never count from it; count from `state`. `--state` and + `--status` match only the stored field. `--standing invalidated` (or + `superseded`, `retracted`, any standing a card prints) lists records by + where they stand, links included. `--rests-on invalidated` lists the live + records that derive from, cite, or give as evidence a record in that + standing. Browse digests (`list`, `records`, `relations`, `blocking-decisions`) excerpt each body at 600 chars / 12 lines (`[…truncated]`). **`proof get` digests diff --git a/.agents/skills/proof/reference.md b/.agents/skills/proof/reference.md index b5412c38..bf981882 100644 --- a/.agents/skills/proof/reference.md +++ b/.agents/skills/proof/reference.md @@ -249,7 +249,7 @@ digest or `summary` as a data feed. ```bash flatbread proof get [--resolve exact|head] [consistency flags] flatbread proof list [--status active,paused,...] [--limit n] [--cursor c] [consistency flags] -flatbread proof records [--kinds k1,k2] [--state s1,s2] [--status s1,s2] [--kind k1,k2] [--since iso] [--until iso] [--retracted exclude|include|only] [--limit n] [--cursor c] [consistency flags] +flatbread proof records [--kinds k1,k2] [--state s1,s2] [--status s1,s2] [--standing s1,s2] [--rests-on s1,s2] [--kind k1,k2] [--since iso] [--until iso] [--retracted exclude|include|only] [--limit n] [--cursor c] [consistency flags] flatbread proof relations --relations r1,r2 [--limit n] [--cursor c] [consistency flags] flatbread proof blocking-decisions [consistency flags] flatbread proof brief [consistency flags] @@ -276,6 +276,20 @@ flatbread proof serve [--socket .flatbread/proof/serve.sock] ordered by `created_at` ascending, then `id`. - Filter semantics: AND across different flags, OR within a comma list. `--since`/`--until` bound `created_at` (gte/lte, ISO strings). +- `--state` and `--status` match the stored field only. `--state` takes + `proposed`, `accepted`, `rejected`, `superseded`, `deprecated`, `open`, + `mitigated`, `realized`; `--status` takes `open`, `resolved`, `deferred`, + `wontfix`. Any other value is refused with `PROOF_INVALID_ARGUMENT`, + never answered with an empty page. +- `--standing` matches the standing cards and `proof state` print, which + links decide before the stored field: `retracted`, then `superseded`, + then `invalidated`, else the Issue's status, the Decision's or Risk's + state, or `live`. Asking for `retracted` shows retracted records. +- `--rests-on` takes `superseded`, `invalidated`, `rejected`, `retracted`. + It lists live records, compactions left out, that derive from, cite, or + give as evidence a record in one of those standings, in any Effort. The + record that overturned it does not count. This is the rule behind the + count in `proof state`'s Dead evidence. - `--relations` values: `derives_from`, `supersedes`, `superseded_by`, `invalidates`, `invalidated_by`, `rejected_by`, `mitigated_by`, `resolved_by`, `evidence`, `cites` (one hop, explicit only). Missing targets @@ -293,6 +307,13 @@ flatbread proof serve [--socket .flatbread/proof/serve.sock] Decision to mitigate the Risk. - `--resolve head`: follow `superseded_by` to the current tip; ancestors render as checkpoint lines (max 5, then a count). +- `get` on a compaction adds `## Compaction`: whether it is its Effort's + head (or what superseded it), `Stale: yes|no` with the reason and the + rule, and the exact number of records that differ from the live set it + covers. `## Changes since compaction` then lists them as `brief` does, + up to 25 lines; past that, `- … N more` ends the list, and nothing pages + the rest (on the head, `brief` lists up to 50). For the head, these match + `brief`'s `compaction` field. - `blocking-decisions` membership (frozen): Decision in the effort with `state: proposed` whose `derives_from` directly contains an Issue in the same effort with `kind: blocker` and `status: open`. For "what blockers @@ -436,8 +457,13 @@ The Markdown has these sections, in order: record that live records still derive from, cite, or give as evidence, with those live records and, for every dead record but a replaced version, the live records one link further. The record that overturned it is not counted as - resting on it. The first line reports the head compaction: how many - records changed since it was written, and which of its supports moved. + resting on it. +- **Head compaction:** shown when the Effort has one. Whether it is stale + and why, with the rule, and the exact number of records that differ from + the live set it covers, then up to 10 of them as `id (new)` or + `id (before -> now)`. A large Effort cuts that list first, to 5 and then + none; the count stays, and the line names `proof get `, which lists + up to 25 of them. - **Lifecycle:** dated changes no section above shows, the most recently changed first: closed Issues other than settled questions, proposed or reopened @@ -452,7 +478,8 @@ that date. Records with nothing to report, such as a question Issue resolved by its one accepted Decision, appear only in the counts. Citations and Blobs have no live or dead standing, so the live total leaves them out. On a large Effort the lists shrink to fit, cutting -lifecycle and retractions first and counts never. Any cut list, a whole +lifecycle, retractions, and the head compaction's list first and counts +never. Any cut list, a whole section's or one inside an entry, ends with `… N more` or `and N more`, and then `cap_reasons` holds `state_limit`. In `state.sections`, `total` counts a section's entries in full and `shown` those it prints; for diff --git a/benchmarks/proof-recall/README.md b/benchmarks/proof-recall/README.md index f75e5e51..a8930f16 100644 --- a/benchmarks/proof-recall/README.md +++ b/benchmarks/proof-recall/README.md @@ -26,6 +26,7 @@ Each question has a known answer and a stated path through the graph. | `scripts/compare.mjs` | Compares two runs question by question | | `scripts/find-recall.mjs` | Checks offline whether `proof find` reaches the evidence | | `scripts/state-recall.mjs` | Checks offline whether `proof state` shows the answer | +| `scripts/read-recall.mjs` | Checks offline what `get` and `records` add for a reader | | `runs/` | Answers and costs from each pilot and full run | | `.flatbread-proof/` | The built graph, committed so runs need no build | | `flatbread.config.js` | Lets `flatbread proof …` read the graph from this folder | @@ -311,12 +312,39 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`. both rates, the rates for value and record answers apart, and the mean and largest output in characters, measured with the temporary folder's path removed so the size does not depend on the machine. It writes - `state_hash`, a hash of the `state` source. Its bar for the paid Run D: + `state_hash`, a hash of the `state` source (`state.ts` and + `compaction.ts`). Its bar for the paid Run D: on holdout, the strict rule finds the answer for at least 60% of the aggregate, lifecycle, blocker, and dead-evidence questions taken together, and the mean output is at most 16,000 characters. `--split` works as above. + After a change to `get` or `records`, run the third free check: + `node scripts/read-recall.mjs --graph --split tune`. It first + checks three invariants on every Effort and fails if any does not hold: + each `records --standing ` total equals that standing's total in + `state`'s Counts; the `records --rests-on` total over all four dead + standings equals `state`'s "N live records rest directly on"; and `get` + on the head compaction gives the same count and stale verdict as + `brief`'s `compaction` field. It then scores two groups of questions. + For `compaction` questions it runs `get` on each evidence record that + is a compaction: a count answer scores when it is the number on the + "differ" line, a boolean when it matches the `Stale:` line, and a record + answer when every id is in the change list. A boolean question with the + word "stale" wants the verdict, one with "still" ("does that still + hold?") wants its opposite, and any other boolean question is not + scored. For + `dead-evidence` and `invalidation` questions with record or count + answers it pages through `records --rests-on` with each dead standing + and with all four, and `records --standing invalidated`: a record answer + scores when all its ids fall inside one read's set, a count when it + equals one read's total. It prints how often a count matches a total + for another Effort by chance. `--split` works as above, and questions + outside the split are not read. Run files record `read_hash`, a hash of + `read.ts`, `state.ts`, and `compaction.ts`, the sources of `get`, + `records`, and `brief`. From this change on, `state_hash` covers + `state.ts` and `compaction.ts`; older run files hash `state.ts` alone. + 3. Run the benchmark: ```bash @@ -1069,3 +1097,123 @@ So the search gain carries over to real memory at about half the reading. The gain from `state` on the 10x graph's counts and blockers does not show on graphs a twenty-fifth that size or less, where paging `records` is cheap. + +### `get` on a compaction, and `records` by standing + +The tune failures of Runs A, C, D and the Sonnet run fell mostly into two +groups. In 16 failed runs the agent could not tell whether a compaction +(the Finding that sums up an Effort) was stale or how many records had +changed since it was written: only `brief` printed that, and the `find` +arms never call `brief`. In 8 more the agent asked for +`records --state invalidated`, which no record stores, and got an empty, +complete page. Two reads changed; the task texts and `find` did not, so +`find_hash` is unchanged. + +- `proof get` on a compaction adds `## Compaction`: whether it is its + Effort's head or what superseded it, `Stale: yes|no` with the records + that moved and the rule ("A compaction is stale when a record it derives + from changed standing after it was written, or when a Finding + invalidated it."), and the exact number of records that differ from the + live set it covers. `## Changes since compaction` lists them, up to 25; + past that, `- … N more; the count above is exact` ends the list, and on + the head it points to `brief`, which lists up to 50. + `brief`, `get`, and `state` all take these from one function, + `compactionReport`. `get` on any other record prints what it printed + before. +- `proof state` moves the compaction line out of Dead evidence into its own + section, `Head compaction`, with the same verdict and count and up to 10 + changed records. A large Effort cuts that list to 5 along with lifecycle + and retractions, and to none along with the replaced-support lists; the + count stays. +- `proof records` takes `--standing ` (the standing cards print, links + included, so `invalidated` and `retracted` work) and `--rests-on ` (live records that derive from, cite, or give as evidence a + record in that standing, by the rule `state` uses for Dead evidence). A + `--state` or `--status` value that no record stores is refused with an + error that names both new flags. + +`scripts/read-recall.mjs` checks these reads offline. On every Effort of +the 10x graph, all 85 invariants hold: each `--standing` total matches +`state`'s Counts, the `--rests-on` total matches Dead evidence, and `get` +on the head compaction matches `brief`. + +Free gates on the 10x graph, before and after. `find` output is the same +on every question (`find_hash` `e5be6532…`): + +| `find-recall.mjs`, stripped set | Tune, before | Tune, after | Holdout, before | Holdout, after | +| ------------------------------- | ------------ | ----------- | --------------- | -------------- | +| Evidence in the top 5 | 79% | 79% | 85% | 85% | +| Answer on a card | 59% | 59% | 66% | 66% | +| Mean reply, characters | 6,369 | 6,369 | 6,365 | 6,365 | + +| `state-recall.mjs` | Tune, before | Tune, after | Holdout, before | Holdout, after | +| ----------------------- | ------------ | ----------- | --------------- | -------------- | +| Bar categories, strict | 53% (18/34) | 53% (18/34) | 55% (17/31) | 55% (17/31) | +| Bar categories, loose | 65% (22/34) | 65% (22/34) | 61% (19/31) | 61% (19/31) | +| Every question, strict | 50% (47/95) | 50% (47/95) | 44% (38/86) | 44% (38/86) | +| Mean output, characters | 15,171 | 15,652 | 14,981 | 15,474 | +| Largest output | 28,058 | 29,130 | 58,681 | 61,797 | + +The new section finds no answer `state` did not already find, and costs +about 3% more output (`state_hash` `be19faeb…`, which now covers +`state.ts` and `compaction.ts`). The mean stays under the 16,000 bar. Every +"after" figure in this section comes from the final build. + +`read-recall.mjs` on the 10x graph. Before this change no read printed the +count or the verdict, so the baseline is 0 by construction for counts and +booleans; it was not run. The boolean rule was changed once after reading +tune rows: a question with "still" ("does that still hold?") wants the +opposite of the `Stale:` verdict. That raised tune booleans from 2/4 to +4/4, so the tune boolean figure measures the scorer as much as the read. +Holdout has no boolean compaction questions. Holdout was scored from the +script totals only. + +| Group | Tune | Holdout | +| ------------------------------------------------------------- | ----------- | ---------- | +| Compaction, all | 92% (12/13) | 14% (1/7) | +| Compaction, counts | 3/3 | 1/2 | +| Compaction, booleans (after the scorer change) | 4/4 | none | +| Compaction, records | 3/4 | 0/3 | +| Compaction, values | 2/2 | 0/2 | +| Dead evidence and invalidation, `--rests-on` and `--standing` | 39% (5/13) | 30% (3/10) | + +The holdout result is weak, and its totals do not say why. On tune, the +one compaction miss asks which compaction was the newest on a past date. +Of the 8 tune misses with `--rests-on` and `--standing`, 3 ask for the +Finding that overturned a support, which neither flag lists, and 5 need a +second hop or a date past the record that rests on dead evidence (for +example, the open Risk that derives from a Decision that rests on an +invalidated Finding). The paid run, not this check, has to show whether the agents who failed +on staleness and on `--state invalidated` now get them right. + +Read size. Every `proof` read in the tune transcripts of +`x10-find-r{1,2}`, `x10adv-find-r1`, `x10-sonnet-r1` and `x10-rund-r{1,2}` +(`find` and `findstate` arms, 3,807 reads) was replayed against the old +and the new build, each from its own `proof serve`, measuring the JSON +envelope plus the digest: + +| Read | Calls | Old mean | New mean | Old max | New max | Output changed | +| --------- | ----- | -------- | -------- | ------- | ------- | -------------- | +| All | 3,807 | 9,201 | 9,243 | 32,503 | 32,503 | 216 | +| `find` | 1,201 | 14,752 | 14,752 | 32,503 | 32,503 | 0 | +| `get` | 1,346 | 2,154 | 2,259 | 3,435 | 7,120 | 68 | +| `records` | 774 | 13,533 | 13,456 | 24,683 | 24,683 | 72 | +| `state` | 76 | 24,825 | 25,851 | 31,292 | 31,757 | 76 | +| Other | 410 | same | same | same | same | 0 | + +The new mean is 1.005 times the old (the new sizes are from the final +build, replayed on its own `proof serve`). Every changed `get` is on a +compaction (2,898 to 4,974 characters on average); every changed +`records` call is a refused value: `invalidated` as a state or status +in 42 calls, `reopened` in 26, and 4 others. Each of those calls used to +get an empty, complete page; the error that replaces it is shorter (569 +characters against 1,396). No read reaches `reopen_history` yet, so the +`reopened` calls learn only that the value is not stored. `brief` prints the same bytes on all +ten Efforts of both 10x graphs, and `get` on each of their 24 compactions +prints the old digest once the two new sections are taken out. The +`state` calls grow by the new section, about 1,000 characters. + +On the two private graphs (totals only), the find and state gates give the +same figures before and after on graph A and graph B, neither graph holds a +compaction question, and the read invariants hold on every Effort (165 on +A, 90 on B). The `--rests-on` reads hold the answer for 1 of 3 scored +questions on A and 1 of 1 on B. diff --git a/benchmarks/proof-recall/scripts/collect.mjs b/benchmarks/proof-recall/scripts/collect.mjs index cfe0a842..5eab07fe 100644 --- a/benchmarks/proof-recall/scripts/collect.mjs +++ b/benchmarks/proof-recall/scripts/collect.mjs @@ -31,19 +31,39 @@ export async function findHash() { .digest('hex'); } -// The `proof state` source: its sections, caps, and size aim. -const STATE_SOURCE = resolve( - dirname(fileURLToPath(import.meta.url)), - '../../../packages/flatbread/src/proof/state.ts' -); +const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../../..'); +// What `proof state` prints: its sections, caps, and size aim, and the +// compaction rule it shares with `get` and `brief`. +const STATE_SOURCES = [ + 'packages/flatbread/src/proof/state.ts', + 'packages/proof/src/compaction.ts', +]; +// What `get`, `records`, and `brief` print and filter. They also use the +// compaction rule and the rests-on rule from `state.ts`. +const READ_SOURCES = [ + 'packages/flatbread/src/proof/read.ts', + 'packages/flatbread/src/proof/state.ts', + 'packages/proof/src/compaction.ts', +]; -/** A sha256 of the `proof state` source, as `findHash` is of `find`'s. */ -export async function stateHash() { - return createHash('sha256') - .update(await readFile(STATE_SOURCE)) - .digest('hex'); +/** A sha256 over these source files, in order, each with its path. */ +async function hashOf(paths) { + const hash = createHash('sha256'); + for (const path of paths) + hash.update(`${path}\n`).update(await readFile(join(ROOT, path))); + return hash.digest('hex'); } +/** + * A sha256 of the `proof state` source and the compaction rule it uses, as + * `findHash` is of `find`'s. Before `get` on a compaction was added, it + * hashed `state.ts` alone, so older run files hold that hash. + */ +export const stateHash = () => hashOf(STATE_SOURCES); + +/** A sha256 of the `get`, `records`, and `brief` sources, as `stateHash` is of `state`'s. */ +export const readHash = () => hashOf(READ_SOURCES); + /** * `context_tokens` counts what entered the context once (new input, cache * writes, output): how much the agent had to read. `processed_tokens` adds diff --git a/benchmarks/proof-recall/scripts/read-recall.mjs b/benchmarks/proof-recall/scripts/read-recall.mjs new file mode 100644 index 00000000..5a21af56 --- /dev/null +++ b/benchmarks/proof-recall/scripts/read-recall.mjs @@ -0,0 +1,497 @@ +#!/usr/bin/env node +// The free offline check for what `proof get` prints on a compaction and +// what `proof records --standing` and `--rests-on` list, before any paid +// agent run. +// +// node scripts/read-recall.mjs [--graph ] [--ids a,b] +// [--only ambiguous] [--split tune|holdout|all] [--out ] +// +// Invariants, on every Effort of the graph, with no questions: +// - for each standing, the `records --standing ` total equals the sum +// over kinds of that standing in `state`'s Counts; +// - the `records --rests-on superseded,invalidated,rejected,retracted` +// total equals `state`'s "N live records rest directly on"; +// - `get` on the head compaction gives the count and stale verdict that +// `brief` gives in its `compaction` field. +// The script fails (exit code 1) when any of them does not hold. +// +// Questions, scored as the other gates score them: +// compaction for each evidence id that is a compaction, `get` it. A +// count answer scores when it is the number on the "differ" +// line, and a boolean answer when it matches the `Stale:` +// line. A boolean question with the word "stale" asks for +// the verdict; one with "still" ("does that still hold?") +// asks for its opposite; any other boolean question is not +// scored. A record answer scores when every id is in the +// change list, and a value answer as find-recall.mjs scores +// a card. +// rests-on dead-evidence and invalidation questions with a record, +// records, or count answer. Each reads, for its Effort, +// `records --rests-on` with each dead standing and with all +// four, and `records --standing invalidated`, every page. A +// record answer scores when all its ids fall inside one +// read's set; a count when it equals one read's total. +// Text, "unknown", and other answers are not scored. A count can equal a +// total by chance, so the table also prints, for each count question in +// the split, the share of other Efforts whose reads give that total, +// averaged. Questions outside `--split` are not read at all. +// +// Before this change, `get` printed neither the count nor the verdict and +// `records` had neither flag, so the baseline is 0 by construction for +// counts and booleans. Read tune rows to debug; score holdout once. +// +// The graph is copied, without its key map, into a temporary folder, and one +// `proof serve` there answers every read. The folder is removed at the end. +import { spawn } from 'node:child_process'; +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { MULTI_EFFORT_SETS, RUNS_DIR, pickQuestions } from './check.mjs'; +import { readHash } from './collect.mjs'; +import { + answerOnCard, + flatbread, + pct, + rate, + serverAnswers, + sleep, + splitOf, + targets, +} from './find-recall.mjs'; +import { loadGraph, qualify } from './graph.mjs'; +import { prepareFolder } from './prepare-run.mjs'; + +const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const SPLITS = ['tune', 'holdout', 'all']; +const KINDS = 'issue,finding,decision,constraint,risk,citation,blob'; +const DEAD = ['superseded', 'invalidated', 'rejected', 'retracted']; +const STANDINGS = [ + 'open', + 'resolved', + 'deferred', + 'wontfix', + 'proposed', + 'accepted', + 'rejected', + 'deprecated', + 'realized', + 'mitigated', + 'live', + ...DEAD.filter((s) => s !== 'rejected'), +]; +const REST_CATEGORIES = ['dead-evidence', 'invalidation']; + +function args(argv) { + const out = { + graph: join(BENCH, '.flatbread-proof'), + ids: [], + only: [], + out: join(RUNS_DIR, 'read-recall.json'), + split: 'all', + }; + for (let i = 0; i < argv.length; i++) { + const flag = argv[i]; + if (flag === '--graph') out.graph = resolve(argv[++i]); + else if (flag === '--ids') out.ids = argv[++i].split(','); + else if (flag === '--only') out.only = argv[++i].split(','); + else if (flag === '--out') out.out = resolve(argv[++i]); + else if (flag === '--split') out.split = argv[++i]; + else throw new Error(`Unknown argument ${flag}`); + } + if (!SPLITS.includes(out.split)) + throw new Error(`--split must be one of ${SPLITS.join(', ')}`); + return out; +} + +const isCompaction = (record) => + record?.kind === 'finding' && record.frontmatter.kind === 'compaction'; + +/** Standing totals from `state`'s Counts: the top-level line of each kind. */ +function stateCounts(markdown) { + const counts = new Map(); + const start = markdown.indexOf('### Counts\n'); + const end = markdown.indexOf('\n### ', start + 1); + for (const line of markdown.slice(start, end).split('\n')) { + const match = line.match( + /^- (?:Issues|Decisions|Findings|Constraints|Risks|Citations|Blobs) \d+: (.*)$/ + ); + if (!match) continue; + for (const part of match[1].split(', ')) { + const [word, n] = part.split(' '); + counts.set(word, (counts.get(word) ?? 0) + Number(n)); + } + } + return counts; +} + +/** Ids and total of a `records` read, every page. */ +async function readAll(read, argv) { + const ids = []; + let total = null; + let cursor; + do { + const out = JSON.parse( + await read([...argv, ...(cursor ? ['--cursor', cursor] : [])]) + ); + const digest = await readFile(out.artifact_path, 'utf8'); + total ??= Number(digest.match(/"total_known":(\d+)/)?.[1] ?? 0); + ids.push(...[...digest.matchAll(/^### (\S+)$/gm)].map((m) => m[1])); + cursor = out.page.next_cursor; + } while (cursor); + return { ids, total }; +} + +/** The total of a `records` read, from its first page of one record. */ +async function totalOf(read, argv) { + const out = JSON.parse(await read([...argv, '--limit', '1'])); + const digest = await readFile(out.artifact_path, 'utf8'); + return { total: Number(digest.match(/"total_known":(\d+)/)?.[1] ?? 0) }; +} + +/** What `get` prints on a compaction: the count, the verdict, the change list. */ +function compactionFacts(digest) { + const differ = digest.match( + /^- Since it was written, (\d+) records? differs? from the live set/m + ); + const stale = digest.match(/^- Stale: (yes|no)\./m); + const start = digest.indexOf('## Changes since compaction\n'); + const end = digest.indexOf('\n## ', start + 1); + const changes = + start < 0 + ? [] + : [ + ...digest + .slice(start, end) + .matchAll(/^- (?:added|changed|left) \| (\S+) \|/gm), + ].map((m) => m[1]); + return { + differ: differ ? Number(differ[1]) : null, + stale: stale ? stale[1] === 'yes' : null, + changes, + }; +} + +/** + * Whether a boolean compaction question is answered by a `Stale:` line: + * "is it stale?" wants the verdict, "does it still hold?" its opposite. + * Any other wording is not scored (null). + */ +function booleanHit(question, gold, facts) { + const yes = gold === true || gold === 'true'; + const want = /\bstale\b/i.test(question) + ? yes + : /\bstill\b/i.test(question) + ? !yes + : null; + if (want === null) return null; + return facts.some((f) => f.stale === want); +} + +function tally(rows) { + return { + n: rows.length, + hit: rate(rows, 'hit'), + mean_chars: Math.round( + rows.reduce((sum, r) => sum + r.chars, 0) / (rows.length || 1) + ), + }; +} + +async function main() { + const options = args(process.argv.slice(2)); + const graph = await loadGraph(options.graph); + const read_hash = await readHash(); + const efforts = [...graph.records.values()] + .filter((record) => record.kind === 'effort') + .map((record) => record.id) + .sort(); + const effortOfKey = Object.fromEntries( + Object.entries(graph.keymap) + .filter(([key]) => key.endsWith('.effort')) + .map(([key, id]) => [key.split('.')[0], id]) + ); + + const folder = await mkdtemp(join(tmpdir(), 'read-recall-')); + const socket = join(folder, 'serve.sock'); + await prepareFolder(folder, [], options.graph); + const server = spawn( + join(folder, 'flatbread'), + ['proof', 'serve', '--socket', socket], + { cwd: folder, stdio: 'ignore' } + ); + const read = (argv) => + flatbread(folder, ['proof', ...argv], { FLATBREAD_PROOF_SERVER: socket }); + const invariants = []; + const restReads = new Map(); + const gets = new Map(); + const rows = []; + try { + for (let i = 0; !(await serverAnswers(socket)); i++) { + if (i > 600) throw new Error('proof serve did not start'); + await sleep(200); + } + const check = (effort, name, want, got) => + invariants.push({ effort, name, want, got, ok: want === got }); + const getCompaction = async (id) => { + if (!gets.has(id)) { + const out = await read(['get', id]); + const digest = await readFile(JSON.parse(out).artifact_path, 'utf8'); + gets.set(id, { + chars: out.length + digest.length, + digest, + ...compactionFacts(digest), + }); + } + return gets.get(id); + }; + + const restsOn = async (effort) => { + if (!restReads.has(effort)) { + const reads = {}; + for (const standing of [...DEAD, DEAD.join(',')]) + reads[`rests-on ${standing}`] = await readAll(read, [ + 'records', + effort, + '--rests-on', + standing, + '--kinds', + KINDS, + ]); + reads['standing invalidated'] = await readAll(read, [ + 'records', + effort, + '--standing', + 'invalidated', + '--kinds', + KINDS, + ]); + restReads.set(effort, reads); + } + return restReads.get(effort); + }; + + for (const effort of efforts) { + // Every Effort's reads, so the chance rate below does not depend on + // which Efforts the split's questions name. + await restsOn(effort); + const state = JSON.parse(await read(['state', effort])); + const counts = stateCounts(state.markdown); + for (const standing of STANDINGS) { + const { total } = await totalOf(read, [ + 'records', + effort, + '--standing', + standing, + '--kinds', + KINDS, + ]); + check(effort, `standing ${standing}`, counts.get(standing) ?? 0, total); + } + const resting = Number( + state.markdown.match(/(\d+) live records? rests? directly on/)?.[1] + ); + const all = await readAll(read, [ + 'records', + effort, + '--rests-on', + DEAD.join(','), + '--kinds', + KINDS, + ]); + check(effort, 'rests-on all', resting, all.total); + const brief = JSON.parse(await read(['brief', effort])).compaction; + if (brief?.head_id) { + const got = await getCompaction(brief.head_id); + check(effort, 'compaction changes', brief.changes, got.differ); + check(effort, 'compaction stale', brief.stale, got.stale); + } + } + + for (const set of await pickQuestions(options)) + for (const q of set.questions) { + if (options.split !== 'all' && splitOf(q.id) !== options.split) + continue; + const { answers, evidence } = targets(graph, set.effort, q); + const kind = q.answer.kind; + const base = { + id: q.id, + effort: set.effort, + categories: q.categories, + answer_kind: kind, + }; + if (q.categories.includes('compaction')) { + const compactions = [...evidence].filter((id) => + isCompaction(graph.records.get(id)) + ); + if (compactions.length) { + const facts = await Promise.all(compactions.map(getCompaction)); + const gold = q.answer.value; + const hit = + kind === 'count' + ? facts.some((f) => f.differ === Number(gold)) + : kind === 'boolean' + ? booleanHit(q.question, gold, facts) + : kind === 'record' || kind === 'records' + ? facts.some((f) => + answers.every((id) => f.changes.includes(id)) + ) + : answerOnCard( + q, + answers, + facts.map((f) => f.digest).join('\n') + ); + rows.push({ + ...base, + read: 'compaction', + hit, + chars: facts.reduce((sum, f) => sum + f.chars, 0), + }); + } + } + if ( + q.categories.some((c) => REST_CATEGORIES.includes(c)) && + ['record', 'records', 'count'].includes(kind) + ) { + const keys = MULTI_EFFORT_SETS.includes(set.effort) + ? [ + ...new Set( + [ + ...(q.evidence ?? []), + ...(kind === 'count' ? [] : [q.answer.value].flat()), + ].map((key) => qualify(set.effort, key).split('.')[0]) + ), + ] + : [set.effort]; + const reads = []; + for (const key of keys) + if (effortOfKey[key]) + reads.push(...Object.values(await restsOn(effortOfKey[key]))); + const hit = + kind === 'count' + ? reads.some((r) => r.total === Number(q.answer.value)) + : reads.some((r) => answers.every((id) => r.ids.includes(id))); + rows.push({ + ...base, + read: 'rests-on', + hit, + efforts: keys, + gold: kind === 'count' ? Number(q.answer.value) : undefined, + chars: 0, + }); + } + } + } finally { + server.kill(); + await rm(folder, { recursive: true, force: true }); + } + + // How often a count matches some read's total for another Effort. + const totals = new Map( + [...restReads].map(([effort, reads]) => [ + effort, + Object.values(reads).map((r) => r.total), + ]) + ); + const chance = rows + .filter((r) => r.read === 'rests-on' && r.answer_kind === 'count') + .map((r) => { + const own = new Set(r.efforts.map((key) => effortOfKey[key])); + const others = [...totals].filter(([effort]) => !own.has(effort)); + return others.length + ? others.filter(([, list]) => list.includes(r.gold)).length / + others.length + : 0; + }); + + const groups = { + all: tally(rows), + compaction: tally(rows.filter((r) => r.read === 'compaction')), + ' count': tally( + rows.filter((r) => r.read === 'compaction' && r.answer_kind === 'count') + ), + ' boolean': tally( + rows.filter((r) => r.read === 'compaction' && r.answer_kind === 'boolean') + ), + ' record': tally( + rows.filter( + (r) => + r.read === 'compaction' && + ['record', 'records'].includes(r.answer_kind) + ) + ), + ' value': tally( + rows.filter((r) => r.read === 'compaction' && r.answer_kind === 'value') + ), + 'rests-on': tally(rows.filter((r) => r.read === 'rests-on')), + ' count ': tally( + rows.filter((r) => r.read === 'rests-on' && r.answer_kind === 'count') + ), + ' record ': tally( + rows.filter((r) => r.read === 'rests-on' && r.answer_kind !== 'count') + ), + }; + console.log( + `\ngraph ${options.graph}, split ${options.split} (${rows.length} scored rows)` + ); + console.log( + 'group'.padEnd(16) + + 'n'.padStart(5) + + 'scored'.padStart(8) + + 'hit'.padStart(7) + + 'chars'.padStart(8) + ); + for (const [name, t] of Object.entries(groups)) + console.log( + name.padEnd(16) + + String(t.n).padStart(5) + + String(t.hit.n).padStart(8) + + pct(t.hit).padStart(7) + + String(t.mean_chars).padStart(8) + ); + const failed = invariants.filter((i) => !i.ok); + for (const i of failed) + console.log( + ` invariant failed: ${i.effort} ${i.name}: want ${i.want}, got ${i.got}` + ); + const meanChance = chance.length + ? chance.reduce((a, b) => a + b, 0) / chance.length + : 0; + console.log( + `TOTAL compaction ${pct(groups.compaction.hit)} of ${ + groups.compaction.hit.n + }; rests-on ${pct(groups['rests-on'].hit)} of ${ + groups['rests-on'].hit.n + }; count chance rate ${Math.round(meanChance * 100)}% over ${ + chance.length + } count questions` + ); + console.log( + `INVARIANTS ${invariants.length - failed.length} of ${ + invariants.length + } hold on ${efforts.length} Efforts${failed.length ? ' FAIL' : ''}` + ); + await writeFile( + options.out, + `${JSON.stringify( + { + graph: options.graph, + split: options.split, + read_hash, + invariants: { held: invariants.length - failed.length, failed }, + count_chance_rate: +meanChance.toFixed(3), + groups, + rows: rows, + }, + null, + 2 + )}\n` + ); + console.log(`read_hash ${read_hash}\nWrote ${options.out}`); + if (failed.length) process.exitCode = 1; +} + +main().catch((error) => { + console.error(error.message); + process.exitCode = 1; +}); diff --git a/benchmarks/proof-recall/scripts/run.mjs b/benchmarks/proof-recall/scripts/run.mjs index 16e08c92..729fbc3e 100644 --- a/benchmarks/proof-recall/scripts/run.mjs +++ b/benchmarks/proof-recall/scripts/run.mjs @@ -17,9 +17,10 @@ // answer.json, and transcript.jsonl. A rerun skips sessions that already // have both an answer and a transcript, so an interrupted run picks up where // it stopped. Keep outside the repository, so no CLAUDE.md or -// AGENTS.md reaches the agent. Each run file records `find_hash` and -// `state_hash`, hashes of the `proof find` and `proof state` sources, to -// match against the offline gates' output. +// AGENTS.md reaches the agent. Each run file records `find_hash`, +// `state_hash`, and `read_hash`, hashes of the `proof find`, `proof state`, +// and `get`/`records`/`brief` sources, to match against the offline gates' +// output. // // Runs share two things, kept under /flatbread-proof-recall/: // @@ -52,7 +53,7 @@ import { request } from 'node:http'; import { tmpdir } from 'node:os'; import { dirname, join, relative, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; -import { collect, findHash, stateHash } from './collect.mjs'; +import { collect, findHash, readHash, stateHash } from './collect.mjs'; import { copyPublicGraph, prepareFolder, @@ -338,6 +339,7 @@ async function main() { const questions = await publicQuestions(options); const find_hash = await findHash(); const state_hash = await stateHash(); + const read_hash = await readHash(); // Alternate strategies question by question, so a change in API speed or // load over the run falls on every strategy alike. const sessions = []; @@ -398,7 +400,7 @@ async function main() { })), `${options.name}-${strategy}`, join(RUNS_DIR, `${options.name}-${strategy}.json`), - { find_hash, state_hash } + { find_hash, state_hash, read_hash } ); } diff --git a/packages/flatbread/src/cli/proof.test.ts b/packages/flatbread/src/cli/proof.test.ts index 6c817dcb..acf13845 100644 --- a/packages/flatbread/src/cli/proof.test.ts +++ b/packages/flatbread/src/cli/proof.test.ts @@ -1714,3 +1714,261 @@ export default { ); } ); + +test.serial( + 'get on a compaction says whether it is head, whether it is stale and why, and what changed', + async (t) => { + const cwd = await createTempProject('flatbread-effort-get-compaction-', t); + await writeFile( + join(cwd, 'flatbread.config.js'), + `import { source } from '@flatbread/source-filesystem'; +import { transformer } from '@flatbread/transformer-markdown'; +import { proofContent } from '@flatbread/proof'; +export default { source: source(), transformer: transformer(), content: proofContent('.flatbread-proof') };` + ); + const write = async (payload: Record) => + handleEffortWrite(JSON.stringify(payload), { cwd }); + const digest = async (id: string) => + readFile((await handleEffortGet(id, { cwd })).artifact_path, 'utf8'); + const effortId = ( + await write({ type: 'CreateEffort', title: 'Get', body: '' }) + ).artifacts[0].id; + const measured = ( + await write({ + type: 'WriteFinding', + effort: effortId, + title: 'Reads take 4 ms', + body: 'Measured on the dev kit.', + kind: 'measurement', + }) + ).artifacts[0].id; + const gap = ( + await write({ + type: 'WriteIssue', + effort: effortId, + title: 'No cold-cache numbers', + body: '', + kind: 'gap', + }) + ).artifacts[0].id; + const decision = ( + await write({ + type: 'WriteDecision', + effort: effortId, + title: 'Keep the reader', + body: 'It is fast enough.', + derives_from: [measured], + }) + ).artifacts[0].id; + const decisionBefore = await digest(decision); + const firstId = ( + await write({ + type: 'WriteCompaction', + effort: effortId, + title: 'Reader is fast enough', + body: 'One measurement, one Decision.', + derives_from: [measured], + }) + ).artifacts[0].id; + const fresh = await digest(firstId); + t.true( + fresh.includes('## Compaction\n- Head compaction of its Effort: yes\n') + ); + t.true( + fresh.includes( + '- Stale: no. None of the 1 record it derives from changed standing, and nothing invalidated it. A compaction is stale when' + ) + ); + t.true( + fresh.includes( + '- Since it was written, 0 records differ from the live set it covers.' + ) + ); + t.true(fresh.includes('## Changes since compaction\n- none\n')); + + // An Issue it does not derive from closes in place and a record is + // added: the compaction is not stale, but two records differ. + await write({ + type: 'ResolveIssue', + issueId: gap, + resolution: 'deferred', + resolvedBy: [measured], + }); + const added = ( + await write({ + type: 'WriteFinding', + effort: effortId, + title: 'Cold reads take 9 ms', + body: '', + kind: 'measurement', + }) + ).artifacts[0].id; + const moved = await digest(firstId); + t.true(moved.includes('- Stale: no.')); + t.true( + moved.includes( + '- Since it was written, 2 records differ from the live set it covers. This count is exact: 1 new live, 1 covered and no longer live, 0 covered with a new standing.' + ) + ); + t.true( + moved.includes( + `- left | ${gap} | issue | open -> deferred | No cold-cache numbers` + ) + ); + t.true( + moved.includes( + `- added | ${added} | finding | none -> live | Cold reads take 9 ms` + ) + ); + const brief = await handleEffortBrief(effortId, { cwd }); + t.like(brief.compaction, { head_id: firstId, stale: false, changes: 2 }); + + // Replacing the record it derives from makes it stale, and says which. + await write({ type: 'Supersede', supersederId: added, targetId: measured }); + const stale = await digest(firstId); + t.true( + stale.includes( + `- Stale: yes. 1 of the 1 record it derives from changed standing: ${measured} live -> superseded. A compaction is stale when` + ) + ); + t.like((await handleEffortBrief(effortId, { cwd })).compaction, { + stale: true, + }); + + // A newer compaction makes this one history. + const second = await write({ + type: 'WriteCompaction', + effort: effortId, + title: 'Cold reads matter', + body: 'Cold reads take 9 ms.', + derives_from: [added], + }); + const secondId = second.artifacts[0].id; + const old = await digest(firstId); + t.regex( + old, + new RegExp( + `- Head compaction of its Effort: no; superseded by ${secondId} \\(\\d{4}-\\d{2}-\\d{2}\\)` + ) + ); + t.true((await digest(secondId)).includes('- Stale: no.')); + + // Other records print what they printed before. + const decisionAfter = await digest(decision); + t.false(decisionAfter.includes('## Compaction')); + t.false(decisionAfter.includes('## Changes since compaction')); + t.is( + decisionAfter.replace(/^served_generation: .*$/m, ''), + decisionBefore.replace(/^served_generation: .*$/m, '') + ); + } +); + +test.serial( + 'records filters by standing and by dead support, and refuses a state no record stores', + async (t) => { + const cwd = await createTempProject('flatbread-effort-standing-', t); + await writeFile( + join(cwd, 'flatbread.config.js'), + `import { source } from '@flatbread/source-filesystem'; +import { transformer } from '@flatbread/transformer-markdown'; +import { proofContent } from '@flatbread/proof'; +export default { source: source(), transformer: transformer(), content: proofContent('.flatbread-proof') };` + ); + const write = async (payload: Record) => + handleEffortWrite(JSON.stringify(payload), { cwd }); + const effortId = ( + await write({ type: 'CreateEffort', title: 'Standing', body: '' }) + ).artifacts[0].id; + const wrong = ( + await write({ + type: 'WriteFinding', + effort: effortId, + title: 'Reads take 4 ms', + body: '', + kind: 'measurement', + }) + ).artifacts[0].id; + const decision = ( + await write({ + type: 'WriteDecision', + effort: effortId, + title: 'Keep the reader', + body: '', + derives_from: [wrong], + }) + ).artifacts[0].id; + const right = ( + await write({ + type: 'WriteFinding', + effort: effortId, + title: 'Reads take 9 ms cold', + body: '', + kind: 'measurement', + derives_from: [wrong], + }) + ).artifacts[0].id; + await write({ type: 'Invalidate', findingId: right, targetId: wrong }); + // A compaction rests on everything, so it is never listed as resting. + await write({ + type: 'WriteCompaction', + effort: effortId, + title: 'Reader', + body: '', + derives_from: [wrong], + }); + const ids = async (options: Record) => { + const read = await handleEffortRecords(effortId, { cwd, ...options }); + const text = await readFile(read.artifact_path, 'utf8'); + return [...text.matchAll(/^### (\S+)$/gm)].map((match) => match[1]); + }; + t.deepEqual(await ids({ standing: ['invalidated'] }), [wrong]); + // The Finding that disproved it derives from it but does not rest on it. + t.deepEqual(await ids({ restsOn: ['invalidated'] }), [decision]); + t.deepEqual(await ids({ restsOn: ['superseded'] }), []); + t.deepEqual(await ids({ state: ['proposed'] }), [decision]); + const plain = await handleEffortRecords(effortId, { + cwd, + state: ['proposed'], + }); + const header = (await readFile(plain.artifact_path, 'utf8')).split('\n')[1]; + t.false(header.includes('"standing"')); + t.false(header.includes('"rests_on"')); + await t.throwsAsync( + handleEffortRecords(effortId, { cwd, state: ['invalidated'] }), + { message: /`invalidated` is not a stored state.*--standing invalidated/ } + ); + await t.throwsAsync( + handleEffortRecords(effortId, { cwd, status: ['bogus'] }), + { message: /`bogus` is not a stored status.*--standing/ } + ); + await t.throwsAsync( + handleEffortRecords(effortId, { cwd, restsOn: ['open'] }), + { message: /Invalid rests-on standing: open/ } + ); + t.deepEqual( + mapEffortCliOptions({ standing: 'invalidated', 'rests-on': 'a,b' }), + { standing: ['invalidated'], restsOn: ['a', 'b'] } + ); + const spawned = await runCli( + cwd, + 'proof', + 'records', + effortId, + '--rests-on', + 'invalidated,rejected' + ); + t.is(spawned.code, 0, spawned.stderr); + t.regex(JSON.parse(spawned.stdout).summary, /^1 record;/); + const refused = await runCli( + cwd, + 'proof', + 'records', + effortId, + '--state', + 'invalidated' + ); + t.not(refused.code, 0); + t.regex(JSON.parse(refused.stderr).error.message, /--standing/); + } +); diff --git a/packages/flatbread/src/cli/proof.ts b/packages/flatbread/src/cli/proof.ts index cfa3ba84..94b1506e 100644 --- a/packages/flatbread/src/cli/proof.ts +++ b/packages/flatbread/src/cli/proof.ts @@ -43,6 +43,8 @@ export interface EffortCliOptions { cursor?: string; relations?: string[]; retracted?: RetractedMode; + standing?: string[]; + restsOn?: string[]; effort?: string; all?: boolean; asOf?: string; @@ -81,6 +83,8 @@ export function mapEffortCliOptions( typeof options.retracted === 'string' ? (options.retracted as RetractedMode) : undefined, + standing: split(options.standing), + restsOn: split(options['rests-on']), effort: typeof options.effort === 'string' ? options.effort : undefined, all: options.all === true || options.all === 'true' ? true : undefined, // The option parser turns `--as-of 2026` into a number; keep it as @@ -223,6 +227,8 @@ export async function handleEffortRecords( : {}), }, retracted: options.retracted, + standing: options.standing, + restsOn: options.restsOn, page: { limit: options.limit, cursor: options.cursor }, }); } @@ -590,6 +596,14 @@ export function registerProofCommands(prog: any): void { '--retracted ', 'exclude (default), include, or only retracted records' ) + .option( + '--standing ', + 'Comma-separated standings, as cards and proof state print them, links included: superseded, invalidated, retracted, …' + ) + .option( + '--rests-on ', + 'Live records that derive from, cite, or give as evidence a record in these dead standings: superseded, invalidated, rejected, retracted' + ) ); recordsOptions( prog.command('proof records ', 'Read records in an Effort') diff --git a/packages/flatbread/src/cli/serve.test.ts b/packages/flatbread/src/cli/serve.test.ts index b35214c7..a26951ce 100644 --- a/packages/flatbread/src/cli/serve.test.ts +++ b/packages/flatbread/src/cli/serve.test.ts @@ -369,6 +369,62 @@ test.serial( } ); +test.serial( + 'a served records read passes --standing and --rests-on through', + async (t) => { + const cwd = await createProject(t); + const write = async (payload: Record) => + handleEffortWrite(JSON.stringify(payload), { cwd }); + const effortId = ( + await write({ type: 'CreateEffort', title: 'E', body: '' }) + ).artifacts[0].id; + const wrong = ( + await write({ + type: 'WriteFinding', + effort: effortId, + title: 'Wrong', + body: '', + kind: 'measurement', + }) + ).artifacts[0].id; + await write({ + type: 'WriteDecision', + effort: effortId, + title: 'Built on it', + body: '', + derives_from: [wrong], + }); + const right = ( + await write({ + type: 'WriteFinding', + effort: effortId, + title: 'Right', + body: '', + kind: 'measurement', + }) + ).artifacts[0].id; + await write({ type: 'Invalidate', findingId: right, targetId: wrong }); + const standing = await runRead( + ['records', effortId, '--standing', 'invalidated'], + cwd + ); + t.is(standing.code, 0, standing.stderr); + t.regex(JSON.parse(standing.stdout).summary, /^1 record;/); + const resting = await runRead( + ['records', effortId, '--rests-on', 'invalidated', '--kinds', 'decision'], + cwd + ); + t.is(resting.code, 0, resting.stderr); + t.regex(JSON.parse(resting.stdout).summary, /^1 record; proposed 1;/); + const refused = await runRead( + ['records', effortId, '--state', 'invalidated'], + cwd + ); + t.is(refused.code, 1); + t.regex(JSON.parse(refused.stderr).error.message, /--standing invalidated/); + } +); + test.serial( 'reads fall back to the local CLI when the server is unreachable', async (t) => { diff --git a/packages/flatbread/src/proof/read.ts b/packages/flatbread/src/proof/read.ts index 0782eea0..87a472cb 100644 --- a/packages/flatbread/src/proof/read.ts +++ b/packages/flatbread/src/proof/read.ts @@ -1,11 +1,12 @@ import { FlatbreadProvider, type LoadedFlatbreadConfig } from '@flatbread/core'; import { canonicalizeReadQuery, + COMPACTION_STALE_RULE, compactionChanges, compactionHeads, - parseCovers, + compactionReport, + isCompaction, RECOMPACT_CHANGE_THRESHOLD, - staleDerivations, type CompactionChange, type CompactionStatus, type DigestSection, @@ -14,6 +15,9 @@ import { ProofDanglingRelationError, ProofInvalidCursorError, ProofReadValidationError, + DecisionFrontmatterSchema, + IssueFrontmatterSchema, + RiskFrontmatterSchema, READ_RELATIONS, readQueryHash, renderDigest, @@ -32,9 +36,15 @@ import { FIND_MAX_LIMIT, findRecords, renderFindCards, + standingOf, type FindIndex, } from './find.js'; -import { buildEffortState, type EffortState } from './state.js'; +import { + buildEffortState, + DEAD_STANDINGS, + restsOnDead, + type EffortState, +} from './state.js'; import { createHash } from 'node:crypto'; import { readdir, realpath, stat } from 'node:fs/promises'; import { join, relative, resolve } from 'node:path'; @@ -661,6 +671,10 @@ export async function getRecord( } const foreignLimit = 25; const overflow = foreignReferences.length - foreignLimit; + const sections = + displayed && isCompaction(standingRecord(displayed)) + ? await compactionSections(projection, displayed) + : undefined; return render( options, { @@ -683,6 +697,7 @@ export async function getRecord( ], anomaly, fullBody: true, + ...(sections ? { sections } : {}), } ); } @@ -699,6 +714,79 @@ const RETRACTED_MODES: readonly RetractedMode[] = [ 'only', ]; +/** + * Every word `proof records --standing` takes: the stored states and + * statuses, `live` for a Finding or Constraint, and the standings that come + * from links. They are the words cards and `proof state` print. + */ +const STANDINGS: readonly string[] = [ + 'open', + 'resolved', + 'deferred', + 'wontfix', + 'proposed', + 'accepted', + 'rejected', + 'deprecated', + 'realized', + 'mitigated', + 'live', + 'superseded', + 'invalidated', + 'retracted', +]; +/** The values `--state` and `--status` match, as the record schemas store them. */ +const STORED_STATES: readonly string[] = [ + ...new Set([ + ...DecisionFrontmatterSchema.shape.state.options, + ...RiskFrontmatterSchema.shape.state.options, + ]), +]; +const STORED_STATUSES: readonly string[] = + IssueFrontmatterSchema.shape.status.options; + +/** + * Refuses a `--state` or `--status` value no record stores. An agent that + * asks for `--state invalidated` would otherwise get an empty, complete + * page, so the message says which flags filter by standing instead. + */ +function checkStored( + field: 'state' | 'status', + values: readonly string[] | undefined, + stored: readonly string[] +): void { + const bad = (values ?? []).filter((value) => !stored.includes(value)); + if (!bad.length) return; + const example = bad.find((value) => STANDINGS.includes(value)); + const dead = + example && DEAD_STANDINGS.includes(example) ? example : 'invalidated'; + throw new ProofReadValidationError( + 'PROOF_INVALID_ARGUMENT', + `${bad.map((value) => `\`${value}\``).join(', ')} ${ + bad.length === 1 ? 'is not a stored' : 'are not stored' + } ${field}. Stored ${field} values: ${stored.join( + ', ' + )}. Standing that links or retraction decide (superseded, invalidated, retracted) is filtered with \`--standing ${ + example ?? 'invalidated' + }\`; it takes ${STANDINGS.join( + ', ' + )}. \`--rests-on ${dead}\` lists live records that derive from, cite, or give as evidence a record in that standing.` + ); +} + +function checkWords( + flag: string, + values: readonly string[] | undefined, + allowed: readonly string[] +): void { + const bad = (values ?? []).filter((value) => !allowed.includes(value)); + if (bad.length) + throw new ProofReadValidationError( + 'PROOF_INVALID_ARGUMENT', + `Invalid ${flag}: ${bad.join(', ')}. Expected ${allowed.join(', ')}.` + ); +} + export async function effortRecords( effortId: string, options: EffortProjectionOptions & { @@ -710,6 +798,10 @@ export async function effortRecords( created_at?: { gte?: string; lte?: string }; }; retracted?: RetractedMode; + /** Keeps records whose standing, as `proof state` prints it, is one of these. */ + standing?: string[]; + /** Keeps live records that rest on a record in one of these dead standings. */ + restsOn?: string[]; page?: { cursor?: string; limit?: number }; } ): Promise { @@ -721,6 +813,22 @@ export async function effortRecords( retracted )}. Expected exclude, include, or only.` ); + checkStored('state', options.where?.state, STORED_STATES); + checkStored('status', options.where?.status, STORED_STATUSES); + const standing = options.standing?.length + ? [...new Set(options.standing)].sort() + : undefined; + const restsOn = options.restsOn?.length + ? [...new Set(options.restsOn)].sort() + : undefined; + checkWords('standing', standing, STANDINGS); + checkWords('rests-on standing', restsOn, DEAD_STANDINGS); + // Retracted records are hidden by default; asking for that standing + // shows them. + const shown: RetractedMode = + standing?.includes('retracted') && retracted === 'exclude' + ? 'include' + : retracted; const projection = await makeProjection(options); const kinds = options.kinds?.length ? [...new Set(options.kinds)].sort() @@ -757,10 +865,21 @@ export async function effortRecords( .filter( (record) => record.frontmatter.effort === effortId && - (retracted === 'include' || - (retracted === 'only') === isRetracted(record)) + (shown === 'include' || (shown === 'only') === isRetracted(record)) && + (!standing || standing.includes(standingOf(record))) ) ); + if (restsOn) { + // A support may be any kind, in any Effort. The `find` index holds + // every record, as `proof state` reads them, and a served graph keeps it. + const index = await projection.findIndex(); + const get = (id: string) => index.records.get(id); + records.splice( + 0, + records.length, + ...records.filter((record) => restsOnDead(record, get, restsOn).length) + ); + } return render( options, { @@ -770,6 +889,8 @@ export async function effortRecords( ...(Object.keys(where).length ? { where } : {}), // Left out by default, so existing queries keep their hashes. ...(retracted === 'exclude' ? {} : { retracted }), + ...(standing ? { standing } : {}), + ...(restsOn ? { rests_on: restsOn } : {}), page: options.page, }, records, @@ -985,22 +1106,123 @@ function standingRecord(record: ReadRecord) { }; } -function changeLines(changes: readonly CompactionChange[]): string[] { +function changeLines( + changes: readonly CompactionChange[], + limit = BRIEF_LINE_LIMIT, + rest = 'write a new compaction' +): string[] { const lines = changes - .slice(0, BRIEF_LINE_LIMIT) + .slice(0, limit) .map( (change) => `- ${change.change} | ${change.id} | ${change.kind ?? 'unknown'} | ${ change.from ?? 'none' } -> ${change.to ?? 'missing'} | ${change.title}` ); - if (changes.length > BRIEF_LINE_LIMIT) - lines.push( - `- … ${changes.length - BRIEF_LINE_LIMIT} more; write a new compaction` - ); + if (changes.length > limit) + lines.push(`- … ${changes.length - limit} more${rest ? `; ${rest}` : ''}`); return lines; } +/** Most change lines `get` shows on a compaction. */ +const GET_CHANGE_LIMIT = 25; + +/** + * What `get` adds on a compaction: whether it is its Effort's head, whether + * it is stale and why, and each record whose standing moved since it was + * written. `brief` works these out the same way for the head, so a reader + * who opens any compaction directly gets the same answer. + */ +async function compactionSections( + projection: EngineProjection, + compaction: ReadRecord +): Promise { + const effortId = owningEffort(compaction); + const records = ( + await Promise.all( + BRIEF_COLLECTIONS.map((collection) => projection.query(collection)) + ) + ) + .flat() + .filter((record) => record.frontmatter.effort === effortId); + const byId = new Map(records.map((record) => [record.id, record])); + const standing = records.map(standingRecord); + const heads = compactionHeads(standing).map((head) => head.id); + const report = compactionReport(standingRecord(compaction), standing); + const derivesFrom = [...new Set(compaction.relations.derives_from ?? [])]; + const dated = (id: string) => + `${id} (${ + String(byId.get(id)?.frontmatter.created_at ?? '').slice(0, 10) || + 'undated' + })`; + const supersededBy = compaction.relations.superseded_by ?? []; + const invalidatedBy = compaction.relations.invalidated_by ?? []; + const head = heads.includes(compaction.id) + ? heads.length > 1 + ? `yes, one of ${heads.length} heads; the next WriteCompaction supersedes all of them` + : 'yes' + : supersededBy.length + ? `no; superseded by ${supersededBy.map(dated).join(', ')}` + : compaction.frontmatter.retracted === true + ? 'no; retracted' + : 'no'; + const reasons = [ + ...(report.stale.length + ? [ + `${report.stale.length} of the ${derivesFrom.length} ${ + derivesFrom.length === 1 ? 'record' : 'records' + } it derives from changed standing: ${report.stale + .map( + (change) => + `${change.id} ${change.from ?? 'none'} -> ${ + change.to ?? 'missing' + }` + ) + .join(', ')}.`, + ] + : []), + ...(report.invalidated + ? [`Invalidated by ${invalidatedBy.map(dated).join(', ')}.`] + : []), + ]; + const tally = (type: CompactionChange['change']) => + report.changes.filter((change) => change.change === type).length; + const differ = report.changes.length; + return [ + { + heading: 'Compaction', + lines: [ + `- Head compaction of its Effort: ${head}`, + report.isStale + ? `- Stale: yes. ${reasons.join(' ')} ${COMPACTION_STALE_RULE}` + : `- Stale: no. None of the ${derivesFrom.length} ${ + derivesFrom.length === 1 ? 'record' : 'records' + } it derives from changed standing, and nothing invalidated it. ${COMPACTION_STALE_RULE}`, + `- Since it was written, ${differ} ${ + differ === 1 ? 'record differs' : 'records differ' + } from the live set it covers. This count is exact: ${tally( + 'added' + )} new live, ${tally('left')} covered and no longer live, ${tally( + 'changed' + )} covered with a new standing.`, + ], + }, + { + heading: 'Changes since compaction', + lines: differ + ? changeLines( + report.changes, + GET_CHANGE_LIMIT, + // Nothing pages the rest. For the head, `brief` lists more. + heads.includes(compaction.id) && effortId + ? `the count above is exact; \`proof brief ${effortId}\` lists up to ${BRIEF_LINE_LIMIT}` + : 'the count above is exact' + ) + : ['- none'], + }, + ]; +} + /** * Where an Effort stands, in one bounded read: the head compaction in full, * older compactions as one-line checkpoints, and each record whose standing @@ -1043,14 +1265,10 @@ export async function effortBrief( lines: changeLines(changes), }); } else { - const covers = parseCovers(head.frontmatter.covers); - changes = compactionChanges(covers, standing); - stale = staleDerivations( - covers, - head.relations.derives_from ?? [], + ({ changes, stale, invalidated } = compactionReport( + standingRecord(head), standing - ); - invalidated = (head.relations.invalidated_by ?? []).length > 0; + )); if (heads.length > 1) anomalies.push( `compaction fork: ${heads.length} head compactions; the next WriteCompaction supersedes all of them` diff --git a/packages/flatbread/src/proof/state.test.ts b/packages/flatbread/src/proof/state.test.ts index 45a9a038..a4815d39 100644 --- a/packages/flatbread/src/proof/state.test.ts +++ b/packages/flatbread/src/proof/state.test.ts @@ -1,5 +1,10 @@ import test from 'ava'; -import type { PrimitiveKind, ReadRecord } from '@flatbread/proof'; +import { + compactionChanges, + parseCovers, + type PrimitiveKind, + type ReadRecord, +} from '@flatbread/proof'; import { buildFindIndex } from './find.js'; import { buildEffortState, STATE_CHARS } from './state.js'; @@ -300,7 +305,7 @@ test('a large Effort shows fewer entries and says how many it left out', (t) => // Sixty contested questions with long ids, each with two Decisions. for (let i = 0; i < 60; i++) { const n = String(i).padStart(2, '0'); - const long = `-${'x'.repeat(50)}--${n}`; + const long = `-${'x'.repeat(170)}--${n}`; records.push( rec(`iss-question${long}`, { kind: 'question', @@ -454,3 +459,156 @@ test('section totals count Constraints and every dead record', (t) => { }); t.is(of('Dead evidence')?.total, 5); }); + +/** A head compaction written when `fnd-old` was live, resting on it. */ +function compacted(records: ReadRecord[]): ReadRecord[] { + return [ + ...records, + rec('fnd-summary', { + kind: 'compaction', + created_at: at('01-20'), + covers: ['dec-live@accepted', 'fnd-old@live', 'iss-blk@open'], + relations: { derives_from: ['fnd-old', 'dec-live'] }, + }), + ]; +} + +function changedIds(records: ReadRecord[]): string[] { + const all = records.filter((record) => record.frontmatter.effort === 'eff-a'); + const head = all.find((record) => record.id === 'fnd-summary') as ReadRecord; + return compactionChanges( + parseCovers(head.frontmatter.covers), + all.map((record) => ({ + id: record.id, + kind: record.kind, + frontmatter: { ...record.frontmatter, ...record.relations }, + })) + ).map((change) => change.id); +} + +test('the head compaction has its own section: stale or not, and what it no longer matches', (t) => { + const records = compacted(fixture()); + const state = buildEffortState(buildFindIndex(records), 'eff-a'); + const section = sectionOf(state.markdown, 'Head compaction'); + t.regex( + section, + /- fnd-summary \(2026-01-20\)\. Stale: yes \(1 of the 2 records it derives from changed standing: fnd-old \(live -> invalidated\)\)\./ + ); + t.true( + section.includes( + '\n- A compaction is stale when a record it derives from changed standing after it was written, or when a Finding invalidated it.' + ) + ); + const changed = changedIds(records); + t.true(changed.length > 10); + const line = section + .split('\n') + .find((entry) => entry.includes('differ from the live set')); + t.regex( + line ?? '', + new RegExp( + `^- ${changed.length} records differ from the live set it covers \\(exact\\): ` + ) + ); + t.deepEqual( + [...(line ?? '').matchAll(/(\S+) \((?:new|[a-z]+ -> [a-z]+)\)/g)].map( + (match) => match[1] + ), + changed.slice(0, 10) + ); + t.true(line?.endsWith(`, and ${changed.length - 10} more`)); + t.true(line?.includes('fnd-old (live -> invalidated)')); + t.true(line?.includes('dec-p (new)')); + t.like( + state.sections.find((entry) => entry.heading === 'Head compaction'), + { total: changed.length, shown: 10 } + ); + t.false(sectionOf(state.markdown, 'Dead evidence').includes('fnd-summary')); + // A compaction rests on everything, so it is not listed as resting. + t.regex( + sectionOf(state.markdown, 'Dead evidence'), + /1 live record rests directly on 1 dead record/ + ); +}); + +test('a compaction nothing moved under reads as not stale', (t) => { + const records = [ + rec('eff-a', { created_at: at('01-01') }), + rec('fnd-m', { kind: 'measurement', created_at: at('01-02') }), + rec('fnd-summary', { + kind: 'compaction', + created_at: at('01-03'), + covers: ['fnd-m@live'], + relations: { derives_from: ['fnd-m'] }, + }), + ]; + const section = sectionOf( + buildEffortState(buildFindIndex(records), 'eff-a').markdown, + 'Head compaction' + ); + t.regex( + section, + /- fnd-summary \(2026-01-03\)\. Stale: no \(none of the 1 record it derives from changed standing; not invalidated\)\.\n- 0 records differ from the live set it covers \(exact\)\.\n/ + ); +}); + +/** The compacted fixture plus sixty contested questions with long ids. */ +function large(idLength: number): ReadRecord[] { + const records = compacted(fixture()); + for (let i = 0; i < 60; i++) { + const n = String(i).padStart(2, '0'); + const long = `-${'x'.repeat(idLength)}--${n}`; + records.push( + rec(`iss-question${long}`, { + kind: 'question', + status: 'resolved', + created_at: at('04-01'), + }), + rec(`dec-winner${long}`, { + state: 'accepted', + created_at: at('04-02'), + relations: { derives_from: [`iss-question${long}`] }, + }), + rec(`dec-loser${long}`, { + state: 'rejected', + created_at: at('04-02'), + relations: { + derives_from: [`iss-question${long}`], + rejected_by: [`dec-winner${long}`], + }, + }) + ); + } + return records; +} + +test('a large Effort cuts the compaction list before other lists, and keeps its count', (t) => { + const shown = (state: ReturnType, heading: string) => + state.sections.find((entry) => entry.heading === heading)?.shown; + // A little over the size aim: the compaction list goes to 5 first, while + // Contested topics keeps all 12 it shows at full size. + const over = buildEffortState(buildFindIndex(large(140)), 'eff-a'); + t.true(over.markdown.length <= STATE_CHARS); + t.is(shown(over, 'Head compaction'), 5); + t.is(shown(over, 'Contested topics'), 12); + // Far over: the list goes, and the exact count stays. + const records = large(180); + const state = buildEffortState(buildFindIndex(records), 'eff-a'); + const changed = changedIds(records); + t.true( + sectionOf(state.markdown, 'Head compaction').includes( + `- ${ + changed.length + } records differ from the live set it covers (exact); \`proof get fnd-summary\` lists ${ + changed.length > 25 ? '25 of them' : 'them' + }.` + ) + ); + t.like( + state.sections.find((entry) => entry.heading === 'Head compaction'), + { total: changed.length, shown: 0 } + ); + t.true(state.capped); + // Counts are never cut. + t.regex(state.markdown, / - question 62: resolved 62/); +}); diff --git a/packages/flatbread/src/proof/state.ts b/packages/flatbread/src/proof/state.ts index 5a3c44b6..153871fe 100644 --- a/packages/flatbread/src/proof/state.ts +++ b/packages/flatbread/src/proof/state.ts @@ -1,10 +1,10 @@ import { - compactionChanges, + COMPACTION_STALE_RULE, compactionHeads, + compactionReport, isCompaction, isLiveStanding, - parseCovers, - staleDerivations, + type CompactionChange, type CompactionRecord, type ReadRecord, type ReadRelation, @@ -39,6 +39,8 @@ export interface StateLimits { readonly lifecycle: number; readonly retractions: number; readonly compactionTargets: number; + /** Records the head compaction no longer matches, listed by id. */ + readonly compactionChanges: number; } export const STATE_LIMITS: StateLimits = { @@ -54,20 +56,22 @@ export const STATE_LIMITS: StateLimits = { lifecycle: 15, retractions: 10, compactionTargets: 10, + compactionChanges: 10, }; /** * Characters the Markdown aims for: under 4,000 tokens at 4 characters a * token. Ids run about 70 characters and an entry about 150, so a large * Effort fills the lists past this. The read then shows fewer entries, - * cutting the catch-all lists first: lifecycle and retractions, then live + * cutting the catch-all lists first: lifecycle, retractions, and the + * records the head compaction no longer matches (its count stays), then live * records that rest only on a replaced version, then older versions in * contested topics, then the topics themselves, then dead evidence and * Risks. Counts are never cut. */ export const STATE_CHARS = 15_000; const SHRINK_STEPS: readonly Partial[] = [ - { lifecycle: 8, retractions: 6 }, - { supersededSupports: 3 }, + { lifecycle: 8, retractions: 6, compactionChanges: 5 }, + { supersededSupports: 3, compactionChanges: 0 }, { membersPerTopic: 4 }, { contested: 8 }, { deadEvidence: 8, risks: 8, lifecycle: 4 }, @@ -117,7 +121,12 @@ const KIND_LABEL: Record = { blob: 'Blobs', }; /** Standings that take a record off the live graph. */ -const DEAD = new Set(['superseded', 'invalidated', 'rejected', 'retracted']); +export const DEAD_STANDINGS: readonly string[] = [ + 'superseded', + 'invalidated', + 'rejected', + 'retracted', +]; /** The worst first: a support shown wrong outranks one merely replaced. */ const DEAD_ORDER = ['invalidated', 'retracted', 'rejected', 'superseded']; /** What a live record rests on: what it derives from, cites, or offers as evidence. */ @@ -202,6 +211,40 @@ function standingRecord(record: ReadRecord): CompactionRecord { }; } +/** + * The dead records a live record rests on: those in `standings` that it + * derives from, cites, or gives as evidence. The record that overturned a + * support, by replacing, disproving, or rejecting it, does not rest on it: + * it derives from what it overturned. A compaction rests on nothing here, + * since a summary rests on everything. `proof state` and + * `proof records --rests-on` both use this rule, so their counts agree. + */ +export function restsOnDead( + record: ReadRecord, + get: (id: string) => ReadRecord | undefined, + standings: readonly string[] = DEAD_STANDINGS +): string[] { + if (!isCurrent(record)) return []; + const dead = new Set(); + for (const relation of SUPPORT_RELATIONS) + for (const id of ids(record, relation)) { + const target = get(id); + if (!target || !standings.includes(standingOf(target))) continue; + const overturned = ( + ['superseded_by', 'invalidated_by', 'rejected_by'] as const + ).some((by) => ids(target, by).includes(record.id)); + if (!overturned) dead.add(target.id); + } + return [...dead].sort(compareIds); +} + +/** `id (new)` or `id (from -> to)`: one record the head compaction no longer matches. */ +function changeRef(change: CompactionChange): string { + return change.change === 'added' + ? `${change.id} (new)` + : `${change.id} (${change.from ?? 'none'} -> ${change.to ?? 'missing'})`; +} + /** * Builds the state of one Effort from the `find` index's records and * chains, within STATE_CHARS when the shrink steps allow it. @@ -734,21 +777,11 @@ function stateWithin( // dead record, so the record that overturned it is named once. const rank = (standing: string) => DEAD_ORDER.indexOf(standing); const resting = new Map>(); - for (const record of all) { - if (!isLive(record) || isCompaction(standingRecord(record))) continue; - for (const relation of SUPPORT_RELATIONS) - for (const target of records(ids(record, relation))) { - // The record that overturned a support, by replacing, disproving, - // or rejecting it, does not rest on it: it derives from what it - // overturned. - const overturned = ( - ['superseded_by', 'invalidated_by', 'rejected_by'] as const - ).some((by) => ids(target, by).includes(record.id)); - if (!DEAD.has(standingOf(target)) || overturned) continue; - if (!resting.has(target.id)) resting.set(target.id, new Set()); - (resting.get(target.id) as Set).add(record.id); - } - } + for (const record of all) + for (const target of restsOnDead(record, get)) { + if (!resting.has(target)) resting.set(target, new Set()); + (resting.get(target) as Set).add(record.id); + } const deadSupports = records([...resting.keys()]).sort( (a, b) => rank(standingOf(a)) - rank(standingOf(b)) || byCreated(a, b) ); @@ -766,42 +799,6 @@ function stateWithin( ); const worst = new Map(); for (const support of deadSupports) bump(worst, standingOf(support)); - const compaction: string[] = []; - const standingRecords = all.map(standingRecord); - const head = records( - compactionHeads(standingRecords).map((record) => record.id) - ).at(-1); - if (head) { - const covers = parseCovers(head.frontmatter.covers); - const changes = compactionChanges(covers, standingRecords); - const stale = staleDerivations( - covers, - ids(head, 'derives_from'), - standingRecords - ); - const listed = stale.slice(0, limits.compactionTargets); - compaction.push( - `- Head compaction ${head.id} (${date( - head - )}): since it was written, ${count( - changes.length, - 'record' - )} differ from the live set it covers (new, no longer live, or with a new standing). ${ - stale.length - } of the records it rests on changed standing${ - stale.length - ? `: ${listed - .map( - (change) => - `${change.id} (${change.from ?? 'none'} -> ${ - change.to ?? 'missing' - })` - ) - .join(', ')}${more(stale.length, listed.length)}` - : '' - }.` - ); - } section( 'Dead evidence', [ @@ -810,7 +807,6 @@ function stateWithin( } directly on ${count(deadSupports.length, 'dead record')}${ deadSupports.length ? ` (${tallyLine(worst)})` : '' }. Resting on a record means deriving from it, citing it, or giving it as evidence.`, - ...compaction, ], deadShown.map((support) => { const live = records([...(resting.get(support.id) ?? [])]); @@ -883,6 +879,70 @@ function stateWithin( ); if (replacedLeft.length) capped = true; + // Head compaction: whether it is stale, and each record it no longer + // matches. Its count is exact even when the list is cut. + const standingRecords = all.map(standingRecord); + const head = records( + compactionHeads(standingRecords).map((record) => record.id) + ).at(-1); + if (head) { + const report = compactionReport(standingRecord(head), standingRecords); + const from = new Set(ids(head, 'derives_from')).size; + const moved = report.stale.slice(0, limits.compactionTargets); + const reasons = [ + ...(report.stale.length + ? [ + `${report.stale.length} of the ${count( + from, + 'record' + )} it derives from changed standing: ${moved + .map(changeRef) + .join(', ')}${more(report.stale.length, moved.length)}`, + ] + : []), + ...(report.invalidated + ? [`invalidated by ${refList(ids(head, 'invalidated_by'))}`] + : []), + ]; + const changes = report.changes.slice(0, limits.compactionChanges); + const differ = `${count(report.changes.length, 'record')} ${ + report.changes.length === 1 ? 'differs' : 'differ' + } from the live set it covers (exact)`; + lines.push( + '', + '### Head compaction', + `- ${head.id} (${date(head)}). Stale: ${ + report.isStale + ? `yes (${reasons.join('; ')})` + : `no (none of the ${count( + from, + 'record' + )} it derives from changed standing; not invalidated)` + }.`, + changes.length + ? `- ${differ}: ${changes.map(changeRef).join(', ')}${more( + report.changes.length, + changes.length + )}` + : `- ${differ}${ + report.changes.length + ? `; \`proof get ${head.id}\` lists ${ + // `get` lists at most 25 changes. + report.changes.length > 25 ? '25 of them' : 'them' + }` + : '' + }.`, + `- ${COMPACTION_STALE_RULE}` + ); + if (changes.length < report.changes.length) capped = true; + // Its total is the records the head no longer matches. + sections.push({ + heading: 'Head compaction', + total: report.changes.length, + shown: changes.length, + }); + } + // Lifecycle: the Effort, then dated changes to Issues, Decisions, and // records shown wrong that no list above shows. A question resolved by a // live Decision and an accepted Decision never reopened are settled and diff --git a/packages/proof/skills/proof/SKILL.md b/packages/proof/skills/proof/SKILL.md index 18427ce0..18ba10ed 100644 --- a/packages/proof/skills/proof/SKILL.md +++ b/packages/proof/skills/proof/SKILL.md @@ -191,9 +191,16 @@ Pick the read by what you need, in this order: live records that rest on superseded, disproved, rejected, or retracted ones; dated lifecycles; and retractions. It holds no facts from record bodies: use `find` for those. -3. **Read one record in full: `proof get `.** +3. **Read one record in full: `proof get `.** On a compaction, `get` + also says whether it is stale, and why, gives the exact number of + records that changed since it was written, and lists up to 25 of them. 4. **Page through records: `proof records`.** `find` shows only the top - hits, so never count from it; count from `state`. + hits, so never count from it; count from `state`. `--state` and + `--status` match only the stored field. `--standing invalidated` (or + `superseded`, `retracted`, any standing a card prints) lists records by + where they stand, links included. `--rests-on invalidated` lists the live + records that derive from, cite, or give as evidence a record in that + standing. Browse digests (`list`, `records`, `relations`, `blocking-decisions`) excerpt each body at 600 chars / 12 lines (`[…truncated]`). **`proof get` digests diff --git a/packages/proof/skills/proof/reference.md b/packages/proof/skills/proof/reference.md index b5412c38..bf981882 100644 --- a/packages/proof/skills/proof/reference.md +++ b/packages/proof/skills/proof/reference.md @@ -249,7 +249,7 @@ digest or `summary` as a data feed. ```bash flatbread proof get [--resolve exact|head] [consistency flags] flatbread proof list [--status active,paused,...] [--limit n] [--cursor c] [consistency flags] -flatbread proof records [--kinds k1,k2] [--state s1,s2] [--status s1,s2] [--kind k1,k2] [--since iso] [--until iso] [--retracted exclude|include|only] [--limit n] [--cursor c] [consistency flags] +flatbread proof records [--kinds k1,k2] [--state s1,s2] [--status s1,s2] [--standing s1,s2] [--rests-on s1,s2] [--kind k1,k2] [--since iso] [--until iso] [--retracted exclude|include|only] [--limit n] [--cursor c] [consistency flags] flatbread proof relations --relations r1,r2 [--limit n] [--cursor c] [consistency flags] flatbread proof blocking-decisions [consistency flags] flatbread proof brief [consistency flags] @@ -276,6 +276,20 @@ flatbread proof serve [--socket .flatbread/proof/serve.sock] ordered by `created_at` ascending, then `id`. - Filter semantics: AND across different flags, OR within a comma list. `--since`/`--until` bound `created_at` (gte/lte, ISO strings). +- `--state` and `--status` match the stored field only. `--state` takes + `proposed`, `accepted`, `rejected`, `superseded`, `deprecated`, `open`, + `mitigated`, `realized`; `--status` takes `open`, `resolved`, `deferred`, + `wontfix`. Any other value is refused with `PROOF_INVALID_ARGUMENT`, + never answered with an empty page. +- `--standing` matches the standing cards and `proof state` print, which + links decide before the stored field: `retracted`, then `superseded`, + then `invalidated`, else the Issue's status, the Decision's or Risk's + state, or `live`. Asking for `retracted` shows retracted records. +- `--rests-on` takes `superseded`, `invalidated`, `rejected`, `retracted`. + It lists live records, compactions left out, that derive from, cite, or + give as evidence a record in one of those standings, in any Effort. The + record that overturned it does not count. This is the rule behind the + count in `proof state`'s Dead evidence. - `--relations` values: `derives_from`, `supersedes`, `superseded_by`, `invalidates`, `invalidated_by`, `rejected_by`, `mitigated_by`, `resolved_by`, `evidence`, `cites` (one hop, explicit only). Missing targets @@ -293,6 +307,13 @@ flatbread proof serve [--socket .flatbread/proof/serve.sock] Decision to mitigate the Risk. - `--resolve head`: follow `superseded_by` to the current tip; ancestors render as checkpoint lines (max 5, then a count). +- `get` on a compaction adds `## Compaction`: whether it is its Effort's + head (or what superseded it), `Stale: yes|no` with the reason and the + rule, and the exact number of records that differ from the live set it + covers. `## Changes since compaction` then lists them as `brief` does, + up to 25 lines; past that, `- … N more` ends the list, and nothing pages + the rest (on the head, `brief` lists up to 50). For the head, these match + `brief`'s `compaction` field. - `blocking-decisions` membership (frozen): Decision in the effort with `state: proposed` whose `derives_from` directly contains an Issue in the same effort with `kind: blocker` and `status: open`. For "what blockers @@ -436,8 +457,13 @@ The Markdown has these sections, in order: record that live records still derive from, cite, or give as evidence, with those live records and, for every dead record but a replaced version, the live records one link further. The record that overturned it is not counted as - resting on it. The first line reports the head compaction: how many - records changed since it was written, and which of its supports moved. + resting on it. +- **Head compaction:** shown when the Effort has one. Whether it is stale + and why, with the rule, and the exact number of records that differ from + the live set it covers, then up to 10 of them as `id (new)` or + `id (before -> now)`. A large Effort cuts that list first, to 5 and then + none; the count stays, and the line names `proof get `, which lists + up to 25 of them. - **Lifecycle:** dated changes no section above shows, the most recently changed first: closed Issues other than settled questions, proposed or reopened @@ -452,7 +478,8 @@ that date. Records with nothing to report, such as a question Issue resolved by its one accepted Decision, appear only in the counts. Citations and Blobs have no live or dead standing, so the live total leaves them out. On a large Effort the lists shrink to fit, cutting -lifecycle and retractions first and counts never. Any cut list, a whole +lifecycle, retractions, and the head compaction's list first and counts +never. Any cut list, a whole section's or one inside an entry, ends with `… N more` or `and N more`, and then `cap_reasons` holds `state_limit`. In `state.sections`, `total` counts a section's entries in full and `shown` those it prints; for diff --git a/packages/proof/src/compaction.ts b/packages/proof/src/compaction.ts index 5e16ea6b..7dc67a0c 100644 --- a/packages/proof/src/compaction.ts +++ b/packages/proof/src/compaction.ts @@ -203,3 +203,43 @@ export function staleDerivations( return before === now ? [] : [change('changed', id, record, before)]; }); } + +/** What makes a compaction stale, in the words every read prints beside its verdict. */ +export const COMPACTION_STALE_RULE = + 'A compaction is stale when a record it derives from changed standing after it was written, or when a Finding invalidated it.'; + +export interface CompactionReport { + /** Records whose standing moved since the compaction's `covers` snapshot. */ + readonly changes: CompactionChange[]; + /** `derives_from` targets whose standing moved. */ + readonly stale: CompactionChange[]; + /** True when a Finding invalidated the compaction. */ + readonly invalidated: boolean; + /** True when any `derives_from` target moved or the compaction was invalidated. */ + readonly isStale: boolean; +} + +/** + * How a compaction stands against the Effort's records now: what changed + * since its `covers` snapshot, and whether it is stale. `brief`, `get`, and + * `state` all read it here, so they give one answer. + */ +export function compactionReport( + compaction: CompactionRecord, + effortRecords: readonly CompactionRecord[] +): CompactionReport { + const covers = parseCovers(compaction.frontmatter.covers); + const changes = compactionChanges(covers, effortRecords); + const stale = staleDerivations( + covers, + ids(compaction.frontmatter.derives_from), + effortRecords + ); + const invalidated = ids(compaction.frontmatter.invalidated_by).length > 0; + return { + changes, + stale, + invalidated, + isStale: stale.length > 0 || invalidated, + }; +} From 094ab9228e55d86afacabec193ffaabd6fe3dc1d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 15:41:00 +0000 Subject: [PATCH 2/2] docs(bench): record the paid check of round two Haiku ran `find` twice on this build on the 53 compaction, invalidation and dead-evidence questions. Against the earlier build's runs on the same questions: - tune: 60.3 against 50.3 of 62 (10 to 0, p = 0.002); - holdout: 42 against 42 of 44 (1 to 1), at 15% fewer tokens. The accuracy gain stays on the tune questions the change came from. The README and a measurement Finding say so plainly. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW --- ...ds-save-reading-but-s--shrqe59yxa1tbjad.md | 16 + benchmarks/proof-recall/README.md | 19 + .../proof-recall/runs/x10-iter2-r1-find.json | 456 ++++++++++++++++++ .../proof-recall/runs/x10-iter2-r2-find.json | 452 +++++++++++++++++ 4 files changed, 943 insertions(+) create mode 100644 .flatbread-proof/findings/fnd-compaction-and-standing-reads-save-reading-but-s--shrqe59yxa1tbjad.md create mode 100644 benchmarks/proof-recall/runs/x10-iter2-r1-find.json create mode 100644 benchmarks/proof-recall/runs/x10-iter2-r2-find.json diff --git a/.flatbread-proof/findings/fnd-compaction-and-standing-reads-save-reading-but-s--shrqe59yxa1tbjad.md b/.flatbread-proof/findings/fnd-compaction-and-standing-reads-save-reading-but-s--shrqe59yxa1tbjad.md new file mode 100644 index 00000000..034c327f --- /dev/null +++ b/.flatbread-proof/findings/fnd-compaction-and-standing-reads-save-reading-but-s--shrqe59yxa1tbjad.md @@ -0,0 +1,16 @@ +--- +id: fnd-compaction-and-standing-reads-save-reading-but-s--shrqe59yxa1tbjad +effort: eff-effort-graph-memory-and-agent-wedge--szeqvmgqjqnhd002 +title: Compaction and standing reads save reading but show no holdout gain +kind: measurement +created_at: '2026-10-19T15:41:00.659Z' +derives_from: + - fnd-search-cards-raise-accuracy-and-cut-reading-on-t--ydqm2ybyqjedzw9y +--- + +Round two of tuning changed two reads, chosen from wrong answers on the tune half of the benchmark: `get` on a compaction now says whether it is stale and what changed, and `records` filters by standing that links decide (`--standing`, `--rests-on`). Haiku then ran `find` twice on the 53 compaction, invalidation and dead-evidence questions, against the earlier build's runs. + +- Tune half: 60.3 against 50.3 of 62 question-runs (10 to 0, p = 0.002), at a third fewer tokens. +- Holdout half: 42 against 42 of 44 (1 to 1), at 15 percent fewer tokens. + +The accuracy gain stays on the questions the change was built from; holdout, already at 42 of 44, shows none. The reading saving does carry over. Treat a tune-only gain as unproven, and judge later rounds by holdout and the private graphs. diff --git a/benchmarks/proof-recall/README.md b/benchmarks/proof-recall/README.md index a8930f16..887db35c 100644 --- a/benchmarks/proof-recall/README.md +++ b/benchmarks/proof-recall/README.md @@ -1217,3 +1217,22 @@ same figures before and after on graph A and graph B, neither graph holds a compaction question, and the read invariants hold on every Effort (165 on A, 90 on B). The `--rests-on` reads hold the answer for 1 of 3 scored questions on A and 1 of 1 on B. + +#### Paid check of round two + +Haiku ran the unchanged `find` task text twice on this build, on the 53 +questions in the compaction, invalidation and dead-evidence categories +(`run.mjs --strategies find --ids `). The comparison is Run A's +`find` arm on the same questions, which used the build before this round. +The two were run on different days. Run files: `runs/x10-iter2-r{1,2}-find.json`. + +| Split | Question-runs | This build | Before | Only one right | Read tokens, this / before | +| ------- | ------------- | ---------- | ------ | -------------- | -------------------------- | +| tune | 62 | 60.3 | 50.3 | 10 to 0 | 15,498 / 23,236 | +| holdout | 44 | 42 | 42 | 1 to 1 | 13,702 / 16,164 | + +The accuracy gain is all on the tune questions this round was chosen from +(p = 0.002). On holdout there is none, though holdout was at 42 of 44 +before and had little room. The saving in reading carries over: 15% fewer +tokens on holdout. So this round is kept for clearer, cheaper reads; it has +not shown that it raises accuracy on questions it was not built from. diff --git a/benchmarks/proof-recall/runs/x10-iter2-r1-find.json b/benchmarks/proof-recall/runs/x10-iter2-r1-find.json new file mode 100644 index 00000000..7b1a944c --- /dev/null +++ b/benchmarks/proof-recall/runs/x10-iter2-r1-find.json @@ -0,0 +1,456 @@ +{ + "run": { + "name": "x10-iter2-r1-find", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "e5be653299abe9b2329822d98ffe09e90ee7cfe96f8711c813a51dc6500b887f", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "5885ca7d74b81a7cc38372121d2829d9ac8c75b1fa98024b3eafe2a8ad38b3b6" + }, + "answers": { + "cross-012": { + "answer": "61 MB/s", + "tokens": 9365, + "fixed_tokens": 2729, + "processed_tokens": 58644, + "tool_calls": 5, + "turns": 6 + }, + "netcode-010": { + "answer": "dec-use-a-6-frame-rollback-window-with-1-frame-of-in--s53944zhkw103frk", + "tokens": 14756, + "fixed_tokens": 2730, + "processed_tokens": 84867, + "tool_calls": 8, + "turns": 7 + }, + "netcode-013": { + "answer": [ + "dec-prototype-rollback-netcode-for-co-op--2tx2s2p3sfbwj8x4", + "fnd-rollback-re-simulates-a-4-frame-window-in-1-1-ms--xa30a1cez2brphyf", + "rsk-re-simulation-cost-could-blow-the-skiff-frame-bu--sk2njsp170a3qwyf" + ], + "tokens": 23254, + "fixed_tokens": 2736, + "processed_tokens": 257529, + "tool_calls": 18, + "turns": 16 + }, + "ports-009": { + "answer": "dec-load-skiff-fast-travel-without-a-loading-screen--v4qft5znysjv5f24", + "tokens": 9834, + "fixed_tokens": 2733, + "processed_tokens": 59579, + "tool_calls": 5, + "turns": 6 + }, + "ports-017": { + "answer": "fnd-skiff-microsd-slows-sharply-on-full-warm-cards--p1rgh36hat2egpz3", + "tokens": 9877, + "fixed_tokens": 2735, + "processed_tokens": 60258, + "tool_calls": 5, + "turns": 6 + }, + "ports-018": { + "answer": "504p", + "tokens": 8003, + "fixed_tokens": 1103, + "processed_tokens": 59596, + "tool_calls": 5, + "turns": 6 + }, + "ports-024": { + "answer": "fnd-skiff-firmware-1-2-rejects-pipeline-blobs-built--pernvgb2he7m3z3y", + "tokens": 11479, + "fixed_tokens": 1099, + "processed_tokens": 147628, + "tool_calls": 11, + "turns": 12 + }, + "ports-027": { + "answer": [ + "dec-stream-skiff-world-tiles-in-2-mb-reads--3jz9w9gvhhc7eppw", + "rsk-tide-streaming-outruns-skiff-storage--6q8s3b383z2tyeyb" + ], + "tokens": 24644, + "fixed_tokens": 1098, + "processed_tokens": 194073, + "tool_calls": 10, + "turns": 10 + }, + "ports-029": { + "answer": 8, + "tokens": 20748, + "fixed_tokens": 1105, + "processed_tokens": 180609, + "tool_calls": 12, + "turns": 13 + }, + "render-014": { + "answer": "52%", + "tokens": 8116, + "fixed_tokens": 1100, + "processed_tokens": 60150, + "tool_calls": 5, + "turns": 6 + }, + "render-020": { + "answer": "fnd-screen-space-refraction-loses-off-screen-content--k8zvmmdee7wg2fjy", + "tokens": 7457, + "fixed_tokens": 1098, + "processed_tokens": 57881, + "tool_calls": 5, + "turns": 6 + }, + "render-030": { + "answer": "dec-use-screen-space-refraction-for-saltglass--vknzksgdrgr7zkg6", + "tokens": 22751, + "fixed_tokens": 1100, + "processed_tokens": 202854, + "tool_calls": 14, + "turns": 12 + }, + "render-034": { + "answer": "PC workstation (Meridian clock profile)", + "tokens": 9935, + "fixed_tokens": 1100, + "processed_tokens": 90057, + "tool_calls": 7, + "turns": 8 + }, + "render-040": { + "answer": "fnd-lantern-at-alpha--ckpd9ecvdjqbcr7c", + "tokens": 14490, + "fixed_tokens": 1099, + "processed_tokens": 146324, + "tool_calls": 11, + "turns": 12 + }, + "render-041": { + "answer": "4.7 ms", + "tokens": 8333, + "fixed_tokens": 1103, + "processed_tokens": 60713, + "tool_calls": 5, + "turns": 6 + }, + "render-043": { + "answer": "fnd-meshlet-cone-culling-rejects-9-percent-of-triang--t6e5x7trzedex105", + "tokens": 7517, + "fixed_tokens": 1098, + "processed_tokens": 57964, + "tool_calls": 5, + "turns": 6 + }, + "render-045": { + "answer": "false dec-brine-curve-with-an-hdr-shoulder-at-800-nits--fpznx0cnj4avsxk1", + "tokens": 8346, + "fixed_tokens": 1101, + "processed_tokens": 59302, + "tool_calls": 5, + "turns": 6 + }, + "render-047": { + "answer": "It brought the visibility buffer back because a release-build capture put it at 7.4 ms against 8.1 ms for forward+ on Skiff, and 6.3 ms with material resolve at half rate, so the forward+ fallback was no longer worth keeping (dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1). The April 11.2 ms figure that had driven the earlier switch was invalidated by that 7.4 ms finding.", + "tokens": 8695, + "fixed_tokens": 1094, + "processed_tokens": 61122, + "tool_calls": 5, + "turns": 6 + }, + "render-052": { + "answer": "61 percent", + "tokens": 7610, + "fixed_tokens": 1096, + "processed_tokens": 58541, + "tool_calls": 5, + "turns": 6 + }, + "render-053": { + "answer": "every frame", + "tokens": 11301, + "fixed_tokens": 1108, + "processed_tokens": 120893, + "tool_calls": 9, + "turns": 10 + }, + "render-054": { + "answer": "debug-gpu-validation", + "tokens": 11757, + "fixed_tokens": 1095, + "processed_tokens": 121862, + "tool_calls": 9, + "turns": 10 + }, + "render-057": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 13409, + "fixed_tokens": 1097, + "processed_tokens": 104542, + "tool_calls": 7, + "turns": 8 + }, + "render-059": { + "answer": "fnd-sdf-gi-costs-4-7-ms-on-meridian--sm19rt2f274ajsvr", + "tokens": 11468, + "fixed_tokens": 1097, + "processed_tokens": 95093, + "tool_calls": 7, + "turns": 8 + }, + "render-061": { + "answer": "con-ui-composites-at-a-fixed-203-nits-in-hdr--de6aft7cvhdezn5s", + "tokens": 10220, + "fixed_tokens": 1098, + "processed_tokens": 115116, + "tool_calls": 9, + "turns": 10 + }, + "render-065": { + "answer": "dec-ship-tern-as-lantern-s-only-upscaler--fmreq35nzyjc7k4e", + "tokens": 10017, + "fixed_tokens": 1098, + "processed_tokens": 89099, + "tool_calls": 8, + "turns": 8 + }, + "render-070": { + "answer": [ + "dec-use-signed-distance-field-gi--yn366zmp3k7snd9w", + "dec-enable-meshlet-cone-culling-on-all-platforms--cpyweprmzcqhqsv2", + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k" + ], + "tokens": 28055, + "fixed_tokens": 1095, + "processed_tokens": 358620, + "tool_calls": 16, + "turns": 17 + }, + "render-077": { + "answer": true, + "tokens": 10149, + "fixed_tokens": 1101, + "processed_tokens": 91722, + "tool_calls": 8, + "turns": 9 + }, + "render-078": { + "answer": [ + "dec-brine-curve-with-a-separate-hdr-shoulder-at-1-00--be68yywg78dysm2e", + "rsk-hdr-certification-slips-the-release-candidate--49t4xjbzmkgc22a4" + ], + "tokens": 22402, + "fixed_tokens": 1102, + "processed_tokens": 172191, + "tool_calls": 11, + "turns": 12 + }, + "render-079": { + "answer": 5, + "tokens": 51209, + "fixed_tokens": 1100, + "processed_tokens": 521557, + "tool_calls": 17, + "turns": 18 + }, + "render-082": { + "answer": "false", + "tokens": 12185, + "fixed_tokens": 1098, + "processed_tokens": 97157, + "tool_calls": 9, + "turns": 8 + }, + "render-083": { + "answer": 4, + "tokens": 23610, + "fixed_tokens": 1099, + "processed_tokens": 235903, + "tool_calls": 12, + "turns": 13 + }, + "tools-007": { + "answer": "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "tokens": 22004, + "fixed_tokens": 1102, + "processed_tokens": 257763, + "tool_calls": 14, + "turns": 15 + }, + "tools-010": { + "answer": "dec-cap-each-farm-agent-at-6-parallel-shader-compile--q5w4sfjm73tsj8x8", + "tokens": 19931, + "fixed_tokens": 1103, + "processed_tokens": 133056, + "tool_calls": 10, + "turns": 9 + }, + "tools-013": { + "answer": 8, + "tokens": 36682, + "fixed_tokens": 1102, + "processed_tokens": 330413, + "tool_calls": 24, + "turns": 14 + }, + "tools-014": { + "answer": false, + "tokens": 18029, + "fixed_tokens": 1102, + "processed_tokens": 137129, + "tool_calls": 9, + "turns": 10 + }, + "world-019": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 7874, + "fixed_tokens": 1097, + "processed_tokens": 59252, + "tool_calls": 5, + "turns": 6 + }, + "world-026": { + "answer": "fnd-retail-skiff-units-sustain-only-190-mb-s-of-stre--485sdcmm3gweevvn", + "tokens": 7770, + "fixed_tokens": 1095, + "processed_tokens": 59064, + "tool_calls": 5, + "turns": 6 + }, + "world-030": { + "answer": "fnd-world-core-state-at-alpha--tkbsm4867h65jqq8", + "tokens": 13401, + "fixed_tokens": 1103, + "processed_tokens": 94452, + "tool_calls": 9, + "turns": 8 + }, + "world-031": { + "answer": "1.5 GB/s", + "tokens": 8497, + "fixed_tokens": 1107, + "processed_tokens": 61568, + "tool_calls": 5, + "turns": 6 + }, + "world-037": { + "answer": "fnd-hot-reload-drops-suspended-gull-coroutines--va0fn0svryzwbzws", + "tokens": 9285, + "fixed_tokens": 1099, + "processed_tokens": 86330, + "tool_calls": 7, + "turns": 8 + }, + "world-038": { + "answer": "3 archetypes", + "tokens": 10003, + "fixed_tokens": 1106, + "processed_tokens": 88312, + "tool_calls": 7, + "turns": 8 + }, + "world-047": { + "answer": "fnd-world-core-state-at-alpha--tkbsm4867h65jqq8", + "tokens": 17158, + "fixed_tokens": 1097, + "processed_tokens": 137572, + "tool_calls": 9, + "turns": 10 + }, + "world-049": { + "answer": "22%", + "tokens": 7921, + "fixed_tokens": 1103, + "processed_tokens": 59420, + "tool_calls": 5, + "turns": 6 + }, + "world-059": { + "answer": "dec-derive-the-missing-harmonic-phase-from-each-save--gbqrvvgyq2gy2gnw", + "tokens": 11084, + "fixed_tokens": 1100, + "processed_tokens": 93821, + "tool_calls": 9, + "turns": 8 + }, + "world-060": { + "answer": "dec-partition-the-world-into-192-m-cells--bk9r1c5jwb6fh8tc", + "tokens": 9530, + "fixed_tokens": 1101, + "processed_tokens": 100349, + "tool_calls": 8, + "turns": 9 + }, + "world-061": { + "answer": "236 MB", + "tokens": 13867, + "fixed_tokens": 1098, + "processed_tokens": 188682, + "tool_calls": 13, + "turns": 14 + }, + "world-066": { + "answer": "46 reads", + "tokens": 15529, + "fixed_tokens": 1102, + "processed_tokens": 153101, + "tool_calls": 10, + "turns": 11 + }, + "world-067": { + "answer": "1.2 GB/s", + "tokens": 14511, + "fixed_tokens": 1101, + "processed_tokens": 193845, + "tool_calls": 13, + "turns": 14 + }, + "world-068": { + "answer": false, + "tokens": 24066, + "fixed_tokens": 1105, + "processed_tokens": 177810, + "tool_calls": 10, + "turns": 11 + }, + "world-069": { + "answer": 7, + "tokens": 49969, + "fixed_tokens": 1099, + "processed_tokens": 547412, + "tool_calls": 31, + "turns": 18 + }, + "world-070": { + "answer": 3, + "tokens": 25642, + "fixed_tokens": 1101, + "processed_tokens": 204732, + "tool_calls": 12, + "turns": 13 + }, + "world-071": { + "answer": [ + "dec-store-components-in-archetype-tables-and-volatil--yxsjd26wcaz6fgx9", + "dec-compress-streamed-assets-with-kelp--ezwc5p1tgsgm7v8m", + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-hot-reload-gull-functions-by-swapping-bytecode-a--8dxnkg6h8vmcsr42", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 31237, + "fixed_tokens": 1098, + "processed_tokens": 293456, + "tool_calls": 13, + "turns": 14 + }, + "world-081": { + "answer": "rsk-hot-reload-may-corrupt-live-script-state--ec2ht2r1sq6h56ew", + "tokens": 41800, + "fixed_tokens": 1104, + "processed_tokens": 380805, + "tool_calls": 32, + "turns": 16 + } + } +} diff --git a/benchmarks/proof-recall/runs/x10-iter2-r2-find.json b/benchmarks/proof-recall/runs/x10-iter2-r2-find.json new file mode 100644 index 00000000..28175608 --- /dev/null +++ b/benchmarks/proof-recall/runs/x10-iter2-r2-find.json @@ -0,0 +1,452 @@ +{ + "run": { + "name": "x10-iter2-r2-find", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "e5be653299abe9b2329822d98ffe09e90ee7cfe96f8711c813a51dc6500b887f", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "5885ca7d74b81a7cc38372121d2829d9ac8c75b1fa98024b3eafe2a8ad38b3b6" + }, + "answers": { + "cross-012": { + "answer": "61 MB/s", + "tokens": 7948, + "fixed_tokens": 1107, + "processed_tokens": 59286, + "tool_calls": 5, + "turns": 6 + }, + "netcode-010": { + "answer": "dec-use-a-6-frame-rollback-window-with-1-frame-of-in--s53944zhkw103frk", + "tokens": 18732, + "fixed_tokens": 1110, + "processed_tokens": 102488, + "tool_calls": 8, + "turns": 7 + }, + "netcode-013": { + "answer": [ + "dec-prototype-rollback-netcode-for-co-op--2tx2s2p3sfbwj8x4", + "fnd-rollback-re-simulates-a-4-frame-window-in-1-1-ms--xa30a1cez2brphyf", + "rsk-re-simulation-cost-could-blow-the-skiff-frame-bu--sk2njsp170a3qwyf" + ], + "tokens": 19090, + "fixed_tokens": 1103, + "processed_tokens": 109117, + "tool_calls": 16, + "turns": 8 + }, + "ports-009": { + "answer": "dec-load-skiff-fast-travel-without-a-loading-screen--v4qft5znysjv5f24", + "tokens": 7966, + "fixed_tokens": 1099, + "processed_tokens": 59552, + "tool_calls": 5, + "turns": 6 + }, + "ports-017": { + "answer": "fnd-skiff-microsd-slows-sharply-on-full-warm-cards--p1rgh36hat2egpz3", + "tokens": 7977, + "fixed_tokens": 1102, + "processed_tokens": 59124, + "tool_calls": 5, + "turns": 6 + }, + "ports-018": { + "answer": "504p", + "tokens": 8089, + "fixed_tokens": 1100, + "processed_tokens": 59787, + "tool_calls": 5, + "turns": 6 + }, + "ports-024": { + "answer": "fnd-skiff-firmware-1-2-rejects-pipeline-blobs-built--pernvgb2he7m3z3y", + "tokens": 11987, + "fixed_tokens": 1106, + "processed_tokens": 121932, + "tool_calls": 9, + "turns": 10 + }, + "ports-027": { + "answer": [ + "dec-stream-skiff-world-tiles-in-2-mb-reads--3jz9w9gvhhc7eppw", + "rsk-tide-streaming-outruns-skiff-storage--6q8s3b383z2tyeyb" + ], + "tokens": 11537, + "fixed_tokens": 1099, + "processed_tokens": 144741, + "tool_calls": 12, + "turns": 13 + }, + "ports-029": { + "answer": 8, + "tokens": 12047, + "fixed_tokens": 1108, + "processed_tokens": 114472, + "tool_calls": 11, + "turns": 11 + }, + "render-014": { + "answer": "52 percent", + "tokens": 8517, + "fixed_tokens": 1097, + "processed_tokens": 61847, + "tool_calls": 5, + "turns": 6 + }, + "render-020": { + "answer": "fnd-screen-space-refraction-loses-off-screen-content--k8zvmmdee7wg2fjy", + "tokens": 7449, + "fixed_tokens": 1098, + "processed_tokens": 58032, + "tool_calls": 5, + "turns": 6 + }, + "render-030": { + "answer": "dec-use-screen-space-refraction-for-saltglass--vknzksgdrgr7zkg6", + "tokens": 20068, + "fixed_tokens": 1099, + "processed_tokens": 109957, + "tool_calls": 9, + "turns": 7 + }, + "render-034": { + "answer": "PC workstation (Meridian clock profile)", + "tokens": 10127, + "fixed_tokens": 1105, + "processed_tokens": 64671, + "tool_calls": 7, + "turns": 6 + }, + "render-040": { + "answer": "fnd-lantern-at-alpha--ckpd9ecvdjqbcr7c", + "tokens": 19098, + "fixed_tokens": 1097, + "processed_tokens": 188387, + "tool_calls": 12, + "turns": 13 + }, + "render-041": { + "answer": "4.7 ms", + "tokens": 8307, + "fixed_tokens": 1100, + "processed_tokens": 60591, + "tool_calls": 5, + "turns": 6 + }, + "render-043": { + "answer": "fnd-meshlet-cone-culling-rejects-9-percent-of-triang--t6e5x7trzedex105", + "tokens": 7463, + "fixed_tokens": 1096, + "processed_tokens": 57806, + "tool_calls": 5, + "turns": 6 + }, + "render-045": { + "answer": "false: dec-brine-curve-with-an-hdr-shoulder-at-800-nits--fpznx0cnj4avsxk1", + "tokens": 8191, + "fixed_tokens": 1098, + "processed_tokens": 59814, + "tool_calls": 5, + "turns": 6 + }, + "render-047": { + "answer": "It was brought back because a release-build capture measured the visibility buffer at 7.4 ms on Skiff against 8.1 ms for forward+ (6.3 ms with half-rate material resolve), so keeping the forward+ fallback was no longer worth it; the earlier 11.2 ms figure that had driven the prior decision was invalidated (dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1).", + "tokens": 8664, + "fixed_tokens": 1098, + "processed_tokens": 61015, + "tool_calls": 5, + "turns": 6 + }, + "render-052": { + "answer": "61 percent", + "tokens": 7625, + "fixed_tokens": 1098, + "processed_tokens": 58407, + "tool_calls": 5, + "turns": 6 + }, + "render-053": { + "answer": "every frame", + "tokens": 10814, + "fixed_tokens": 1095, + "processed_tokens": 118741, + "tool_calls": 9, + "turns": 10 + }, + "render-054": { + "answer": "debug-gpu-validation", + "tokens": 13387, + "fixed_tokens": 1098, + "processed_tokens": 157891, + "tool_calls": 11, + "turns": 12 + }, + "render-057": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 11775, + "fixed_tokens": 1095, + "processed_tokens": 94740, + "tool_calls": 7, + "turns": 8 + }, + "render-059": { + "answer": "fnd-sdf-gi-costs-4-7-ms-on-meridian--sm19rt2f274ajsvr", + "tokens": 11102, + "fixed_tokens": 1096, + "processed_tokens": 91091, + "tool_calls": 7, + "turns": 8 + }, + "render-061": { + "answer": "con-ui-composites-at-a-fixed-203-nits-in-hdr--de6aft7cvhdezn5s", + "tokens": 10871, + "fixed_tokens": 1099, + "processed_tokens": 143531, + "tool_calls": 11, + "turns": 12 + }, + "render-065": { + "answer": "dec-ship-tern-as-lantern-s-only-upscaler--fmreq35nzyjc7k4e", + "tokens": 9330, + "fixed_tokens": 1098, + "processed_tokens": 86706, + "tool_calls": 7, + "turns": 8 + }, + "render-070": { + "answer": ["dec-use-signed-distance-field-gi--yn366zmp3k7snd9w"], + "tokens": 68947, + "fixed_tokens": 1096, + "processed_tokens": 677931, + "tool_calls": 18, + "turns": 19 + }, + "render-077": { + "answer": true, + "tokens": 8724, + "fixed_tokens": 1100, + "processed_tokens": 77993, + "tool_calls": 8, + "turns": 8 + }, + "render-078": { + "answer": [ + "dec-brine-curve-with-a-separate-hdr-shoulder-at-1-00--be68yywg78dysm2e", + "rsk-hdr-certification-slips-the-release-candidate--49t4xjbzmkgc22a4" + ], + "tokens": 19895, + "fixed_tokens": 1102, + "processed_tokens": 178873, + "tool_calls": 12, + "turns": 13 + }, + "render-079": { + "answer": 6, + "tokens": 42137, + "fixed_tokens": 1108, + "processed_tokens": 428181, + "tool_calls": 32, + "turns": 18 + }, + "render-082": { + "answer": "false", + "tokens": 12485, + "fixed_tokens": 1104, + "processed_tokens": 126459, + "tool_calls": 11, + "turns": 10 + }, + "render-083": { + "answer": 4, + "tokens": 21154, + "fixed_tokens": 1100, + "processed_tokens": 168845, + "tool_calls": 9, + "turns": 10 + }, + "tools-007": { + "answer": "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "tokens": 14698, + "fixed_tokens": 1103, + "processed_tokens": 232224, + "tool_calls": 17, + "turns": 18 + }, + "tools-010": { + "answer": "dec-cap-each-farm-agent-at-6-parallel-shader-compile--q5w4sfjm73tsj8x8", + "tokens": 8548, + "fixed_tokens": 1102, + "processed_tokens": 69387, + "tool_calls": 6, + "turns": 7 + }, + "tools-013": { + "answer": 9, + "tokens": 75489, + "fixed_tokens": 1106, + "processed_tokens": 2055972, + "tool_calls": 50, + "turns": 43 + }, + "tools-014": { + "answer": "false fnd-flaky-nightly-builds-are-compiler-processes-runn--1sabnpb1s7004jkb", + "tokens": 12099, + "fixed_tokens": 1099, + "processed_tokens": 103794, + "tool_calls": 8, + "turns": 9 + }, + "world-019": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 7985, + "fixed_tokens": 1099, + "processed_tokens": 59482, + "tool_calls": 5, + "turns": 6 + }, + "world-026": { + "answer": "fnd-retail-skiff-units-sustain-only-190-mb-s-of-stre--485sdcmm3gweevvn", + "tokens": 7926, + "fixed_tokens": 1097, + "processed_tokens": 59495, + "tool_calls": 5, + "turns": 6 + }, + "world-030": { + "answer": "fnd-world-core-state-at-alpha--tkbsm4867h65jqq8", + "tokens": 12180, + "fixed_tokens": 1098, + "processed_tokens": 68157, + "tool_calls": 7, + "turns": 6 + }, + "world-031": { + "answer": "1.5 GB/s", + "tokens": 8447, + "fixed_tokens": 1100, + "processed_tokens": 61378, + "tool_calls": 5, + "turns": 6 + }, + "world-037": { + "answer": "fnd-hot-reload-drops-suspended-gull-coroutines--va0fn0svryzwbzws", + "tokens": 8022, + "fixed_tokens": 1101, + "processed_tokens": 59079, + "tool_calls": 5, + "turns": 6 + }, + "world-038": { + "answer": "3", + "tokens": 11280, + "fixed_tokens": 1102, + "processed_tokens": 94254, + "tool_calls": 9, + "turns": 8 + }, + "world-047": { + "answer": "fnd-world-core-state-at-alpha--tkbsm4867h65jqq8", + "tokens": 19089, + "fixed_tokens": 1098, + "processed_tokens": 159478, + "tool_calls": 10, + "turns": 11 + }, + "world-049": { + "answer": "22%", + "tokens": 7898, + "fixed_tokens": 1102, + "processed_tokens": 59356, + "tool_calls": 5, + "turns": 6 + }, + "world-059": { + "answer": "dec-derive-the-missing-harmonic-phase-from-each-save--gbqrvvgyq2gy2gnw", + "tokens": 12491, + "fixed_tokens": 1094, + "processed_tokens": 127057, + "tool_calls": 11, + "turns": 10 + }, + "world-060": { + "answer": "dec-partition-the-world-into-192-m-cells--bk9r1c5jwb6fh8tc", + "tokens": 9459, + "fixed_tokens": 1101, + "processed_tokens": 86728, + "tool_calls": 7, + "turns": 8 + }, + "world-061": { + "answer": "236 MB", + "tokens": 14966, + "fixed_tokens": 1100, + "processed_tokens": 197058, + "tool_calls": 13, + "turns": 14 + }, + "world-066": { + "answer": 46, + "tokens": 12544, + "fixed_tokens": 1106, + "processed_tokens": 142556, + "tool_calls": 10, + "turns": 11 + }, + "world-067": { + "answer": "1.2 GB/s", + "tokens": 10374, + "fixed_tokens": 1100, + "processed_tokens": 91174, + "tool_calls": 7, + "turns": 8 + }, + "world-068": { + "answer": false, + "tokens": 18523, + "fixed_tokens": 1099, + "processed_tokens": 113471, + "tool_calls": 8, + "turns": 9 + }, + "world-069": { + "answer": 7, + "tokens": 20243, + "fixed_tokens": 1100, + "processed_tokens": 197961, + "tool_calls": 19, + "turns": 13 + }, + "world-070": { + "answer": 3, + "tokens": 22901, + "fixed_tokens": 1092, + "processed_tokens": 171467, + "tool_calls": 10, + "turns": 11 + }, + "world-071": { + "answer": [ + "dec-store-components-in-archetype-tables-and-volatil--yxsjd26wcaz6fgx9", + "dec-compress-streamed-assets-with-kelp--ezwc5p1tgsgm7v8m", + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-hot-reload-gull-functions-by-swapping-bytecode-a--8dxnkg6h8vmcsr42", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 25699, + "fixed_tokens": 1099, + "processed_tokens": 286101, + "tool_calls": 14, + "turns": 15 + }, + "world-081": { + "answer": "rsk-hot-reload-may-corrupt-live-script-state--ec2ht2r1sq6h56ew", + "tokens": 29413, + "fixed_tokens": 1107, + "processed_tokens": 424199, + "tool_calls": 29, + "turns": 21 + } + } +}