diff --git a/.agents/skills/proof/SKILL.md b/.agents/skills/proof/SKILL.md index a3381bdb..18427ce0 100644 --- a/.agents/skills/proof/SKILL.md +++ b/.agents/skills/proof/SKILL.md @@ -183,9 +183,17 @@ Pick the read by what you need, in this order: terms name query words no hit contains. Ordinary words often land there; a missing distinctive term, such as a name, a number, or a code reference, is the cue that the answer may not be recorded. -2. **Read one record in full: `proof get `.** -3. **Count or page through records: `proof records`.** `find` shows only - the top hits, so never count from it. +2. **Counts, lists, and status history: `proof state `.** One + read, built from frontmatter and links, that shows exact counts by kind + and standing; open blockers and what they gate; Risks with their + mitigation; contested topics, with each rejected or replaced alternative, + what rejected or replaced it, and when; Issues closed without a fix; + live records that rest on superseded, disproved, rejected, or retracted + ones; dated lifecycles; and retractions. It holds no facts from record + bodies: use `find` for those. +3. **Read one record in full: `proof get `.** +4. **Page through records: `proof records`.** `find` shows only the top + hits, so never count from it; count from `state`. Browse digests (`list`, `records`, `relations`, `blocking-decisions`) excerpt each body at 600 chars / 12 lines (`[…truncated]`). **`proof get` digests @@ -198,6 +206,9 @@ byte cap). Zoom in with `get`, then Read/grep that digest — do not open flatbread proof find "tern history buffer precision" --effort flatbread proof find "shadow pool overflow" --as-of 2026-03-01T00:00:00Z +# Counts, open blockers, Risks, contested topics, dead evidence, lifecycles +flatbread proof state + # What's gating this effort? (proposed Decisions deriving from open blocker Issues) flatbread proof blocking-decisions @@ -255,7 +266,9 @@ server-side. `dec-adopt-a-bounded-status-briefing-fast-path-for-ef--kcw0rw39g3b2ym2h`). 2. **When you need a particular record or fact:** run `proof find` first, then `proof get` the hit whose card fits. Do not page through `records` - to look for it. + to look for it. **When you need a count, a list of open, closed, or + rejected records, a status history, the blockers, or what rests on dead + evidence:** run `proof state ` instead. 3. **When a browse digest shows `[…truncated]` and you need the body:** run `flatbread proof get `, then Read/grep that digest (`artifact_path`) for the full body. Reserve opening `.flatbread-proof/**/*.md` for rare diff --git a/.agents/skills/proof/reference.md b/.agents/skills/proof/reference.md index b8719a47..b5412c38 100644 --- a/.agents/skills/proof/reference.md +++ b/.agents/skills/proof/reference.md @@ -182,7 +182,7 @@ survivor (score 4/4 if it adds claims) plus `Retract` on the rest. `generation` is a durable, monotonic journal token — the input to strict reads. -## The 7 read queries +## The 8 read queries All reads execute through Flatbread's query engine (in-process GraphQL over the generated schema) and return a `ReadEnvelope`: @@ -211,6 +211,8 @@ lists), one-hop related records, and an edge table. Body policy: record appears as one line. - **`proof find`:** the hit cards (below) under `## Find hits`, and each hit's record with the browse excerpt. The envelope also carries the cards inline. +- **`proof state`:** no bodies at all. The state (below) under + `## Effort state`; the envelope also carries it inline as `markdown`. - **`list` / `records` / `relations` / `blocking-decisions`:** body excerpt capped at 600 chars / 12 lines (`[…truncated]`). **Blob bodies are omitted** from these digests — use `proof get` for the payload. Citation bodies @@ -233,8 +235,9 @@ order: error; do not retry the same page. 3. If `cap_reasons` is not empty, narrow the query or fail closed. It can hold several sorted, duplicate-free values from `primary_records`, - `displayed_edges`, `bytes`, and `find_limit` (`proof find` matched more - records than it shows). + `displayed_edges`, `bytes`, `find_limit` (`proof find` matched more + records than it shows), and `state_limit` (`proof state` cut at least one + list short; each cut list says how many it left out). 4. A hard cap alone leaves `page.has_more: false` and `next_cursor: null`. Paging and hard caps can occur together, but caps never create a cursor. @@ -251,13 +254,14 @@ flatbread proof relations --relations r1,r2 [--limit n] [--c flatbread proof blocking-decisions [consistency flags] flatbread proof brief [consistency flags] flatbread proof find "" [--effort | --all] [--kinds k1,k2] [--as-of iso] [--retracted exclude|include|only] [--limit n] [consistency flags] +flatbread proof state [consistency flags] flatbread proof cache prune flatbread proof serve [--socket .flatbread/proof/serve.sock] ``` -- `proof serve` keeps the graph loaded in one process and answers the seven +- `proof serve` keeps the graph loaded in one process and answers the eight reads above (`list`, `records`, `get`, `relations`, `blocking-decisions`, - `brief`, `find`); it keeps the `find` index with the graph and serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its + `brief`, `find`, `state`); it keeps the `find` index with the graph and serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its socket and those read commands go to it, in any folder whose graph root resolves to the same files. Each later read then takes a fraction of a second instead of rebuilding the graph. A write bumps the journal @@ -396,7 +400,64 @@ Each card has these lines: The card header reads `find "" in ( records searched, matched, shown)`, then `matched: … | missing: …`. The envelope adds `cards` (that Markdown) and `find`: `{"query","effort_id","searched","matched","shown","terms":{"matched","missing"},"efforts"?,"hits":[{"id","matched_id"?,"score"}]}`. -`find` shows only the top hits; use `records` to count. +`find` shows only the top hits; use `state` or `records` to count. + +### `proof state` + +`state` shows where one Effort stands, in one read of at most about 4,000 +tokens. It is built from frontmatter and links only, never from body text, +so it holds no facts or measurements: use `find` for those. It takes no +query and no `--as-of` (it refuses one); its lines date each change, and +`find --as-of` shows the version current at a date. + +The envelope adds `markdown` (the state, read it inline) and `state`: +`{"effort_id","records","live","sections":[{"heading","total","shown"}],"capped"}`. +The Markdown has these sections, in order: + +- **Counts:** every record in the Effort by kind and standing, with Issues, + Findings, and Constraints also by their own `kind` (`blocker`, `question`, + `hard`, …), how many Decisions were ever reopened, and how many records + are live or retracted. Exact numbers, never cut. +- **Open Issues and blockers:** each open Issue, blockers first, with the + live Decisions that derive from it (`gates`), other live records built on + it, and what it comes from. +- **Risks:** realized and open Risks with their evidence, mitigation, and + source; then mitigated and accepted ones, one line each. +- **Constraints in force:** the live Constraints, by kind. +- **Contested topics:** one entry per question Issue with more than one + Decision (every version counted, retracted ones left out), and per record + with more than one version. Letters (`[a]`, `[b]`) name the records + shown: the live ones (`now`), then each replaced, rejected, or disproved + one with what did it and when, and any reopening, each group newest + first. Open contests come first, then the most recently changed. +- **Closed without a fix:** Issues closed `deferred` or `wontfix`, or + resolved by no Decision or Finding. +- **Dead evidence:** each superseded, invalidated, rejected, or retracted + record that live records still derive from, cite, or give as evidence, + with those live records and, for every dead record but a replaced + version, the live records one link further. The record that overturned it is not counted as + resting on it. The first line reports the head compaction: how many + records changed since it was written, and which of its supports moved. +- **Lifecycle:** dated changes no section above shows, the most recently + changed first: + closed Issues other than settled questions, proposed or reopened + Decisions, and records shown wrong or replaced. +- **Retractions:** each retracted record with its date and reason. None is + live. + +A date on a change is the `created_at` of the record that made it, since +standings carry no dates of their own. When that record already existed, +as when a Supersede links two old records, the change came on or after +that date. Records with nothing to report, such +as a question Issue resolved by its one accepted Decision, appear only in +the counts. Citations and Blobs have no live or dead standing, so the live +total leaves them out. On a large Effort the lists shrink to fit, cutting +lifecycle and retractions first and counts never. Any cut list, a whole +section's or one inside an entry, ends with `… N more` or `and N more`, +and then `cap_reasons` holds `state_limit`. In `state.sections`, `total` +counts a section's entries in full and `shown` those it prints; for +Constraints in force and Dead evidence an entry is one Constraint or one +dead record. ### Consistency flags diff --git a/benchmarks/proof-recall/README.md b/benchmarks/proof-recall/README.md index 42c42791..c3380d8f 100644 --- a/benchmarks/proof-recall/README.md +++ b/benchmarks/proof-recall/README.md @@ -25,6 +25,7 @@ Each question has a known answer and a stated path through the graph. | `scripts/task.mjs` | The task text each session reads | | `scripts/compare.mjs` | Compares two runs question by question | | `scripts/find-recall.mjs` | Checks offline whether `proof find` reaches the evidence | +| `scripts/state-recall.mjs` | Checks offline whether `proof state` shows the answer | | `runs/` | Answers and costs from each pilot and full run | | `.flatbread-proof/` | The built graph, committed so runs need no build | | `flatbread.config.js` | Lets `flatbread proof …` read the graph from this folder | @@ -289,6 +290,33 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`. questions, chosen by a hash of each id. Read tune misses to decide what to change in `find`; run holdout once the change is done. + Before a paid run of the `findstate` strategy, run the free gate for + `proof state` too: + `node scripts/state-recall.mjs --graph --split holdout [--out ]`. + For each question it reads `proof state` for the question's Effort, or, + for a cross-Effort question, for every Effort its evidence and answer + records belong to, joined. It scores the answer in that output two ways. + The loose rule is the card rule above: every answer id appears anywhere, + or the value appears as a whole token. The strict rule scores values the + same way, but a record answer counts only when a reader could pick it + out: each answer id shares a list entry with an evidence id that is not + an answer, or some section that lists it lists no other record of the + same kind, standing, and Issue or Constraint kind. The strict rule + exists because `state` prints about 90 ids on a large Effort, so an id + somewhere in it is not yet the answer. Count, boolean, text, and + "unknown" answers are not scored. A value is set aside when it appears + in more than 10% of the outputs for questions about other Efforts, + since `state` takes no query and prints the same text for every + question about one Effort. The script prints a table by category, with + both rates, the rates for value and record answers apart, and the mean + and largest output in characters, measured with the temporary folder's + path removed so the size does not depend on the machine. It writes + `state_hash`, a hash of the `state` source. Its bar for the paid Run D: + on holdout, the strict rule finds the answer for at least 60% of the + aggregate, lifecycle, blocker, and dead-evidence questions taken + together, and the mean output is at most 16,000 characters. `--split` + works as above. + 3. Run the benchmark: ```bash @@ -297,8 +325,14 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`. `run.mjs` gives every question its own folder and its own fresh, headless `claude -p` session, once per strategy. `--strategies` picks - them from `brief` (start from `proof brief`), `nobrief`, and `find` - (start from `proof find`); the default is `brief,nobrief`. Each folder + them from `brief` (start from `proof brief`), `nobrief`, `find` + (start from `proof find`), and `findstate` (the `find` text plus one line + that says what `proof state` holds: exact counts, open Issues and + blockers, Risks, Constraints in force, contested topics, Issues closed + without a fix, dead evidence, other dated changes, and retractions, with + long lists cut); the default is `brief,nobrief`. A server started before + `proof state` existed refuses it, so stop the server and let the run + start a new one after rebuilding. Each folder holds the graph without the key map, the config, a `./flatbread` wrapper, `TASK.md`, and, afterwards, the agent's `answer.json` and `transcript.jsonl`. Sessions get only the @@ -316,14 +350,19 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`. sessions. When every session ends, `run.mjs` writes `runs/-.json` for each strategy, taking tokens and tool calls from the transcripts. Each run file records the same `find_hash` - as the gate, so you can check that both used the same search. + as the gate, so you can check that both used the same search, and the + same `state_hash` as the `state` gate. 4. Score each run with `node scripts/grade.mjs runs/-brief.json`. + Its `bySplit` field scores the tune and holdout halves apart, and + `--split holdout` scores only holdout. 5. Compare two strategies question by question with `node scripts/compare.mjs runs/-brief.json runs/-nobrief.json`. It reports both scores, the questions only one got right, a sign test on those, and read tokens, which leave out each session's fixed first - turn. + turn. `--split holdout` compares only holdout questions. Judge a read + that was changed after reading tune misses, such as `proof state`, on + holdout. 6. To pool repeated runs, give each side a comma-separated list of run files, in the same order: `node scripts/compare.mjs runs/r1-find.json,runs/r2-find.json runs/r1-nobrief.json,runs/r2-nobrief.json`. @@ -832,3 +871,99 @@ What this does not show: - The ambiguous questions come from tiers 0 to 2. They show that search finds the right Effort, not that agents handle a hard question without a named Effort. + +### The offline gate for `proof state` + +`proof state ` is one read, built from frontmatter and links, +that shows what search cannot: exact counts by kind and standing, open +blockers with the Decisions they gate, Risks and their mitigation, +Constraints in force, contested topics with each rejected or replaced +alternative, Issues closed without a fix, live records that rest on dead +ones, dated lifecycles, and retractions. Settled records, such as the +filler's question Issues and their one accepted Decision, show only in the +counts. `scripts/state-recall.mjs` scores it as the find gate scores cards, +and adds the strict rule described under "Running the benchmark". + +Only tune misses were read. On the 10x graph, 12 tune questions in the +four bar categories missed the answer. Why: + +| Where the answer was | Misses | +| -------------------------------------------------------------- | ------ | +| A value in a record body, or worked out from two body values | 5 | +| A value in a Blob payload | 4 | +| A record's author | 1 | +| A record three or more links from any record `state` lists | 1 | +| A list of the Constraints in force, which `state` only counted | 1 | + +`state` reads no bodies by design, so 10 of the 12 are `find`'s to answer. +One change came from this list: a `Constraints in force` section, made for +the tune question that asks which hard Constraints are in force. Two came +from reading the output for size, before the gate: a live record that +overturned a support (by replacing, disproving, or rejecting it) no longer +counts as resting on it, and dead evidence is grouped by the dead record, +so the record that overturned it is named once. On a large Effort the +lists shrink to fit 15,000 characters of Markdown, cutting lifecycle and +retractions first and counts never. + +Tune questions shaped `state` in two more ways. The Constraints section +answers that tune question outright. The head-compaction line first used +the exact words of a tune count question about records that joined or +left the live set; it now says "differ from the live set it covers" in +the code's own terms, but it still prints that count. A paid Run D asks +every question, tune ones included, so judge it with +`grade.mjs --split holdout` and `compare.mjs --split holdout`. + +How the strict rule came about. A review found that the loose rule's +holdout pass rested on record ids appearing anywhere: of 19 holdout hits, +18 were record answers. The strict rule was first written as "shares an +entry with a non-answer evidence id" alone. On holdout that gave 52% of +27 scored. On tune it rejected three answers that `state` plainly gives, +such as the `hard 4:` line that lists exactly the hard Constraints a +question asks for, because those questions' other evidence ids are +distractors, not the record the question is about. The "set apart" test +was added after reading those tune cases, and holdout was then scored +once more. The four tune answers it still rejects are real misses: in +each, the id shows only in a role the question does not ask about. + +Answer in the output, 10x graph (`state_hash` +`b73c82ecb0649c1db200b189b827c76250569b75327ba5847d9939352dcf1679`). +Count, boolean, text, and "unknown" answers are not scored; 18 of the 98 +bar-category questions are counts. + +| Group | Tune | Holdout | All | +| ------------------------------------- | ----------- | ----------- | ------------ | +| Bar categories, strict rule | 53% (18/34) | 55% (17/31) | 54% (35/65) | +| Bar categories, loose rule | 65% (22/34) | 61% (19/31) | 63% (41/65) | +| Bar categories, record answers strict | 74% (17/23) | 67% (16/24) | 70% (33/47) | +| Bar categories, record answers loose | 91% (21/23) | 75% (18/24) | 83% (39/47) | +| Bar categories, value answers | 9% (1/11) | 14% (1/7) | 11% (2/18) | +| aggregate, strict | 88% (7/8) | 67% (6/9) | 76% (13/17) | +| lifecycle, strict | 50% (9/18) | 56% (10/18) | 53% (19/36) | +| blocker, strict | 38% (3/8) | 44% (4/9) | 41% (7/17) | +| dead-evidence, strict | 75% (6/8) | 80% (4/5) | 77% (10/13) | +| Every question, strict | 49% (47/95) | 44% (38/86) | 47% (85/181) | +| Mean output, characters | 15,171 | 14,981 | 15,081 | +| Largest output | 28,058 | 58,681 | 58,681 | + +**The holdout does not clear the Run D bar.** The strict rule finds the +answer for 55% against the 60% bar; the mean output, 14,981 characters, +is within its 16,000. The loose rule's 61% is one question above the bar, +and two of its 19 holdout hits fail the strict rule. One Effort's output runs from 6,876 +characters (tools) to 16,127 (world), so a single read of the world +Effort is above 16,000; the bar is on the mean. The largest outputs are +cross-Effort questions, which join two to five Efforts. + +The value hits in the bar categories are Effort status answers. The +chance check sets aside standing words that `state` prints in the right +place and short numbers, so for values it errs toward missing hits, not +toward finding them. + +On the bar-category questions both gates score, the stripped `find` cards +hold the answer for 15 of 31 holdout questions, `state` for 17 by the +strict rule (19 by the loose one), and one of the two for 24 (25 loose). +The find gate, rerun with the same `find_hash` (`e5be6532…`), still gives +81% evidence in the top 5 and 62% answers on a card on the stripped set. + +The paid Run D compares `findstate` against `find` on these questions. +Before paying for it, decide whether a read that fails its own bar is +worth a paid run. diff --git a/benchmarks/proof-recall/scripts/collect.mjs b/benchmarks/proof-recall/scripts/collect.mjs index 9b0b7676..cfe0a842 100644 --- a/benchmarks/proof-recall/scripts/collect.mjs +++ b/benchmarks/proof-recall/scripts/collect.mjs @@ -31,6 +31,19 @@ export async function findHash() { .digest('hex'); } +// The `proof state` source: its sections, caps, and size aim. +const STATE_SOURCE = resolve( + dirname(fileURLToPath(import.meta.url)), + '../../../packages/flatbread/src/proof/state.ts' +); + +/** A sha256 of the `proof state` source, as `findHash` is of `find`'s. */ +export async function stateHash() { + return createHash('sha256') + .update(await readFile(STATE_SOURCE)) + .digest('hex'); +} + /** * `context_tokens` counts what entered the context once (new input, cache * writes, output): how much the agent had to read. `processed_tokens` adds diff --git a/benchmarks/proof-recall/scripts/compare.mjs b/benchmarks/proof-recall/scripts/compare.mjs index d103a2bc..9a99c197 100644 --- a/benchmarks/proof-recall/scripts/compare.mjs +++ b/benchmarks/proof-recall/scripts/compare.mjs @@ -2,13 +2,16 @@ // Compares two runs over the questions both answered, pair by pair. // // node scripts/compare.mjs runs/.json runs/.json [--skip id1,id2] -// [--only ambiguous] +// [--only ambiguous] [--split tune|holdout|all] // node scripts/compare.mjs runs/a-r1.json,runs/a-r2.json \ // runs/b-r1.json,runs/b-r2.json // // It compares the original 246 questions. `--only` names the sets to compare // instead, as run.mjs takes them, so a run that also asked the // ambiguous-Effort set reports that set apart: `--only ambiguous`. +// `--split` compares only the tune or holdout questions, split by a hash of +// each id as in find-recall.mjs. Judge a read that was changed after reading +// tune misses, such as `proof state`, on `--split holdout`. // // With several files on each side, the runs are pooled: run i of side A is // paired with run i of side B, and each question in each pair of runs is one @@ -26,6 +29,7 @@ import { readFile } from 'node:fs/promises'; import { dirname, join, resolve } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { loadQuestions } from './check.mjs'; +import { splitOf } from './find-recall.mjs'; import { score } from './grade.mjs'; import { loadGraph } from './graph.mjs'; @@ -59,10 +63,14 @@ async function main() { const skip = new Set(at >= 0 ? argv.splice(at, 2)[1].split(',') : []); const atOnly = argv.indexOf('--only'); const only = atOnly >= 0 ? argv.splice(atOnly, 2)[1].split(',') : []; + const atSplit = argv.indexOf('--split'); + const split = atSplit >= 0 ? argv.splice(atSplit, 2)[1] : 'all'; + if (!['tune', 'holdout', 'all'].includes(split)) + throw new Error('--split must be one of tune, holdout, all'); const [pathA, pathB] = argv; if (!pathB) throw new Error( - 'usage: compare.mjs [,more] [,more] [--skip ids] [--only sets]' + 'usage: compare.mjs [,more] [,more] [--skip ids] [--only sets] [--split tune|holdout|all]' ); const load = (paths) => Promise.all( @@ -101,6 +109,7 @@ async function main() { const x = a.answers?.[q.id]; const y = runsB[i].answers?.[q.id]; if (!x || !y || skip.has(q.id)) continue; + if (split !== 'all' && splitOf(q.id) !== split) continue; rows.push({ id: pooled ? `${q.id}#${i + 1}` : q.id, question: q.id, @@ -136,6 +145,7 @@ async function main() { a: name(runsA, pathA), b: name(runsB, pathB), ...(pooled ? { runs: runsA.length } : {}), + split, questions: rows.length, correct: { a: total('a'), b: total('b') }, byTier, diff --git a/benchmarks/proof-recall/scripts/find-recall.mjs b/benchmarks/proof-recall/scripts/find-recall.mjs index 45265703..f46af727 100644 --- a/benchmarks/proof-recall/scripts/find-recall.mjs +++ b/benchmarks/proof-recall/scripts/find-recall.mjs @@ -79,7 +79,7 @@ import { mkdtemp, rm, writeFile } from 'node:fs/promises'; import { request } from 'node:http'; import { tmpdir } from 'node:os'; import { dirname, join, resolve } from 'node:path'; -import { fileURLToPath } from 'node:url'; +import { fileURLToPath, pathToFileURL } from 'node:url'; import { MULTI_EFFORT_SETS, RUNS_DIR, pickQuestions } from './check.mjs'; import { findHash } from './collect.mjs'; import { loadGraph, qualify } from './graph.mjs'; @@ -96,7 +96,7 @@ const EFFORT_WORDS = ['render', 'world', 'ports', 'netcode', 'tools']; const GATE = { evidence_top5: 0.75, answer_on_card: 0.6 }; // A value answer that matches more than this share of other questions' cards // is set aside: a card shows it by accident. -const CHANCE_CUTOFF = 0.1; +export const CHANCE_CUTOFF = 0.1; const SPLITS = ['tune', 'holdout', 'all']; /** `tune` or `holdout`, from the first byte of the SHA-256 of the id. */ @@ -130,9 +130,9 @@ function args(argv) { return out; } -const sleep = (ms) => new Promise((done) => setTimeout(done, ms)); +export const sleep = (ms) => new Promise((done) => setTimeout(done, ms)); -function serverAnswers(socket) { +export function serverAnswers(socket) { return new Promise((answer) => { const req = request({ socketPath: socket, path: '/' }, (res) => { res.resume(); @@ -144,7 +144,7 @@ function serverAnswers(socket) { } /** Runs `./flatbread` in `folder`; resolves with stdout. */ -function flatbread(folder, argv, env = {}) { +export function flatbread(folder, argv, env = {}) { return new Promise((done, fail) => { const child = spawn(join(folder, 'flatbread'), argv, { cwd: folder, @@ -191,7 +191,7 @@ const list = (value) => Array.isArray(value) ? value : value === undefined ? [] : [value]; /** Record ids that count as evidence, and the answer ids, for a question. */ -function targets(graph, effort, q) { +export function targets(graph, effort, q) { const id = (key) => graph.keymap[qualify(effort, key)]; const answers = ['record', 'records'].includes(q.answer.kind) ? list(q.answer.value).map(id) @@ -210,7 +210,7 @@ const number = (text) => Number(text.replace(/,/g, '')); * accepted alternatives, each as a whole token, so "6" does not match "16" * or "6.2". Null when no card can show the answer. */ -function valueForms(q) { +export function valueForms(q) { const { kind, value, accept } = q.answer; if (kind !== 'value' || String(value).toLowerCase() === 'unknown') return null; @@ -229,7 +229,7 @@ function valueForms(q) { } /** True, false, or null (no card can show it) for one question's cards. */ -function answerOnCard(q, answers, cards) { +export function answerOnCard(q, answers, cards) { const { kind } = q.answer; if (kind === 'record' || kind === 'records') return answers.every((id) => cards.includes(id)); @@ -287,7 +287,7 @@ function scoreReply(graph, effortIds, row, stdout) { }; } -function rate(rows, field) { +export function rate(rows, field) { const known = rows.filter((r) => r[field] !== null); if (!known.length) return { n: 0, rate: null }; const yes = known.filter((r) => r[field]).length; @@ -310,7 +310,8 @@ function tally(rows) { }; } -const pct = (r) => (r.rate === null ? ' -' : `${Math.round(r.rate * 100)}%`); +export const pct = (r) => + r.rate === null ? ' -' : `${Math.round(r.rate * 100)}%`; function printTable(label, groups) { console.log(`\n${label}`); @@ -472,7 +473,8 @@ async function main() { console.log(`find_hash ${find_hash}\nWrote ${options.out}`); } -main().catch((error) => { - console.error(error.message); - process.exitCode = 1; -}); +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) + main().catch((error) => { + console.error(error.message); + process.exitCode = 1; + }); diff --git a/benchmarks/proof-recall/scripts/grade.mjs b/benchmarks/proof-recall/scripts/grade.mjs index e84bf7f9..882ec05e 100644 --- a/benchmarks/proof-recall/scripts/grade.mjs +++ b/benchmarks/proof-recall/scripts/grade.mjs @@ -2,10 +2,15 @@ // Scores one benchmark run. // // node scripts/grade.mjs runs/.json [--only ambiguous] +// [--split tune|holdout|all] // // It scores the original 246 questions that the run answered. `--only` // names the sets to score instead, as run.mjs takes them: `ambiguous` for -// the ambiguous-Effort set alone, `default,ambiguous` for both. +// the ambiguous-Effort set alone, `default,ambiguous` for both. `--split` +// scores only the tune or holdout questions, split by a hash of each id as +// in find-recall.mjs. A read that was changed after reading tune misses, +// such as `proof state`, should be judged on holdout. `bySplit` always +// reports both halves. // // A run file holds one entry per question id: // { "run": { "strategy": "brief", "model": "..." }, @@ -17,6 +22,7 @@ import { readFile } from 'node:fs/promises'; import { dirname, join, resolve } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { loadQuestions } from './check.mjs'; +import { splitOf } from './find-recall.mjs'; import { loadGraph, qualify, sameAnswer } from './graph.mjs'; const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); @@ -122,15 +128,22 @@ async function main() { const argv = process.argv.slice(2); const at = argv.indexOf('--only'); const only = at >= 0 ? argv.splice(at, 2)[1].split(',') : []; + const atSplit = argv.indexOf('--split'); + const split = atSplit >= 0 ? argv.splice(atSplit, 2)[1] : 'all'; + if (!['tune', 'holdout', 'all'].includes(split)) + throw new Error('--split must be one of tune, holdout, all'); const [runPath] = argv; - if (!runPath) throw new Error('usage: grade.mjs [--only sets]'); + if (!runPath) + throw new Error( + 'usage: grade.mjs [--only sets] [--split tune|holdout|all]' + ); const run = JSON.parse(await readFile(resolve(runPath), 'utf8')); const graph = await loadGraph(join(BENCH, '.flatbread-proof')); const rows = []; for (const set of await loadQuestions(only)) for (const q of set.questions) { const entry = run.answers?.[q.id]; - if (!entry) continue; + if (!entry || (split !== 'all' && splitOf(q.id) !== split)) continue; rows.push({ id: q.id, effort: set.effort, @@ -145,8 +158,10 @@ async function main() { JSON.stringify( { run: run.run ?? {}, + split, answered: rows.length, overall: summarize(rows, () => 'all').all, + bySplit: summarize(rows, (r) => splitOf(r.id)), byTier: summarize(rows, (r) => `tier-${r.tier}`), byEffort: summarize(rows, (r) => r.effort), byCategory: summarize(rows, (r) => r.categories), diff --git a/benchmarks/proof-recall/scripts/run.mjs b/benchmarks/proof-recall/scripts/run.mjs index 2603e378..16e08c92 100644 --- a/benchmarks/proof-recall/scripts/run.mjs +++ b/benchmarks/proof-recall/scripts/run.mjs @@ -3,7 +3,7 @@ // and per strategy, then collects answers and cost into run files. // // node scripts/run.mjs --name [--model haiku] -// [--strategies brief,nobrief,find] [--only render] [--ids a,b] +// [--strategies brief,nobrief,find,findstate] [--only render] [--ids a,b] // [--jobs 8] [--timeout 600] [--graph ] // [--server ] [--no-serve] // node scripts/run.mjs --stop-server [--server ] @@ -17,8 +17,9 @@ // answer.json, and transcript.jsonl. A rerun skips sessions that already // have both an answer and a transcript, so an interrupted run picks up where // it stopped. Keep outside the repository, so no CLAUDE.md or -// AGENTS.md reaches the agent. Each run file records `find_hash`, a hash of -// the `proof find` source, to match against the offline gate's output. +// AGENTS.md reaches the agent. Each run file records `find_hash` and +// `state_hash`, hashes of the `proof find` and `proof state` sources, to +// match against the offline gates' output. // // Runs share two things, kept under /flatbread-proof-recall/: // @@ -51,7 +52,7 @@ import { request } from 'node:http'; import { tmpdir } from 'node:os'; import { dirname, join, relative, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; -import { collect, findHash } from './collect.mjs'; +import { collect, findHash, stateHash } from './collect.mjs'; import { copyPublicGraph, prepareFolder, @@ -336,6 +337,7 @@ async function main() { if (options.stopServer) return stopServer(options.server); const questions = await publicQuestions(options); const find_hash = await findHash(); + const state_hash = await stateHash(); // Alternate strategies question by question, so a change in API speed or // load over the run falls on every strategy alike. const sessions = []; @@ -395,8 +397,8 @@ async function main() { transcript: join(s.folder, 'transcript.jsonl'), })), `${options.name}-${strategy}`, - join(BENCH, 'runs', `${options.name}-${strategy}.json`), - { find_hash } + join(RUNS_DIR, `${options.name}-${strategy}.json`), + { find_hash, state_hash } ); } diff --git a/benchmarks/proof-recall/scripts/state-recall.mjs b/benchmarks/proof-recall/scripts/state-recall.mjs new file mode 100644 index 00000000..434d85f5 --- /dev/null +++ b/benchmarks/proof-recall/scripts/state-recall.mjs @@ -0,0 +1,390 @@ +#!/usr/bin/env node +// The free offline gate for `proof state`: reads the state of each +// question's Effort and checks whether the answer is in it, before any paid +// agent run. It scores the way find-recall.mjs scores cards. +// +// node scripts/state-recall.mjs [--graph ] [--ids a,b] +// [--only ambiguous] [--split tune|holdout|all] [--out ] +// +// For each question it reads `proof state ` for the question's own +// Effort. For a question in a multi-Effort set (cross-Effort, ambiguous +// Effort), it reads the state of every Effort that its evidence and answer +// records belong to, and joins the outputs. `state` takes no query, so each +// Effort is read once and its output reused for every question about it. +// +// Per question it records: +// answer_on_output the loose rule, that of answer_on_card in +// find-recall.mjs: every answer id appears anywhere, or +// the gold value or an accepted alternative appears as a +// whole token. Null for count, boolean, text, and +// "unknown" answers. A value answer is also set aside +// (null) when it appears in more than 10% of the outputs +// for questions about other Efforts, since the state +// prints it whatever the question. Outputs of the same +// Efforts are left out of that comparison: they are the +// same text, so they would always match. +// answer_strict the strict rule. A value answer scores as above. The +// state prints about 90 ids on a large Effort, so an id +// somewhere in it is not yet the answer. A record answer +// scores only when each answer id is placed where a +// reader could pick it out, in one of two ways: +// - next to the question's subject: it shares an entry +// (a top-level list item and the lines indented under +// it) with an evidence id that is not an answer and +// not the Effort; or +// - set apart: some section that lists it lists no other +// record of the same sort that is not an answer, as +// when "Open Issues" lists exactly the open Issues the +// question asks for. A sort is what the state prints +// beside an id: its kind and standing, and for Issues +// and Constraints their own kind (blocker, hard, …). +// output_chars the length of what the CLI printed, the JSON envelope +// with the Markdown inside it, summed over joined +// Efforts, with the temporary folder's path removed so +// the length does not depend on the machine +// +// The bar for the paid Run D: on the holdout split, the strict rule finds the +// answer for at least 60% of the questions in the aggregate, lifecycle, +// blocker, and dead-evidence categories taken together (each question once), +// with a mean output of at most 16,000 characters on the 10x graph. The table +// also prints the loose rate, and both rates by answer kind. +// +// `--split` works as in find-recall.mjs: read tune misses to decide what to +// change in `state`, and score holdout once the change is done. The chance +// rates always come from every question, so a split's rows are its part of +// an `all` run. Questions are read only to score; nothing here tunes `state`. +// +// The graph is copied, without its key map, into a temporary folder, and one +// `proof serve` there answers every read. The folder is removed at the end. +import { spawn } from 'node:child_process'; +import { mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { MULTI_EFFORT_SETS, RUNS_DIR, pickQuestions } from './check.mjs'; +import { stateHash } from './collect.mjs'; +import { + answerOnCard, + CHANCE_CUTOFF, + flatbread, + pct, + rate, + serverAnswers, + sleep, + splitOf, + targets, + valueForms, +} from './find-recall.mjs'; +import { loadGraph, qualify } from './graph.mjs'; +import { prepareFolder } from './prepare-run.mjs'; + +const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const SPLITS = ['tune', 'holdout', 'all']; +const BAR = { + categories: ['aggregate', 'lifecycle', 'blocker', 'dead-evidence'], + answer_strict: 0.6, + mean_output_chars: 16000, +}; + +/** + * The entries of a state's Markdown: each top-level `- ` line with the + * indented lines under it. Headings and section intros are not entries. + */ +function entries(markdown) { + const out = []; + for (const line of markdown.split('\n')) { + if (line.startsWith('- ')) out.push(line); + else if (line.startsWith(' ') && out.length) + out[out.length - 1] += `\n${line}`; + else out.push(''); + } + return out.filter(Boolean); +} + +/** A record id as the state prints one. Efforts are left out. */ +const RECORD_ID = + /\b(?:iss|dec|fnd|con|rsk|cit|blb)-[a-z0-9-]+?--[a-z0-9]{16}\b/g; + +/** A record's standing, as `proof state` shows it in parentheses. */ +function standing(record) { + const fm = record?.frontmatter ?? {}; + const some = (value) => [value].flat().filter(Boolean).length > 0; + if (fm.retracted === true) return 'retracted'; + if (some(fm.superseded_by)) return 'superseded'; + if (some(fm.invalidated_by)) return 'invalidated'; + if (record?.kind === 'issue') return String(fm.status ?? 'open'); + if (record?.kind === 'decision' || record?.kind === 'risk') + return String(fm.state ?? 'proposed'); + return 'live'; +} + +/** The strict rule for a record answer; see the header. */ +function answerStrict(graph, answers, evidence, markdown) { + const context = [...evidence].filter( + (id) => id && !answers.includes(id) && !id.startsWith('eff-') + ); + const blocks = entries(markdown); + const nextToSubject = (id) => + blocks.some( + (block) => + block.includes(id) && context.some((other) => block.includes(other)) + ); + const sections = markdown.split(/\n### /); + const sort = (id) => { + const record = graph.records.get(id); + const own = ['issue', 'constraint'].includes(record?.kind) + ? record.frontmatter.kind + : ''; + return `${record?.kind}:${standing(record)}:${own ?? ''}`; + }; + const setApart = (id) => + sections.some( + (part) => + part.includes(id) && + [...new Set(part.match(RECORD_ID) ?? [])].every( + (other) => answers.includes(other) || sort(other) !== sort(id) + ) + ); + return answers.every((id) => nextToSubject(id) || setApart(id)); +} + +function args(argv) { + const out = { + graph: join(BENCH, '.flatbread-proof'), + ids: [], + only: [], + out: join(RUNS_DIR, 'state-recall.json'), + split: 'all', + }; + for (let i = 0; i < argv.length; i++) { + const flag = argv[i]; + if (flag === '--graph') out.graph = resolve(argv[++i]); + else if (flag === '--ids') out.ids = argv[++i].split(','); + else if (flag === '--only') out.only = argv[++i].split(','); + else if (flag === '--out') out.out = resolve(argv[++i]); + else if (flag === '--split') out.split = argv[++i]; + else throw new Error(`Unknown argument ${flag}`); + } + if (!SPLITS.includes(out.split)) + throw new Error(`--split must be one of ${SPLITS.join(', ')}`); + return out; +} + +/** The Effort keys a question is about: its own, or its records' Efforts. */ +function effortsOf(setEffort, q) { + if (!MULTI_EFFORT_SETS.includes(setEffort)) return [setEffort]; + const keys = [ + ...(q.evidence ?? []), + ...(['record', 'records'].includes(q.answer.kind) + ? [q.answer.value].flat() + : []), + ]; + return [ + ...new Set(keys.map((key) => qualify(setEffort, key).split('.')[0])), + ].sort(); +} + +function tally(rows) { + const chars = rows.map((r) => r.output_chars); + return { + n: rows.length, + answer_on_output: rate(rows, 'answer_on_output'), + answer_strict: rate(rows, 'answer_strict'), + set_aside_by_chance: rows.filter((r) => r.chance_rate > CHANCE_CUTOFF) + .length, + mean_output_chars: Math.round( + chars.reduce((a, b) => a + b, 0) / (chars.length || 1) + ), + max_output_chars: Math.max(0, ...chars), + }; +} + +function printTable(label, groups) { + console.log(`\n${label}`); + console.log( + 'group'.padEnd(24) + + 'n'.padStart(5) + + 'scored'.padStart(8) + + 'loose'.padStart(8) + + 'scored'.padStart(8) + + 'strict'.padStart(8) + + 'chance'.padStart(8) + + 'chars'.padStart(8) + + 'max'.padStart(8) + ); + for (const [name, t] of Object.entries(groups)) + console.log( + name.padEnd(24) + + String(t.n).padStart(5) + + String(t.answer_on_output.n).padStart(8) + + pct(t.answer_on_output).padStart(8) + + String(t.answer_strict.n).padStart(8) + + pct(t.answer_strict).padStart(8) + + String(t.set_aside_by_chance).padStart(8) + + String(t.mean_output_chars).padStart(8) + + String(t.max_output_chars).padStart(8) + ); +} + +async function main() { + const options = args(process.argv.slice(2)); + const graph = await loadGraph(options.graph); + const effortIds = Object.fromEntries( + Object.entries(graph.keymap) + .filter(([key]) => key.endsWith('.effort')) + .map(([key, id]) => [key.split('.')[0], id]) + ); + if (graph.records.size < 1000) + console.log( + `Note: this graph holds ${graph.records.size} records. The bar that ` + + 'counts is the one on the 10x graph (--graph ).' + ); + const state_hash = await stateHash(); + const jobs = []; + for (const set of await pickQuestions(options)) + for (const q of set.questions) + jobs.push({ q, effort: set.effort, efforts: effortsOf(set.effort, q) }); + + const folder = await mkdtemp(join(tmpdir(), 'state-recall-')); + const socket = join(folder, 'serve.sock'); + await prepareFolder(folder, [], options.graph); + const server = spawn( + join(folder, 'flatbread'), + ['proof', 'serve', '--socket', socket], + { cwd: folder, stdio: 'ignore' } + ); + const outputs = new Map(); + try { + for (let i = 0; !(await serverAnswers(socket)); i++) { + if (i > 600) throw new Error('proof serve did not start'); + await sleep(200); + } + for (const key of [...new Set(jobs.flatMap((job) => job.efforts))].sort()) + outputs.set( + key, + await flatbread(folder, ['proof', 'state', effortIds[key]], { + FLATBREAD_PROOF_SERVER: socket, + }) + ); + } finally { + server.kill(); + await rm(folder, { recursive: true, force: true }); + } + + const rows = jobs.map(({ q, effort, efforts }) => { + const printed = efforts.map((key) => outputs.get(key)); + const text = printed.map((out) => JSON.parse(out).markdown).join('\n\n'); + const { answers, evidence } = targets(graph, effort, q); + const loose = answerOnCard(q, answers, text); + return { + id: q.id, + effort, + efforts, + categories: q.categories, + answer_kind: q.answer.kind, + answer_on_output: loose, + answer_strict: answers.length + ? answerStrict(graph, answers, evidence, text) + : loose, + output_chars: printed.reduce( + (sum, out) => sum + out.split(folder).join('.').length, + 0 + ), + chance_rate: null, + q, + text, + }; + }); + // A value the state prints for other Efforts' questions too shows by chance. + for (const row of rows) { + const forms = valueForms(row.q); + if (!forms) continue; + const others = rows.filter( + (r) => r.id !== row.id && !r.efforts.some((e) => row.efforts.includes(e)) + ); + if (!others.length) continue; + const hits = others.filter((r) => + forms.some((form) => form.test(r.text)) + ).length; + row.chance_rate = +(hits / others.length).toFixed(3); + if (row.chance_rate > CHANCE_CUTOFF) + row.answer_on_output = row.answer_strict = null; + } + for (const row of rows) { + delete row.q; + delete row.text; + } + + const inSplit = (id) => + options.split === 'all' || splitOf(id) === options.split; + const kept = rows.filter((r) => inSplit(r.id)); + const groups = { all: tally(kept) }; + const target = kept.filter((r) => + r.categories.some((c) => BAR.categories.includes(c)) + ); + groups['bar categories'] = tally(target); + groups[' value answers'] = tally( + target.filter((r) => !['record', 'records'].includes(r.answer_kind)) + ); + groups[' record answers'] = tally( + target.filter((r) => ['record', 'records'].includes(r.answer_kind)) + ); + const byCategory = {}; + for (const r of kept) + for (const c of r.categories) (byCategory[c] ??= []).push(r); + for (const [c, group] of Object.entries(byCategory).sort(([a], [b]) => + a.localeCompare(b) + )) + groups[c] = tally(group); + printTable( + `graph ${options.graph}, split ${options.split} (${kept.length} questions)`, + groups + ); + const t = groups['bar categories']; + const all = groups.all; + const pass = + options.split !== 'tune' && + t.answer_strict.rate >= BAR.answer_strict && + all.mean_output_chars <= BAR.mean_output_chars; + console.log( + `\nRun D bar (${BAR.categories.join( + ', ' + )}): answer found by the strict rule for ${pct(t.answer_strict)} of ${ + t.answer_strict.n + } scored (needs ${ + BAR.answer_strict * 100 + }% on holdout; the loose rule, any id anywhere, finds ${pct( + t.answer_on_output + )} of ${t.answer_on_output.n}); mean output ${ + all.mean_output_chars + } characters (needs at most ${BAR.mean_output_chars}).${ + options.split === 'holdout' ? ` ${pass ? 'PASS' : 'FAIL'}` : '' + }` + ); + const efforts = Object.fromEntries( + [...outputs].map(([key, out]) => [key, out.split(folder).join('.').length]) + ); + await writeFile( + options.out, + `${JSON.stringify( + { + graph: options.graph, + split: options.split, + questions: kept.length, + state_hash, + bar: { ...BAR, chance_cutoff: CHANCE_CUTOFF, pass }, + output_chars_by_effort: efforts, + groups, + rows: kept, + }, + null, + 2 + )}\n` + ); + console.log(`state_hash ${state_hash}\nWrote ${options.out}`); +} + +main().catch((error) => { + console.error(error.message); + process.exitCode = 1; +}); diff --git a/benchmarks/proof-recall/scripts/task.mjs b/benchmarks/proof-recall/scripts/task.mjs index abc7bbe4..cbcded5e 100644 --- a/benchmarks/proof-recall/scripts/task.mjs +++ b/benchmarks/proof-recall/scripts/task.mjs @@ -2,14 +2,18 @@ // commands its strategy allows, and one question. Never the answer. import { join } from 'node:path'; -export const STRATEGIES = ['brief', 'nobrief', 'find']; +export const STRATEGIES = ['brief', 'nobrief', 'find', 'findstate']; const BRIEF_LINE = '- `./flatbread proof brief `: an Effort\'s latest summary (a "compaction") in full, plus every record that changed since. Its envelope\'s `compaction.stale` is true when a record the summary relies on has changed. Start each Effort you need here.'; const FIND_LINE = "- `./flatbread proof find [--all | --effort ] [--kinds k1,k2] [--as-of iso] [--retracted include|only] [--limit 5]` ranks records by the words they contain, counting all versions of a record as one hit. Its envelope's `cards` field shows each hit's id, standing, best-matching passage, versions, and links."; +const STATE_LINE = + "- `./flatbread proof state ` prints where one Effort stands, built from its records' frontmatter and links, in the envelope's `markdown` field; its digest repeats it, so you need not read that. It holds exact counts by kind and standing; open Issues and blockers with what they gate; Risks; Constraints in force; contested topics, with each rejected or replaced version; Issues closed without a fix; live records that rest on dead ones; other dated changes; and retractions. Records with nothing to report, such as an accepted Decision never reopened, appear only in the counts. Long lists are cut and end with `… N more` or `and N more`; page the rest with `records`. It holds nothing from record bodies."; const NO_BRIEF = '\nDo not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n'; +const FIND_FIRST_STATE = + '\nStart with `find`, or with `state` for counts and the lists it holds. Pass `--all` when the question names no Effort, and `--effort ` when it does (`list` gives Effort ids). Run `get` on a hit to read its full text. Use `records` to page through records. Do not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n'; const FIND_FIRST = '\nStart with `find`. Pass `--all` when the question names no Effort, and `--effort ` when it does (`list` gives Effort ids). Run `get` on a hit to read its full text. Use `records` only to count records or to page through them. Do not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n'; @@ -18,7 +22,9 @@ export function taskText({ folder, question, format, strategy }) { if (!STRATEGIES.includes(strategy)) throw new Error(`Unknown strategy ${strategy}`); const brief = strategy === 'brief'; - const find = strategy === 'find'; + // `findstate` is the `find` text plus the `state` line. + const find = strategy === 'find' || strategy === 'findstate'; + const state = strategy === 'findstate'; return `# One-question recall test Answer one question about a project-memory graph, using only its read commands. Every command you run is counted, so be efficient, but a right answer matters most. @@ -29,16 +35,16 @@ Answer one question about a project-memory graph, using only its read commands. ## Read commands Run from the folder. Each prints a JSON envelope; its \`artifact_path\` names a Markdown digest. Read or grep that digest for evidence. -${ - find ? FIND_LINE + '\n' : '' -}- \`./flatbread proof list --status active,paused,completed,abandoned\` lists Efforts (the default shows only active ones). +${find ? FIND_LINE + '\n' : ''}${ + state ? STATE_LINE + '\n' : '' + }- \`./flatbread proof list --status active,paused,completed,abandoned\` lists Efforts (the default shows only active ones). ${ brief ? BRIEF_LINE + '\n' : '' }- \`./flatbread proof records [--kinds issue,finding,decision,constraint,risk,citation,blob] [--state s] [--status s] [--kind k] [--retracted include|only] [--limit 25] [--cursor c]\` browses records. Bodies are cut at 600 characters or 12 lines; Blob bodies are left out. Retracted records are left out unless you pass \`--retracted\`, and are marked RETRACTED when shown. - \`./flatbread proof get [--resolve head]\` shows one record in full, Blob payloads included. \`--resolve head\` follows replacements to the newest version. - \`./flatbread proof relations --relations derives_from,supersedes,superseded_by,invalidates,invalidated_by,resolved_by,rejected_by,mitigated_by,evidence,cites\` follows one hop of links. - \`./flatbread proof blocking-decisions \`. -${brief ? '' : find ? FIND_FIRST : NO_BRIEF} +${brief ? '' : state ? FIND_FIRST_STATE : find ? FIND_FIRST : NO_BRIEF} Records can be superseded (replaced), invalidated (shown wrong), rejected, reopened, or retracted (noise, hidden from browse reads unless asked; \`get\` still shows them). Titles repeat on purpose. Efforts never link to each other; one may name another's record by title in prose. A summary can be out of date; check its claims against the records. ## The question diff --git a/packages/flatbread/bin/flatbread.js b/packages/flatbread/bin/flatbread.js index d436f3b0..eba4a855 100755 --- a/packages/flatbread/bin/flatbread.js +++ b/packages/flatbread/bin/flatbread.js @@ -35,6 +35,7 @@ const SERVED_READS = new Set([ 'blocking-decisions', 'brief', 'find', + 'state', ]); /** diff --git a/packages/flatbread/src/cli/proof.ts b/packages/flatbread/src/cli/proof.ts index 78bc74ba..cfa3ba84 100644 --- a/packages/flatbread/src/cli/proof.ts +++ b/packages/flatbread/src/cli/proof.ts @@ -17,7 +17,9 @@ import { effortBrief, effortFind, effortRecords, + effortState, type FindEnvelope, + type StateEnvelope, relations, getRecord, listEfforts, @@ -299,6 +301,21 @@ export async function handleEffortFind( }); } +/** `proof state`: where one Effort stands, from frontmatter and links. */ +export async function handleEffortState( + effortId: string, + options: EffortCliOptions = {} +): Promise { + const cwd = options.cwd ?? process.cwd(); + return effortState(effortId, { + cwd, + rootDir: await rootFor(cwd), + cacheRoot: resolve(cwd, '.flatbread/proof'), + consistency: consistency(options), + asOf: options.asOf, + }); +} + export interface EffortBootstrapReport { readonly status: 'ready' | 'action_required'; readonly config_path: string | null; @@ -608,6 +625,16 @@ export function registerProofCommands(prog: any): void { ) ) ); + consistencyOptions( + prog.command( + 'proof state ', + 'Read counts, contested topics, blockers, risks, dead evidence, and lifecycles' + ) + ) + .option('--as-of ', 'Not supported; refused with a pointer to find') + .action(async (effortId: string, options: Record) => + printResult(handleEffortState(effortId, mapEffortCliOptions(options))) + ); paginationOptions( consistencyOptions( prog.command('proof list', 'List Efforts by lifecycle status') diff --git a/packages/flatbread/src/cli/serve.test.ts b/packages/flatbread/src/cli/serve.test.ts index 0c773d51..b35214c7 100644 --- a/packages/flatbread/src/cli/serve.test.ts +++ b/packages/flatbread/src/cli/serve.test.ts @@ -282,6 +282,93 @@ test.serial( } ); +test.serial( + 'proof state prints the same envelope locally and through proof serve', + async (t) => { + t.timeout(180_000); + const cwd = await createProject(t); + const write = async (input: Record) => + (await handleEffortWrite(JSON.stringify(input), { cwd })).artifacts[0].id; + const effortId = await write({ + type: 'CreateEffort', + title: 'Renderer', + body: '', + }); + const question = await write({ + type: 'WriteIssue', + effort: effortId, + title: 'Which shadow technique?', + body: '', + kind: 'question', + }); + const chosen = await write({ + type: 'WriteDecision', + effort: effortId, + title: 'Use virtual shadow maps', + body: '', + derives_from: [question], + }); + const passed = await write({ + type: 'WriteDecision', + effort: effortId, + title: 'Use cascaded shadow maps', + body: '', + derives_from: [question], + }); + await write({ type: 'AcceptDecision', decisionId: chosen }); + const blocker = await write({ + type: 'WriteIssue', + effort: effortId, + title: 'Shadow pages run out', + body: '', + kind: 'blocker', + }); + const socket = join(cwd, '.flatbread/proof/serve.sock'); + const server = await serveProofReads(socket); + t.teardown(() => new Promise((done) => server.close(() => done()))); + const served = { FLATBREAD_PROOF_SERVER: socket }; + + const local = await runCli(cwd, {}, 'proof', 'state', effortId); + t.is(local.code, 0, local.stderr); + const remote = await runCli(cwd, served, 'proof', 'state', effortId); + t.is(remote.code, 0, remote.stderr); + t.deepEqual(JSON.parse(remote.stdout), JSON.parse(local.stdout)); + const envelope = JSON.parse(local.stdout); + t.is(envelope.state.effort_id, effortId); + t.false(envelope.state.capped); + t.true(envelope.complete); + t.regex(envelope.markdown, /- Decisions 2: accepted 1, rejected 1/); + t.regex( + envelope.markdown, + new RegExp(`\\[b\\] ${passed}: rejected by \\[a\\]`) + ); + t.regex(envelope.markdown, new RegExp(`- ${blocker} \\(open\\), blocker`)); + t.regex(envelope.summary, /^state of /); + t.regex( + await readFile(envelope.artifact_path, 'utf8'), + /## Effort state\n/ + ); + + for (const env of [{}, served] as Record[]) { + // --as-of is refused, with a pointer to find, both ways. + const asOf = await runCli( + cwd, + env, + 'proof', + 'state', + effortId, + '--as-of', + '2026-01-01' + ); + t.is(asOf.code, 1); + t.regex(JSON.parse(asOf.stderr).error.message, /proof find --as-of/); + const missing = await runCli(cwd, env, 'proof', 'state', question); + t.is(missing.code, 1); + t.is(JSON.parse(missing.stderr).error.code, 'PROOF_INVALID_ARGUMENT'); + } + } +); + test.serial( 'reads fall back to the local CLI when the server is unreachable', async (t) => { diff --git a/packages/flatbread/src/cli/serve.ts b/packages/flatbread/src/cli/serve.ts index e0b5d69a..17c8b1ce 100644 --- a/packages/flatbread/src/cli/serve.ts +++ b/packages/flatbread/src/cli/serve.ts @@ -13,6 +13,7 @@ import { handleEffortList, handleEffortRecords, handleEffortRelations, + handleEffortState, mapEffortCliOptions, } from './proof.js'; @@ -30,6 +31,7 @@ export const SERVED_READS: Readonly> = { 'blocking-decisions': 1, brief: 1, find: 1, + state: 1, }; /** Options that take no value. */ @@ -130,6 +132,8 @@ export async function runRead( ? await handleEffortBlockingDecisions(first, mapped) : command === 'find' ? await handleEffortFind(first, mapped) + : command === 'state' + ? await handleEffortState(first, mapped) : await handleEffortBrief(first, mapped); return { stdout: `${JSON.stringify(result)}\n`, stderr: '', code: 0 }; } catch (error) { diff --git a/packages/flatbread/src/proof/read.ts b/packages/flatbread/src/proof/read.ts index 217da941..0782eea0 100644 --- a/packages/flatbread/src/proof/read.ts +++ b/packages/flatbread/src/proof/read.ts @@ -34,6 +34,7 @@ import { renderFindCards, type FindIndex, } from './find.js'; +import { buildEffortState, type EffortState } from './state.js'; import { createHash } from 'node:crypto'; import { readdir, realpath, stat } from 'node:fs/promises'; import { join, relative, resolve } from 'node:path'; @@ -1296,3 +1297,58 @@ export async function effortFind( cards, }; } + +/** A `state` envelope: the usual read fields plus the state inline. */ +export interface StateEnvelope extends ReadEnvelope { + /** Exact counts and how much of each list is shown. */ + state: Omit; + /** The state as Markdown, so the agent needs no second Read. */ + markdown: string; +} + +/** + * Where an Effort stands, from frontmatter and links alone: exact counts, + * contested topics, open blockers and risks, Issues closed without a fix, + * live records on dead evidence, dated lifecycles, and retractions. It + * shares the `find` index, so a served graph builds both once. + */ +export async function effortState( + effortId: string, + options: EffortProjectionOptions & { asOf?: string } +): Promise { + if (options.asOf !== undefined) + throw new ProofReadValidationError( + 'PROOF_INVALID_ARGUMENT', + 'proof state does not take --as-of. Its lifecycle lines date each change; proof find --as-of shows the version current at a date.' + ); + const projection = await makeProjection(options); + const index = await projection.findIndex(); + if (index.records.get(effortId)?.kind !== 'effort') + throw new ProofReadValidationError( + 'PROOF_INVALID_ARGUMENT', + `Effort ${effortId} does not exist` + ); + const { markdown, ...state } = buildEffortState(index, effortId); + const envelope = await render( + options, + { type: 'effortState', effort_id: effortId }, + [], + [], + [`effortRecords("${effortId}")`, `blockingDecisions("${effortId}")`], + { + sections: [{ heading: 'Effort state', lines: markdown.split('\n') }], + capReasons: state.capped ? ['state_limit'] : [], + } + ); + return { + ...envelope, + summary: `state of ${effortId}: ${state.live} live of ${ + state.records + } records; ${state.sections + .filter((section) => section.heading !== 'Counts') + .map((section) => `${section.heading.toLowerCase()} ${section.total}`) + .join(', ')}${state.capped ? '; some lists cut' : ''}`, + state, + markdown, + }; +} diff --git a/packages/flatbread/src/proof/state.test.ts b/packages/flatbread/src/proof/state.test.ts new file mode 100644 index 00000000..45a9a038 --- /dev/null +++ b/packages/flatbread/src/proof/state.test.ts @@ -0,0 +1,456 @@ +import test from 'ava'; +import type { PrimitiveKind, ReadRecord } from '@flatbread/proof'; +import { buildFindIndex } from './find.js'; +import { buildEffortState, STATE_CHARS } from './state.js'; + +function rec( + id: string, + fields: Record & { + relations?: ReadRecord['relations']; + } = {} +): ReadRecord { + const prefix = id.split('-', 1)[0]; + const kind = ( + { + eff: 'effort', + iss: 'issue', + fnd: 'finding', + dec: 'decision', + con: 'constraint', + rsk: 'risk', + cit: 'citation', + blb: 'blob', + } as Record + )[prefix]; + const { relations, ...frontmatter } = fields; + return { + id, + kind, + path: `${id}.md`, + frontmatter: { + title: id, + created_at: '2026-01-01T00:00:00.000Z', + ...(kind === 'effort' ? { status: 'active' } : { effort: 'eff-a' }), + ...frontmatter, + }, + body_excerpt: '', + relations: relations ?? {}, + }; +} + +const at = (day: string) => `2026-${day}T10:00:00.000Z`; + +/** A small Effort with one record or more for every section. */ +function fixture(): ReadRecord[] { + return [ + rec('eff-a', { title: 'Renderer', created_at: at('01-01') }), + // A contested question: one rejected Decision, one replaced, one live. + rec('iss-q', { + kind: 'question', + status: 'resolved', + created_at: at('01-02'), + relations: { resolved_by: ['dec-b'] }, + }), + rec('dec-a', { + state: 'rejected', + created_at: at('01-03'), + relations: { derives_from: ['iss-q'], rejected_by: ['dec-b'] }, + }), + rec('dec-b', { + state: 'accepted', + created_at: at('01-03'), + relations: { derives_from: ['iss-q'], superseded_by: ['dec-c'] }, + }), + rec('dec-c', { + state: 'accepted', + created_at: at('02-01'), + relations: { supersedes: ['dec-b'] }, + }), + // Settled filler: a question its one accepted Decision resolves. + rec('iss-filler', { + kind: 'question', + status: 'resolved', + created_at: at('01-04'), + relations: { resolved_by: ['dec-filler'] }, + }), + rec('dec-filler', { + state: 'accepted', + created_at: at('01-04'), + relations: { derives_from: ['iss-filler'] }, + }), + // An open blocker that gates a proposed Decision. + rec('fnd-capture', { kind: 'measurement', created_at: at('01-05') }), + rec('iss-blk', { + kind: 'blocker', + status: 'open', + created_at: at('01-06'), + relations: { derives_from: ['fnd-capture'] }, + }), + rec('dec-p', { + state: 'proposed', + created_at: at('01-07'), + relations: { derives_from: ['iss-blk'] }, + }), + // Risks in three states. + rec('rsk-open', { + state: 'open', + created_at: at('01-08'), + relations: { derives_from: ['dec-c'] }, + }), + rec('fnd-ev', { kind: 'measurement', created_at: at('03-01') }), + rec('rsk-real', { + state: 'realized', + created_at: at('01-09'), + relations: { evidence: ['fnd-ev'] }, + }), + rec('rsk-mit', { + state: 'mitigated', + created_at: at('01-10'), + relations: { mitigated_by: ['dec-c'] }, + }), + // Closed without a fix: won't fix, and resolved by a Citation only. + rec('fnd-why', { kind: 'measurement', created_at: at('01-12') }), + rec('iss-gap', { + kind: 'gap', + status: 'wontfix', + created_at: at('01-11'), + relations: { resolved_by: ['fnd-why'] }, + }), + rec('cit-link', { created_at: at('01-13') }), + rec('iss-def', { + kind: 'defect', + status: 'resolved', + created_at: at('01-11'), + relations: { resolved_by: ['cit-link'] }, + }), + // Dead evidence: a live Decision on a disproved Finding, a Risk one link + // further, and the disproving Finding, which does not rest on it. + rec('fnd-old', { + kind: 'measurement', + created_at: at('01-14'), + relations: { invalidated_by: ['fnd-new'] }, + }), + rec('fnd-new', { + kind: 'measurement', + created_at: at('02-10'), + relations: { derives_from: ['fnd-old'], invalidates: ['fnd-old'] }, + }), + rec('dec-live', { + state: 'accepted', + created_at: at('01-15'), + relations: { derives_from: ['fnd-old'] }, + }), + rec('rsk-on', { + state: 'open', + created_at: at('01-16'), + relations: { derives_from: ['dec-live'] }, + }), + // Lifecycle: a fixed defect and a reopened Decision. + rec('iss-bug', { + kind: 'defect', + status: 'resolved', + created_at: at('01-17'), + relations: { resolved_by: ['dec-c'] }, + }), + rec('dec-reopened', { + state: 'accepted', + created_at: at('01-18'), + reopen_history: [{ at: at('02-05'), reason: 'x', rejected_by: 'dec-c' }], + }), + // A retraction and a Constraint in force. + rec('fnd-ret', { + kind: 'measurement', + created_at: at('01-19'), + retracted: true, + retracted_at: at('01-20'), + retracted_reason: 'Duplicate write', + }), + rec('con-h', { kind: 'hard', created_at: at('01-21') }), + // Another Effort's record never counts here. + rec('dec-other', { effort: 'eff-b', state: 'rejected' }), + ]; +} + +function sectionOf(markdown: string, heading: string): string { + const start = markdown.indexOf(`### ${heading}\n`); + if (start < 0) return ''; + const next = markdown.indexOf('\n### ', start + 1); + return markdown.slice(start, next < 0 ? undefined : next); +} + +test('counts every record by kind, standing, and own kind', (t) => { + const state = buildEffortState(buildFindIndex(fixture()), 'eff-a'); + const counts = sectionOf(state.markdown, 'Counts'); + t.regex(counts, /- Issues 6: resolved 4, open 1, wontfix 1\n/); + t.regex(counts, / - blocker 1: open 1/); + t.regex(counts, / - question 2: resolved 2/); + t.regex( + counts, + /- Decisions 7: accepted 4, proposed 1, rejected 1, superseded 1/ + ); + t.regex(counts, /- Risks 4: open 2, mitigated 1, realized 1/); + t.regex(counts, /- Findings 6: live 4, invalidated 1, retracted 1/); + t.regex(counts, /- Decisions ever reopened: 1; now accepted 1/); + t.regex( + counts, + /- Live records: 14 of 25; Citations and Blobs \(1\) are never counted live; retracted: 1/ + ); + t.is(state.records, 25); + t.is(state.live, 14); + t.false(state.markdown.includes('dec-other')); +}); + +test('lists contested topics with what replaced or rejected each version', (t) => { + const state = buildEffortState(buildFindIndex(fixture()), 'eff-a'); + const topics = sectionOf(state.markdown, 'Contested topics'); + t.regex(topics, /^1 topic:/m); + t.regex( + topics, + /- iss-q \(resolved\), question with 3 Decisions; resolved by \[b\]/ + ); + t.regex(topics, /\[a\] dec-c: now accepted, written 2026-02-01/); + t.regex(topics, /\[b\] dec-b: superseded by \[a\] on 2026-02-01/); + t.regex(topics, /\[c\] dec-a: rejected by \[b\] on 2026-01-03/); + // Settled filler costs only its counts. + t.false(state.markdown.includes('iss-filler')); + t.false(state.markdown.includes('dec-filler')); +}); + +test('shows open blockers with what they gate, and Risks with their state', (t) => { + const { markdown } = buildEffortState(buildFindIndex(fixture()), 'eff-a'); + const open = sectionOf(markdown, 'Open Issues and blockers'); + t.regex(open, /1 open blocker, 1 open Issue in all/); + t.regex(open, /- iss-blk \(open\), blocker, open since 2026-01-06/); + t.regex(open, / - gates: dec-p \(proposed\)/); + t.regex(open, / - comes from: fnd-capture \(live\)/); + const risks = sectionOf(markdown, 'Risks'); + t.regex( + risks, + /- rsk-real \(realized\), raised 2026-01-09; realized on 2026-03-01, evidence fnd-ev \(live\)\n - mitigation: none/ + ); + t.regex( + risks, + /- rsk-open \(open\), raised 2026-01-08\n - mitigation: none; comes from: dec-c \(accepted\)/ + ); + t.regex( + risks, + /- rsk-mit \(mitigated\), raised 2026-01-10; mitigated on 2026-02-01 by dec-c \(accepted\)$/m + ); + // Realized comes before open, and mitigated after both. + t.true(risks.indexOf('rsk-real') < risks.indexOf('rsk-open')); + t.true(risks.indexOf('rsk-open') < risks.indexOf('rsk-mit')); + t.regex(sectionOf(markdown, 'Constraints in force'), /- hard 1: con-h$/m); +}); + +test('lists Issues closed without a fix', (t) => { + const { markdown } = buildEffortState(buildFindIndex(fixture()), 'eff-a'); + const closed = sectionOf(markdown, 'Closed without a fix'); + t.regex( + closed, + /- iss-gap \(wontfix\), gap, opened 2026-01-11; closed wontfix on 2026-01-12 by fnd-why \(live\)/ + ); + t.regex( + closed, + /- iss-def \(resolved\), defect, opened 2026-01-11; closed resolved on 2026-01-13 by cit-link \(live\)/ + ); + t.false(closed.includes('iss-bug')); +}); + +test('lists live records on dead evidence, but not the record that overturned it', (t) => { + const { markdown } = buildEffortState(buildFindIndex(fixture()), 'eff-a'); + const dead = sectionOf(markdown, 'Dead evidence'); + t.regex( + dead, + /1 live record rests directly on 1 dead record \(invalidated 1\)/ + ); + t.regex( + dead, + /- fnd-old, invalidated by fnd-new on 2026-02-10; rested on by dec-live \(accepted\)\n - and, one link further, by rsk-on \(open\)/ + ); + t.false(dead.includes('rested on by fnd-new')); +}); + +test('dates lifecycle changes not shown elsewhere, and lists retractions', (t) => { + const { markdown } = buildEffortState(buildFindIndex(fixture()), 'eff-a'); + t.regex(markdown, /"Renderer", status active, created 2026-01-01/); + const lifecycle = sectionOf(markdown, 'Lifecycle'); + t.regex( + lifecycle, + /- iss-bug \(resolved\), defect: opened 2026-01-17; resolved on 2026-02-01 by dec-c \(accepted\)/ + ); + t.regex( + lifecycle, + /- dec-reopened \(accepted\): written 2026-01-18; reopened on 2026-02-05 after rejection by dec-c/ + ); + // Newest first. + t.true(lifecycle.indexOf('dec-reopened') < lifecycle.indexOf('iss-bug')); + // Shown in other sections, so not repeated here. + for (const id of ['dec-p', 'dec-a', 'fnd-old', 'iss-gap']) + t.false(lifecycle.includes(id), id); + const retractions = sectionOf(markdown, 'Retractions'); + t.regex(retractions, /1 retracted record, none of them live./); + t.regex( + retractions, + /- fnd-ret, finding \(measurement\), written 2026-01-19, retracted on 2026-01-20: Duplicate write/ + ); +}); + +test('a large Effort shows fewer entries and says how many it left out', (t) => { + const records = fixture(); + // Sixty contested questions with long ids, each with two Decisions. + for (let i = 0; i < 60; i++) { + const n = String(i).padStart(2, '0'); + const long = `-${'x'.repeat(50)}--${n}`; + records.push( + rec(`iss-question${long}`, { + kind: 'question', + status: 'resolved', + created_at: at('04-01'), + }), + rec(`dec-winner${long}`, { + state: 'accepted', + created_at: at('04-02'), + relations: { derives_from: [`iss-question${long}`] }, + }), + rec(`dec-loser${long}`, { + state: 'rejected', + created_at: at('04-02'), + relations: { + derives_from: [`iss-question${long}`], + rejected_by: [`dec-winner${long}`], + }, + }) + ); + } + const state = buildEffortState(buildFindIndex(records), 'eff-a'); + t.true(state.capped); + t.true(state.markdown.length <= STATE_CHARS); + const topics = state.sections.find((s) => s.heading === 'Contested topics'); + t.is(topics?.total, 61); + t.true((topics?.shown ?? 0) < 61); + t.regex( + state.markdown, + new RegExp(`- … ${61 - (topics?.shown ?? 0)} more topics`) + ); + // Counts are never cut. + t.regex(state.markdown, / - question 62: resolved 62/); +}); + +test('a list cut inside an entry marks the state capped', (t) => { + const records = [ + rec('eff-a', { created_at: at('01-01') }), + rec('iss-q', { kind: 'question', status: 'open', created_at: at('01-02') }), + ]; + for (let i = 1; i <= 7; i++) + records.push( + rec(`dec-option-${i}`, { + state: 'proposed', + created_at: at(`01-${String(i + 2).padStart(2, '0')}`), + relations: { derives_from: ['iss-q'] }, + }) + ); + const state = buildEffortState(buildFindIndex(records), 'eff-a'); + t.regex(state.markdown, / - gates: .*, and 3 more/); + t.true(state.capped); + // The topic keeps the newest live versions and names what it left out. + const topics = sectionOf(state.markdown, 'Contested topics'); + t.regex(topics, /\[a\] dec-option-7: now proposed/); + t.regex(topics, / - … 2 more versions, 2 of them live/); + t.false(topics.includes('dec-option-1:')); +}); + +test('a retracted duplicate Decision is not a version in a topic', (t) => { + const records = [ + rec('eff-a', { created_at: at('01-01') }), + rec('iss-q', { + kind: 'question', + status: 'resolved', + created_at: at('01-02'), + relations: { resolved_by: ['dec-one'] }, + }), + rec('dec-one', { + state: 'accepted', + created_at: at('01-03'), + relations: { derives_from: ['iss-q'] }, + }), + rec('dec-dup', { + state: 'accepted', + created_at: at('01-03'), + retracted: true, + retracted_at: at('01-03'), + relations: { derives_from: ['iss-q'] }, + }), + ]; + const { markdown } = buildEffortState(buildFindIndex(records), 'eff-a'); + t.regex(sectionOf(markdown, 'Contested topics'), /^0 topics:/m); + t.regex(sectionOf(markdown, 'Retractions'), /- dec-dup, decision/); +}); + +test('dates a replaced Risk by what replaced it, and sorts lifecycle by change', (t) => { + const records = [ + rec('eff-a', { created_at: at('01-01') }), + rec('fnd-early', { created_at: at('03-02') }), + rec('rsk-old', { + state: 'realized', + created_at: at('01-04'), + relations: { evidence: ['fnd-early'], superseded_by: ['rsk-new'] }, + }), + rec('rsk-new', { + state: 'open', + created_at: at('05-04'), + relations: { supersedes: ['rsk-old'] }, + }), + rec('fnd-late', { created_at: at('09-01') }), + rec('iss-old', { + kind: 'defect', + status: 'resolved', + created_at: at('01-05'), + relations: { resolved_by: ['fnd-late'] }, + }), + rec('iss-new', { + kind: 'defect', + status: 'resolved', + created_at: at('03-01'), + relations: { resolved_by: ['fnd-early'] }, + }), + ]; + const { markdown } = buildEffortState(buildFindIndex(records), 'eff-a'); + t.regex( + sectionOf(markdown, 'Risks'), + /- rsk-old \(superseded\), raised 2026-01-04; superseded by rsk-new on 2026-05-04$/m + ); + const lifecycle = sectionOf(markdown, 'Lifecycle'); + t.true(lifecycle.indexOf('iss-old') < lifecycle.indexOf('iss-new')); +}); + +test('section totals count Constraints and every dead record', (t) => { + const records = [rec('eff-a', { created_at: at('01-01') })]; + for (let i = 1; i <= 3; i++) + records.push(rec(`con-h${i}`, { kind: 'hard', created_at: at('01-02') })); + // Five replaced Findings, each still cited by a live Decision. + for (let i = 1; i <= 5; i++) + records.push( + rec(`fnd-v${i}`, { + created_at: at('01-03'), + relations: { superseded_by: [`fnd-w${i}`] }, + }), + rec(`fnd-w${i}`, { + created_at: at('01-04'), + relations: { supersedes: [`fnd-v${i}`] }, + }), + rec(`dec-on${i}`, { + state: 'accepted', + created_at: at('01-05'), + relations: { derives_from: [`fnd-v${i}`] }, + }) + ); + const state = buildEffortState(buildFindIndex(records), 'eff-a'); + const of = (heading: string) => + state.sections.find((s) => s.heading === heading); + t.deepEqual(of('Constraints in force'), { + heading: 'Constraints in force', + total: 3, + shown: 3, + }); + t.is(of('Dead evidence')?.total, 5); +}); diff --git a/packages/flatbread/src/proof/state.ts b/packages/flatbread/src/proof/state.ts new file mode 100644 index 00000000..5a3c44b6 --- /dev/null +++ b/packages/flatbread/src/proof/state.ts @@ -0,0 +1,991 @@ +import { + compactionChanges, + compactionHeads, + isCompaction, + isLiveStanding, + parseCovers, + staleDerivations, + type CompactionRecord, + type ReadRecord, + type ReadRelation, +} from '@flatbread/proof'; +import { standingOf, type FindIndex } from './find.js'; + +/** + * `proof state`: one bounded read of where an Effort stands, built only from + * frontmatter and links, never from body text. It gives what a search + * cannot: exact counts, contested topics, open blockers and risks, Issues + * closed without a fix, live records that rest on dead ones, dated + * lifecycles, and retractions. Records with nothing to report, such as a + * question Issue resolved by its one accepted Decision, show only in the + * counts, so the read stays small on a large Effort. + * + * Lists name records by id and standing, without titles: a Proof id starts + * with its title's words, so a title would mostly repeat it. + */ + +/** Most entries each list shows. A list cut short says how many it left out. */ +export interface StateLimits { + readonly openIssues: number; + readonly risks: number; + readonly contested: number; + readonly membersPerTopic: number; + readonly closedWithoutFix: number; + readonly deadEvidence: number; + /** Dead-evidence entries whose dead record was only replaced by a newer version. */ + readonly supersededSupports: number; + readonly constraints: number; + readonly linksPerLine: number; + readonly lifecycle: number; + readonly retractions: number; + readonly compactionTargets: number; +} + +export const STATE_LIMITS: StateLimits = { + openIssues: 10, + risks: 10, + contested: 12, + membersPerTopic: 5, + closedWithoutFix: 10, + deadEvidence: 12, + supersededSupports: 12, + constraints: 12, + linksPerLine: 4, + lifecycle: 15, + retractions: 10, + compactionTargets: 10, +}; +/** + * Characters the Markdown aims for: under 4,000 tokens at 4 characters a + * token. Ids run about 70 characters and an entry about 150, so a large + * Effort fills the lists past this. The read then shows fewer entries, + * cutting the catch-all lists first: lifecycle and retractions, then live + * records that rest only on a replaced version, then older versions in + * contested topics, then the topics themselves, then dead evidence and + * Risks. Counts are never cut. + */ +export const STATE_CHARS = 15_000; +const SHRINK_STEPS: readonly Partial[] = [ + { lifecycle: 8, retractions: 6 }, + { supersededSupports: 3 }, + { membersPerTopic: 4 }, + { contested: 8 }, + { deadEvidence: 8, risks: 8, lifecycle: 4 }, + { contested: 6, membersPerTopic: 3, openIssues: 8, closedWithoutFix: 6 }, +]; +const REASON_CHARS = 40; + +export interface StateSection { + readonly heading: string; + /** + * Entries the section has in full. In Constraints in force an entry is + * one Constraint, and in Dead evidence one dead record, though both + * print several on a line. + */ + readonly total: number; + /** Entries it shows; fewer than `total` means the list was cut. */ + readonly shown: number; +} + +export interface EffortState { + readonly effort_id: string; + /** Records in the Effort, the Effort itself left out, and how many are live. */ + readonly records: number; + readonly live: number; + readonly sections: readonly StateSection[]; + /** True when any list was cut. */ + readonly capped: boolean; + readonly markdown: string; +} + +const KIND_ORDER = [ + 'issue', + 'decision', + 'finding', + 'constraint', + 'risk', + 'citation', + 'blob', +] as const; +const KIND_LABEL: Record = { + issue: 'Issues', + decision: 'Decisions', + finding: 'Findings', + constraint: 'Constraints', + risk: 'Risks', + citation: 'Citations', + blob: 'Blobs', +}; +/** Standings that take a record off the live graph. */ +const DEAD = new Set(['superseded', 'invalidated', 'rejected', 'retracted']); +/** The worst first: a support shown wrong outranks one merely replaced. */ +const DEAD_ORDER = ['invalidated', 'retracted', 'rejected', 'superseded']; +/** What a live record rests on: what it derives from, cites, or offers as evidence. */ +const SUPPORT_RELATIONS: readonly ReadRelation[] = [ + 'derives_from', + 'evidence', + 'cites', +]; +const RISK_ORDER = ['realized', 'open', 'mitigated', 'accepted']; + +function text(value: unknown): string { + return typeof value === 'string' ? value : ''; +} + +function compareIds(a: string, b: string): number { + return a < b ? -1 : a > b ? 1 : 0; +} + +function time(record: ReadRecord | undefined): number { + const value = Date.parse(text(record?.frontmatter.created_at)); + return Number.isNaN(value) ? 0 : value; +} + +function byCreated(a: ReadRecord, b: ReadRecord): number { + return time(a) - time(b) || compareIds(a.id, b.id); +} + +function day(value: unknown): string { + return text(value).slice(0, 10) || 'undated'; +} + +function date(record: ReadRecord | undefined): string { + return day(record?.frontmatter.created_at); +} + +function cut(value: string, max: number): string { + const flat = value.replace(/\s+/g, ' ').trim(); + return [...flat].length > max + ? `${[...flat].slice(0, max - 1).join('')}…` + : flat; +} + +function subKind(record: ReadRecord): string { + return text(record.frontmatter.kind); +} + +function ids(record: ReadRecord, relation: ReadRelation): string[] { + return [...(record.relations[relation] ?? [])]; +} + +function isLive(record: ReadRecord): boolean { + return isLiveStanding(record.kind, standingOf(record)); +} + +/** Live and not a compaction: a summary rests on everything, so lists of links leave it out. */ +function isCurrent(record: ReadRecord): boolean { + return isLive(record) && !isCompaction(standingRecord(record)); +} + +/** `1 topic`, `2 topics`: a count with its noun. */ +function count(n: number, noun: string, plural = `${noun}s`): string { + return `${n} ${n === 1 ? noun : plural}`; +} + +function bump(map: Map, key: string): void { + map.set(key, (map.get(key) ?? 0) + 1); +} + +/** Standing words with their counts, most common first, then by name. */ +function tallyLine(counts: ReadonlyMap): string { + return [...counts] + .sort(([a, x], [b, y]) => y - x || compareIds(a, b)) + .map(([word, n]) => `${word} ${n}`) + .join(', '); +} + +function standingRecord(record: ReadRecord): CompactionRecord { + return { + id: record.id, + kind: record.kind, + frontmatter: { ...record.frontmatter, ...record.relations }, + }; +} + +/** + * Builds the state of one Effort from the `find` index's records and + * chains, within STATE_CHARS when the shrink steps allow it. + */ +export function buildEffortState( + index: FindIndex, + effortId: string +): EffortState { + let limits = STATE_LIMITS; + let state = stateWithin(index, effortId, limits); + for (const step of SHRINK_STEPS) { + if (state.markdown.length <= STATE_CHARS) break; + limits = { ...limits, ...step }; + state = stateWithin(index, effortId, limits); + } + return state; +} + +function stateWithin( + index: FindIndex, + effortId: string, + limits: StateLimits +): EffortState { + const effort = index.records.get(effortId); + const all = [...index.records.values()] + .filter((record) => record.frontmatter.effort === effortId) + .sort(byCreated); + const get = (id: string) => index.records.get(id); + const records = (list: readonly string[]): ReadRecord[] => + list + .map(get) + .filter((record): record is ReadRecord => record !== undefined) + .sort(byCreated); + const inbound = (id: string, label: string): ReadRecord[] => + records(index.inbound.get(id)?.get(label) ?? []); + /** The newest of the records `list` names: the one that made a change. */ + const latest = (list: readonly string[]) => records(list).at(-1); + /** `id (standing)`, the form every list uses for a record. */ + const ref = (id: string): string => { + const record = get(id); + if (!record) return `${id} (missing)`; + const foreign = + record.kind !== 'effort' && record.frontmatter.effort !== effortId + ? `, Effort ${text(record.frontmatter.effort)}` + : ''; + return `${id} (${standingOf(record)}${foreign})`; + }; + /** True once any list, or any list inside an entry, leaves records out. */ + let capped = false; + /** `, and N more` for a list cut to `shown` of `total`, and marks the cut. */ + const more = (total: number, shown: number): string => { + if (total <= shown) return ''; + capped = true; + return `, and ${total - shown} more`; + }; + const refList = (list: readonly string[]): string => { + const shown = list.slice(0, limits.linksPerLine); + return `${shown.map(ref).join(', ')}${more(list.length, shown.length)}`; + }; + /** + * What took a record off the live graph, and when: the created date of + * the record that did it. `name` writes the id of that record. + */ + const verdict = ( + record: ReadRecord, + name: (id: string) => string = (id) => id + ): string => { + const standing = standingOf(record); + if (standing === 'retracted') + return `retracted on ${day( + record.frontmatter.retracted_at ?? record.frontmatter.created_at + )}`; + const relation: ReadRelation | undefined = + standing === 'superseded' + ? 'superseded_by' + : standing === 'invalidated' + ? 'invalidated_by' + : standing === 'rejected' + ? 'rejected_by' + : undefined; + const by = relation ? ids(record, relation) : []; + return by.length + ? `${standing} by ${by.map(name).join(', ')} on ${date(latest(by))}` + : standing; + }; + /** + * When a record last changed, as far as its links tell: the newest of + * its own date, its reopenings, and the records that resolved, replaced, + * disproved, rejected, or mitigated it. + */ + const changedAt = (record: ReadRecord): number => + Math.max( + time(record), + ...(Array.isArray(record.frontmatter.reopen_history) + ? (record.frontmatter.reopen_history as Record[]) + : [] + ).map((entry) => { + const value = Date.parse(text(entry?.at)); + return Number.isNaN(value) ? 0 : value; + }), + ...( + [ + 'resolved_by', + 'superseded_by', + 'invalidated_by', + 'rejected_by', + 'mitigated_by', + ] as const + ).flatMap((relation) => records(ids(record, relation)).map(time)) + ); + const reopenings = ( + record: ReadRecord, + name: (id: string) => string = (id) => id + ): string[] => + (Array.isArray(record.frontmatter.reopen_history) + ? (record.frontmatter.reopen_history as Record[]) + : [] + ).map( + (entry) => + `reopened on ${day(entry?.at)}${ + typeof entry?.rejected_by === 'string' + ? ` after rejection by ${name(entry.rejected_by)}` + : '' + }` + ); + + const sections: StateSection[] = []; + const lines: string[] = []; + /** Records some list above already shows, so Lifecycle skips them. */ + const shown = new Set(); + /** + * Adds a section, showing at most `limit` of its entries. Each entry + * stands for one record unless it gives `records` (all it stands for) + * and `listed` (those it names); `extra` counts records the section + * leaves out of its entries altogether. + */ + const section = ( + heading: string, + intro: readonly string[], + entries: readonly { + ids: readonly string[]; + lines: readonly string[]; + records?: number; + listed?: number; + }[], + limit: number, + rest: string, + { tail = [], extra = 0 }: { tail?: readonly string[]; extra?: number } = {} + ) => { + const kept = entries.slice(0, limit); + const sum = (list: typeof entries, field: 'records' | 'listed') => + list.reduce((n, entry) => n + (entry[field] ?? 1), 0); + sections.push({ + heading, + total: sum(entries, 'records') + extra, + shown: sum(kept, 'listed'), + }); + lines.push('', `### ${heading}`, ...intro); + if (!entries.length) lines.push('- none'); + for (const entry of kept) { + lines.push(...entry.lines); + for (const id of entry.ids) shown.add(id); + } + if (entries.length > kept.length) { + capped = true; + lines.push(`- … ${entries.length - kept.length} more ${rest}`); + } + lines.push(...tail); + }; + + // Counts: every record by kind and standing; Issues, Findings, and + // Constraints also by their own kind field. + const countLines: string[] = []; + let live = 0; + let retracted = 0; + for (const kind of KIND_ORDER) { + const members = all.filter((record) => record.kind === kind); + if (!members.length) continue; + const standings = new Map(); + const bySub = new Map>(); + for (const record of members) { + const standing = standingOf(record); + bump(standings, standing); + if (isLive(record)) live++; + if (standing === 'retracted') retracted++; + const sub = subKind(record); + if (sub && kind !== 'blob') { + if (!bySub.has(sub)) bySub.set(sub, new Map()); + bump(bySub.get(sub) as Map, standing); + } + } + countLines.push( + `- ${KIND_LABEL[kind]} ${members.length}: ${tallyLine(standings)}` + ); + if (bySub.size > 1 || (bySub.size === 1 && kind === 'issue')) + for (const [sub, tally] of [...bySub].sort(([a], [b]) => + compareIds(a, b) + )) + countLines.push( + ` - ${sub} ${[...tally.values()].reduce( + (a, b) => a + b, + 0 + )}: ${tallyLine(tally)}` + ); + } + const reopened = all.filter((record) => reopenings(record).length); + if (reopened.length) { + const now = new Map(); + for (const record of reopened) bump(now, standingOf(record)); + countLines.push( + `- Decisions ever reopened: ${reopened.length}; now ${tallyLine(now)}` + ); + } + const neverLive = all.filter( + (record) => record.kind === 'citation' || record.kind === 'blob' + ).length; + countLines.push( + `- Live records: ${live} of ${all.length}${ + neverLive + ? `; Citations and Blobs (${neverLive}) are never counted live` + : '' + }; retracted: ${retracted}` + ); + lines.push( + `## State of ${effortId}`, + `"${cut(text(effort?.frontmatter.title), 120)}", status ${text( + effort?.frontmatter.status + )}, created ${date(effort)}, newest record ${date(all.at(-1))}.`, + 'Built from frontmatter and links only. Each record shows its standing in parentheses; a change is dated by the record that made it.', + '', + '### Counts', + ...countLines + ); + sections.push({ + heading: 'Counts', + total: countLines.length, + shown: countLines.length, + }); + + // Open Issues, blockers first, each with what it gates and comes from. + const openIssues = all + .filter( + (record) => record.kind === 'issue' && standingOf(record) === 'open' + ) + .sort( + (a, b) => + Number(subKind(b) === 'blocker') - Number(subKind(a) === 'blocker') || + byCreated(a, b) + ); + section( + 'Open Issues and blockers', + [ + `${count( + openIssues.filter((issue) => subKind(issue) === 'blocker').length, + 'open blocker' + )}, ${count( + openIssues.length, + 'open Issue' + )} in all. "Gates" lists the live Decisions that derive from an Issue.`, + ], + openIssues.map((issue) => { + const after = inbound(issue.id, 'derived_by').filter(isCurrent); + const gates = after.filter((record) => record.kind === 'decision'); + const others = after.filter((record) => record.kind !== 'decision'); + const from = ids(issue, 'derives_from'); + return { + ids: [issue.id, ...gates.map((record) => record.id)], + lines: [ + `- ${ref(issue.id)}, ${subKind(issue) || 'issue'}, open since ${date( + issue + )}`, + ...(gates.length + ? [` - gates: ${refList(gates.map((record) => record.id))}`] + : []), + ...(others.length + ? [` - built on by: ${refList(others.map((record) => record.id))}`] + : []), + ...(from.length ? [` - comes from: ${refList(from)}`] : []), + ], + }; + }), + limits.openIssues, + `open Issues; \`proof records ${effortId} --kinds issue --status open\`` + ); + + // Risks: realized and open first, with where they came from; then + // mitigated and accepted ones, one line each. + const riskRank = (risk: ReadRecord) => { + const rank = RISK_ORDER.indexOf(standingOf(risk)); + return rank < 0 ? RISK_ORDER.length : rank; + }; + const risks = all + .filter( + (record) => record.kind === 'risk' && standingOf(record) !== 'retracted' + ) + .sort((a, b) => riskRank(a) - riskRank(b) || byCreated(a, b)); + section( + 'Risks', + [], + risks.map((risk) => { + const standing = standingOf(risk); + const mitigations = ids(risk, 'mitigated_by'); + const evidence = ids(risk, 'evidence'); + const from = ids(risk, 'derives_from'); + // A replaced or disproved Risk is dated by the record that replaced + // or disproved it; a realized or accepted one by its evidence, which + // the state change names. + const moved = + standing === 'superseded' || standing === 'invalidated' + ? `; ${verdict(risk)}` + : standing === 'mitigated' && mitigations.length + ? `; mitigated on ${date(latest(mitigations))} by ${refList( + mitigations + )}` + : evidence.length && standing !== 'open' + ? `; ${standing} on ${date(latest(evidence))}, evidence ${refList( + evidence + )}` + : ''; + const first = `- ${ref(risk.id)}, raised ${date(risk)}${moved}`; + // A Risk still in play also says where it came from and whether + // anything mitigates it; a settled one needs only its first line. + if (standing !== 'open' && standing !== 'realized') + return { ids: [risk.id], lines: [first] }; + const details = [ + `mitigation: ${mitigations.length ? refList(mitigations) : 'none'}`, + ...(from.length ? [`comes from: ${refList(from)}`] : []), + ]; + return { + ids: [risk.id], + lines: [first, ` - ${details.join('; ')}`], + }; + }), + limits.risks, + `Risks; \`proof records ${effortId} --kinds risk\`` + ); + + // Constraints in force: the live ones, by their kind. + const constraints = all.filter( + (record) => record.kind === 'constraint' && isLive(record) + ); + const constraintKinds = [ + ...new Set(constraints.map((record) => subKind(record) || 'constraint')), + ].sort(); + section( + 'Constraints in force', + [], + constraintKinds.map((kind) => { + const members = constraints.filter( + (record) => (subKind(record) || 'constraint') === kind + ); + const kept = members.slice(0, limits.constraints); + return { + ids: kept.map((record) => record.id), + records: members.length, + listed: kept.length, + lines: [ + `- ${kind} ${members.length}: ${kept + .map((record) => record.id) + .join(', ')}${more(members.length, kept.length)}`, + ], + }; + }), + constraintKinds.length, + 'kinds' + ); + + // Contested topics: a question Issue with more than one Decision, counting + // every version of each, or a supersession chain with more than one version. + // A retracted record is noise, not a version, so topics leave it out; + // Retractions lists it. + const chainOf = (record: ReadRecord) => + (index.chains.get(index.chainOf.get(record.id) ?? '') ?? [record]).filter( + (member) => standingOf(member) !== 'retracted' + ); + const inTopic = new Set(); + const topics: { + first: ReadRecord; + question?: ReadRecord; + members: ReadRecord[]; + }[] = []; + for (const question of all) { + if ( + question.kind !== 'issue' || + subKind(question) !== 'question' || + standingOf(question) === 'retracted' + ) + continue; + const members = new Map(); + for (const decision of inbound(question.id, 'derived_by')) + if ( + decision.kind === 'decision' && + decision.frontmatter.effort === effortId && + standingOf(decision) !== 'retracted' + ) + for (const member of chainOf(decision)) members.set(member.id, member); + if (members.size < 2) continue; + for (const id of members.keys()) inTopic.add(id); + topics.push({ + first: question, + question, + members: [...members.values()].sort(byCreated), + }); + } + for (const record of all) { + const chain = chainOf(record); + // Compactions replace each other by design; `proof brief` reads them. + if ( + chain.length < 2 || + chain.some((member) => inTopic.has(member.id)) || + isCompaction(standingRecord(record)) + ) + continue; + for (const member of chain) inTopic.add(member.id); + topics.push({ first: chain[0], members: [...chain] }); + } + // Open contests first (more than one live record), then by the newest + // change, so a cut list drops old, settled topics. + const newest = (members: readonly ReadRecord[]) => + Math.max(...members.map((member) => time(member))); + topics.sort( + (a, b) => + Number(b.members.filter(isLive).length > 1) - + Number(a.members.filter(isLive).length > 1) || + newest(b.members) - newest(a.members) || + byCreated(a.first, b.first) + ); + section( + 'Contested topics', + [ + `${count( + topics.length, + 'topic' + )}: questions with more than one Decision (every version counted), and records with more than one version. Open contests come first, then the most recently changed. Letters name the records shown; "now" marks the live ones.`, + ], + topics.map(({ question, members }) => { + // Live records first, then dead ones, each newest first, so a cut + // drops the oldest versions. + const order = [ + ...members.filter(isLive).reverse(), + ...members.filter((member) => !isLive(member)).reverse(), + ]; + const kept = order.slice(0, limits.membersPerTopic); + const left = order.slice(kept.length); + const leftLive = left.filter(isLive).length; + if (left.length) capped = true; + // Letters stand for the records shown; a record left out keeps its id. + const tag = new Map( + kept.map((member, i) => [member.id, `[${String.fromCharCode(97 + i)}]`]) + ); + const name = (id: string) => tag.get(id) ?? id; + const resolvers = question ? ids(question, 'resolved_by') : []; + const head = question + ? `- ${ref(question.id)}, question with ${members.length} Decisions${ + resolvers.length + ? `; resolved by ${resolvers.map(name).join(', ')}` + : '' + }` + : `- ${members.length} versions of one ${members[0].kind}${ + subKind(members[0]) ? ` (${subKind(members[0])})` : '' + }`; + return { + ids: [...(question ? [question.id] : []), ...kept.map((m) => m.id)], + lines: [ + head, + ...kept.map((member) => { + const notes = [ + isLive(member) + ? `now ${standingOf(member)}, written ${date(member)}` + : verdict(member, name), + ...reopenings(member, name), + ]; + return ` - ${name(member.id)} ${member.id}: ${notes.join('; ')}`; + }), + ...(left.length + ? [ + ` - … ${count(left.length, 'more version')}${ + leftLive ? `, ${leftLive} of them live` : '' + }`, + ] + : []), + ], + }; + }), + limits.contested, + 'topics' + ); + + // Issues closed without a fix: deferred or won't fix, or resolved with no + // resolving Decision or Finding. + const closed = all.filter((record) => { + if (record.kind !== 'issue') return false; + const standing = standingOf(record); + if (standing === 'deferred' || standing === 'wontfix') return true; + return ( + standing === 'resolved' && + !ids(record, 'resolved_by').some((id) => + ['decision', 'finding'].includes(get(id)?.kind ?? '') + ) + ); + }); + section( + 'Closed without a fix', + [ + 'Issues closed as deferred or wontfix, or resolved by no Decision or Finding.', + ], + closed.map((issue) => { + const by = ids(issue, 'resolved_by'); + return { + ids: [issue.id], + lines: [ + `- ${ref(issue.id)}, ${subKind(issue) || 'issue'}, opened ${date( + issue + )}; ${ + by.length + ? `closed ${standingOf(issue)} on ${date( + latest(by) + )} by ${refList(by)}` + : 'closed with no resolving record' + }`, + ], + }; + }), + limits.closedWithoutFix, + `Issues; \`proof records ${effortId} --kinds issue --status deferred,wontfix\`` + ); + + // Dead evidence: live records that derive from, cite, or give as evidence + // a superseded, invalidated, rejected, or retracted record. One entry per + // dead record, so the record that overturned it is named once. + const rank = (standing: string) => DEAD_ORDER.indexOf(standing); + const resting = new Map>(); + for (const record of all) { + if (!isLive(record) || isCompaction(standingRecord(record))) continue; + for (const relation of SUPPORT_RELATIONS) + for (const target of records(ids(record, relation))) { + // The record that overturned a support, by replacing, disproving, + // or rejecting it, does not rest on it: it derives from what it + // overturned. + const overturned = ( + ['superseded_by', 'invalidated_by', 'rejected_by'] as const + ).some((by) => ids(target, by).includes(record.id)); + if (!DEAD.has(standingOf(target)) || overturned) continue; + if (!resting.has(target.id)) resting.set(target.id, new Set()); + (resting.get(target.id) as Set).add(record.id); + } + } + const deadSupports = records([...resting.keys()]).sort( + (a, b) => rank(standingOf(a)) - rank(standingOf(b)) || byCreated(a, b) + ); + // Entries whose dead record was only replaced come last, and a large + // Effort shows fewer of them. + const replaced = deadSupports.filter( + (support) => standingOf(support) === 'superseded' + ); + const replacedLeft = replaced.slice(limits.supersededSupports); + const deadShown = deadSupports.filter( + (support) => !replacedLeft.includes(support) + ); + const restingRecords = new Set( + [...resting.values()].flatMap((set) => [...set]) + ); + const worst = new Map(); + for (const support of deadSupports) bump(worst, standingOf(support)); + const compaction: string[] = []; + const standingRecords = all.map(standingRecord); + const head = records( + compactionHeads(standingRecords).map((record) => record.id) + ).at(-1); + if (head) { + const covers = parseCovers(head.frontmatter.covers); + const changes = compactionChanges(covers, standingRecords); + const stale = staleDerivations( + covers, + ids(head, 'derives_from'), + standingRecords + ); + const listed = stale.slice(0, limits.compactionTargets); + compaction.push( + `- Head compaction ${head.id} (${date( + head + )}): since it was written, ${count( + changes.length, + 'record' + )} differ from the live set it covers (new, no longer live, or with a new standing). ${ + stale.length + } of the records it rests on changed standing${ + stale.length + ? `: ${listed + .map( + (change) => + `${change.id} (${change.from ?? 'none'} -> ${ + change.to ?? 'missing' + })` + ) + .join(', ')}${more(stale.length, listed.length)}` + : '' + }.` + ); + } + section( + 'Dead evidence', + [ + `${count(restingRecords.size, 'live record')} ${ + restingRecords.size === 1 ? 'rests' : 'rest' + } directly on ${count(deadSupports.length, 'dead record')}${ + deadSupports.length ? ` (${tallyLine(worst)})` : '' + }. Resting on a record means deriving from it, citing it, or giving it as evidence.`, + ...compaction, + ], + deadShown.map((support) => { + const live = records([...(resting.get(support.id) ?? [])]); + // A replaced version that Contested topics lists gets only its date. + const why = + standingOf(support) === 'superseded' && shown.has(support.id) + ? `superseded on ${date( + latest(ids(support, 'superseded_by')) + )} (see Contested topics)` + : verdict(support); + // Records built on the live ones, one hop further, for every dead + // record but a replaced version. + const further = + standingOf(support) === 'superseded' + ? [] + : records([ + ...new Set( + live.flatMap((record) => + inbound(record.id, 'derived_by') + .filter( + (next) => + isCurrent(next) && + !resting.get(support.id)?.has(next.id) && + !( + [ + 'superseded_by', + 'invalidated_by', + 'rejected_by', + ] as const + ).some((by) => ids(support, by).includes(next.id)) + ) + .map((next) => next.id) + ) + ), + ]); + return { + ids: [support.id, ...live.map((record) => record.id)], + lines: [ + `- ${support.id}, ${why}; rested on by ${refList( + live.map((record) => record.id) + )}`, + ...(further.length + ? [ + ` - and, one link further, by ${refList( + further.map((record) => record.id) + )}`, + ] + : []), + ], + }; + }), + limits.deadEvidence, + 'dead records with live records on them', + { + extra: replacedLeft.length, + tail: replacedLeft.length + ? [ + `- … ${ + replacedLeft.length + } more replaced versions that live records still rest on: ${ + records( + replacedLeft.flatMap((support) => [ + ...(resting.get(support.id) ?? []), + ]) + ).length + } live records`, + ] + : [], + } + ); + if (replacedLeft.length) capped = true; + + // Lifecycle: the Effort, then dated changes to Issues, Decisions, and + // records shown wrong that no list above shows. A question resolved by a + // live Decision and an accepted Decision never reopened are settled and + // show only in the counts. + const lifecycle = all.flatMap((record) => { + if (shown.has(record.id) || inTopic.has(record.id)) return []; + const standing = standingOf(record); + const at = changedAt(record); + if (standing === 'retracted') return []; + if (record.kind === 'issue') { + const by = ids(record, 'resolved_by'); + if ( + subKind(record) === 'question' && + standing === 'resolved' && + records(by).some( + (target) => target.kind === 'decision' && isLive(target) + ) + ) + return []; + return [ + { + at, + ids: [record.id], + lines: [ + `- ${ref(record.id)}, ${subKind(record) || 'issue'}: opened ${date( + record + )}${ + by.length + ? `; ${standing} on ${date(latest(by))} by ${refList(by)}` + : '' + }`, + ], + }, + ]; + } + const notes = + record.kind === 'decision' ? reopenings(record) : ([] as string[]); + if (record.kind === 'decision' && standing === 'accepted' && !notes.length) + return []; + if ( + record.kind !== 'decision' && + standing !== 'invalidated' && + standing !== 'superseded' + ) + return []; + return [ + { + at, + ids: [record.id], + lines: [ + `- ${ref(record.id)}: written ${date(record)}${[ + ...notes, + ...(isLive(record) ? [] : [verdict(record)]), + ] + .map((note) => `; ${note}`) + .join('')}`, + ], + }, + ]; + }); + section( + 'Lifecycle', + [ + 'Issues, Decisions, and records shown wrong that no list above shows, the most recently changed first. The Effort itself is dated in the first line.', + ], + lifecycle.sort((a, b) => b.at - a.at || compareIds(a.ids[0], b.ids[0])), + limits.lifecycle, + 'older entries' + ); + + const retractions = all.filter( + (record) => standingOf(record) === 'retracted' + ); + section( + 'Retractions', + [`${count(retractions.length, 'retracted record')}, none of them live.`], + retractions.map((record) => ({ + ids: [record.id], + lines: [ + `- ${record.id}, ${record.kind}${ + subKind(record) ? ` (${subKind(record)})` : '' + }, written ${date(record)}, retracted on ${day( + record.frontmatter.retracted_at + )}${ + text(record.frontmatter.retracted_reason) + ? `: ${cut( + text(record.frontmatter.retracted_reason), + REASON_CHARS + )}` + : '' + }`, + ], + })), + limits.retractions, + `retracted records; \`proof records ${effortId} --retracted only\`` + ); + + return { + effort_id: effortId, + records: all.length, + live, + sections, + capped, + markdown: lines.join('\n'), + }; +} diff --git a/packages/proof/skills/proof/SKILL.md b/packages/proof/skills/proof/SKILL.md index a3381bdb..18427ce0 100644 --- a/packages/proof/skills/proof/SKILL.md +++ b/packages/proof/skills/proof/SKILL.md @@ -183,9 +183,17 @@ Pick the read by what you need, in this order: terms name query words no hit contains. Ordinary words often land there; a missing distinctive term, such as a name, a number, or a code reference, is the cue that the answer may not be recorded. -2. **Read one record in full: `proof get `.** -3. **Count or page through records: `proof records`.** `find` shows only - the top hits, so never count from it. +2. **Counts, lists, and status history: `proof state `.** One + read, built from frontmatter and links, that shows exact counts by kind + and standing; open blockers and what they gate; Risks with their + mitigation; contested topics, with each rejected or replaced alternative, + what rejected or replaced it, and when; Issues closed without a fix; + live records that rest on superseded, disproved, rejected, or retracted + ones; dated lifecycles; and retractions. It holds no facts from record + bodies: use `find` for those. +3. **Read one record in full: `proof get `.** +4. **Page through records: `proof records`.** `find` shows only the top + hits, so never count from it; count from `state`. Browse digests (`list`, `records`, `relations`, `blocking-decisions`) excerpt each body at 600 chars / 12 lines (`[…truncated]`). **`proof get` digests @@ -198,6 +206,9 @@ byte cap). Zoom in with `get`, then Read/grep that digest — do not open flatbread proof find "tern history buffer precision" --effort flatbread proof find "shadow pool overflow" --as-of 2026-03-01T00:00:00Z +# Counts, open blockers, Risks, contested topics, dead evidence, lifecycles +flatbread proof state + # What's gating this effort? (proposed Decisions deriving from open blocker Issues) flatbread proof blocking-decisions @@ -255,7 +266,9 @@ server-side. `dec-adopt-a-bounded-status-briefing-fast-path-for-ef--kcw0rw39g3b2ym2h`). 2. **When you need a particular record or fact:** run `proof find` first, then `proof get` the hit whose card fits. Do not page through `records` - to look for it. + to look for it. **When you need a count, a list of open, closed, or + rejected records, a status history, the blockers, or what rests on dead + evidence:** run `proof state ` instead. 3. **When a browse digest shows `[…truncated]` and you need the body:** run `flatbread proof get `, then Read/grep that digest (`artifact_path`) for the full body. Reserve opening `.flatbread-proof/**/*.md` for rare diff --git a/packages/proof/skills/proof/reference.md b/packages/proof/skills/proof/reference.md index b8719a47..b5412c38 100644 --- a/packages/proof/skills/proof/reference.md +++ b/packages/proof/skills/proof/reference.md @@ -182,7 +182,7 @@ survivor (score 4/4 if it adds claims) plus `Retract` on the rest. `generation` is a durable, monotonic journal token — the input to strict reads. -## The 7 read queries +## The 8 read queries All reads execute through Flatbread's query engine (in-process GraphQL over the generated schema) and return a `ReadEnvelope`: @@ -211,6 +211,8 @@ lists), one-hop related records, and an edge table. Body policy: record appears as one line. - **`proof find`:** the hit cards (below) under `## Find hits`, and each hit's record with the browse excerpt. The envelope also carries the cards inline. +- **`proof state`:** no bodies at all. The state (below) under + `## Effort state`; the envelope also carries it inline as `markdown`. - **`list` / `records` / `relations` / `blocking-decisions`:** body excerpt capped at 600 chars / 12 lines (`[…truncated]`). **Blob bodies are omitted** from these digests — use `proof get` for the payload. Citation bodies @@ -233,8 +235,9 @@ order: error; do not retry the same page. 3. If `cap_reasons` is not empty, narrow the query or fail closed. It can hold several sorted, duplicate-free values from `primary_records`, - `displayed_edges`, `bytes`, and `find_limit` (`proof find` matched more - records than it shows). + `displayed_edges`, `bytes`, `find_limit` (`proof find` matched more + records than it shows), and `state_limit` (`proof state` cut at least one + list short; each cut list says how many it left out). 4. A hard cap alone leaves `page.has_more: false` and `next_cursor: null`. Paging and hard caps can occur together, but caps never create a cursor. @@ -251,13 +254,14 @@ flatbread proof relations --relations r1,r2 [--limit n] [--c flatbread proof blocking-decisions [consistency flags] flatbread proof brief [consistency flags] flatbread proof find "" [--effort | --all] [--kinds k1,k2] [--as-of iso] [--retracted exclude|include|only] [--limit n] [consistency flags] +flatbread proof state [consistency flags] flatbread proof cache prune flatbread proof serve [--socket .flatbread/proof/serve.sock] ``` -- `proof serve` keeps the graph loaded in one process and answers the seven +- `proof serve` keeps the graph loaded in one process and answers the eight reads above (`list`, `records`, `get`, `relations`, `blocking-decisions`, - `brief`, `find`); it keeps the `find` index with the graph and serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its + `brief`, `find`, `state`); it keeps the `find` index with the graph and serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its socket and those read commands go to it, in any folder whose graph root resolves to the same files. Each later read then takes a fraction of a second instead of rebuilding the graph. A write bumps the journal @@ -396,7 +400,64 @@ Each card has these lines: The card header reads `find "" in ( records searched, matched, shown)`, then `matched: … | missing: …`. The envelope adds `cards` (that Markdown) and `find`: `{"query","effort_id","searched","matched","shown","terms":{"matched","missing"},"efforts"?,"hits":[{"id","matched_id"?,"score"}]}`. -`find` shows only the top hits; use `records` to count. +`find` shows only the top hits; use `state` or `records` to count. + +### `proof state` + +`state` shows where one Effort stands, in one read of at most about 4,000 +tokens. It is built from frontmatter and links only, never from body text, +so it holds no facts or measurements: use `find` for those. It takes no +query and no `--as-of` (it refuses one); its lines date each change, and +`find --as-of` shows the version current at a date. + +The envelope adds `markdown` (the state, read it inline) and `state`: +`{"effort_id","records","live","sections":[{"heading","total","shown"}],"capped"}`. +The Markdown has these sections, in order: + +- **Counts:** every record in the Effort by kind and standing, with Issues, + Findings, and Constraints also by their own `kind` (`blocker`, `question`, + `hard`, …), how many Decisions were ever reopened, and how many records + are live or retracted. Exact numbers, never cut. +- **Open Issues and blockers:** each open Issue, blockers first, with the + live Decisions that derive from it (`gates`), other live records built on + it, and what it comes from. +- **Risks:** realized and open Risks with their evidence, mitigation, and + source; then mitigated and accepted ones, one line each. +- **Constraints in force:** the live Constraints, by kind. +- **Contested topics:** one entry per question Issue with more than one + Decision (every version counted, retracted ones left out), and per record + with more than one version. Letters (`[a]`, `[b]`) name the records + shown: the live ones (`now`), then each replaced, rejected, or disproved + one with what did it and when, and any reopening, each group newest + first. Open contests come first, then the most recently changed. +- **Closed without a fix:** Issues closed `deferred` or `wontfix`, or + resolved by no Decision or Finding. +- **Dead evidence:** each superseded, invalidated, rejected, or retracted + record that live records still derive from, cite, or give as evidence, + with those live records and, for every dead record but a replaced + version, the live records one link further. The record that overturned it is not counted as + resting on it. The first line reports the head compaction: how many + records changed since it was written, and which of its supports moved. +- **Lifecycle:** dated changes no section above shows, the most recently + changed first: + closed Issues other than settled questions, proposed or reopened + Decisions, and records shown wrong or replaced. +- **Retractions:** each retracted record with its date and reason. None is + live. + +A date on a change is the `created_at` of the record that made it, since +standings carry no dates of their own. When that record already existed, +as when a Supersede links two old records, the change came on or after +that date. Records with nothing to report, such +as a question Issue resolved by its one accepted Decision, appear only in +the counts. Citations and Blobs have no live or dead standing, so the live +total leaves them out. On a large Effort the lists shrink to fit, cutting +lifecycle and retractions first and counts never. Any cut list, a whole +section's or one inside an entry, ends with `… N more` or `and N more`, +and then `cap_reasons` holds `state_limit`. In `state.sections`, `total` +counts a section's entries in full and `shown` those it prints; for +Constraints in force and Dead evidence an entry is one Constraint or one +dead record. ### Consistency flags diff --git a/packages/proof/src/digest.ts b/packages/proof/src/digest.ts index e7cd0528..1269c59b 100644 --- a/packages/proof/src/digest.ts +++ b/packages/proof/src/digest.ts @@ -34,12 +34,15 @@ export interface ReadEdge { /** * Why a read is not complete. `find_limit` means `proof find` matched more * records than it shows; it pages no further, so narrow the query instead. + * `state_limit` means `proof state` cut at least one list short; each cut + * list says how many entries it left out. */ export type ReadCapReason = | 'primary_records' | 'displayed_edges' | 'bytes' - | 'find_limit'; + | 'find_limit' + | 'state_limit'; /** Machine-readable state of an Effort's head compaction (`brief` only). */ export interface CompactionStatus {