diff --git a/.agents/skills/proof/SKILL.md b/.agents/skills/proof/SKILL.md index f4d75e3c..65b8e104 100644 --- a/.agents/skills/proof/SKILL.md +++ b/.agents/skills/proof/SKILL.md @@ -11,7 +11,7 @@ repository. It has eight record types: **Effort**, **Issue**, **Finding**, **Citation** stores a source or reference; and **Blob** stores attached content such as a document, JSON, or image. Every record belongs to one Effort. Create and update records through 18 typed mutations, and read them -through 6 bounded queries. Do not hand-edit record frontmatter, although you +through 7 bounded queries. Do not hand-edit record frontmatter, although you may edit record bodies freely. Read [glossary.md](./glossary.md) for the primitive and edge semantics before @@ -169,6 +169,23 @@ an `artifact_path` to a rendered markdown digest (the evidence — spend one Read on it, or grep it), `served_generation`, page info, and ≤10 executable `hints`. Digests cap at 25 records / one-hop expansion / 50 edges / 64 KiB. +Pick the read by what you need, in this order: + +1. **Locate a record by what it says: `proof find ""`.** It ranks + records by their title, full body, and Blob payload, and prints the hit + cards inline in the JSON, so you need no second Read. Each card shows the + newest version of the record (or, with `--as-of`, the version current + then), its standing, the passage that matched, its supersession chain + with dates, and its links in both directions. Leave out `--effort` when + you do not know the Effort: hits then group by Effort, at most 3 each + unless one Effort clearly leads. The `missing` terms name query words no + hit contains. Ordinary words often land there; a missing distinctive + term, such as a name, a number, or a code reference, is the cue that the + answer may not be recorded. +2. **Read one record in full: `proof get `.** +3. **Count or page through records: `proof records`.** `find` shows only + the top hits, so never count from it. + Browse digests (`list`, `records`, `relations`, `blocking-decisions`) excerpt each body at 600 chars / 12 lines (`[…truncated]`). **`proof get` digests always include the full record body** (still subject to the 64 KiB digest @@ -176,6 +193,10 @@ byte cap). Zoom in with `get`, then Read/grep that digest — do not open `.flatbread-proof/**/*.md` for normal full-body recall. ```bash +# Find records by what they say (one Effort, or all when --effort is left out) +flatbread proof find "tern history buffer precision" --effort +flatbread proof find "shadow pool overflow" --as-of 2026-03-01T00:00:00Z + # What's gating this effort? (proposed Decisions deriving from open blocker Issues) flatbread proof blocking-decisions @@ -201,7 +222,8 @@ flatbread proof brief Flags shared by reads: `--strict-min-generation ` (with optional `--timeout-ms `, default 3000) and, on `list`/`records`/`relations`, `--limit` (≤25) and `--cursor` (opaque `next_cursor` from a prior page; only valid for -the same query at the same generation). +the same query at the same generation). `find` takes `--limit` 1 to 10 +(default 5) and has no cursor. `proof list` is bounded Effort discovery. It defaults to `active`; valid statuses are exactly `active`, `paused`, `completed`, and `abandoned`. @@ -230,12 +252,15 @@ server-side. families showed this roughly halves recall tool calls with no loss of answer quality (Decision `dec-adopt-a-bounded-status-briefing-fast-path-for-ef--kcw0rw39g3b2ym2h`). -2. **When a browse digest shows `[…truncated]` and you need the body:** run +2. **When you need a particular record or fact:** run `proof find` first, + then `proof get` the hit whose card fits. Do not page through `records` + to look for it. +3. **When a browse digest shows `[…truncated]` and you need the body:** run `flatbread proof get `, then Read/grep that digest (`artifact_path`) for the full body. Reserve opening `.flatbread-proof/**/*.md` for rare cases (e.g. digest byte-cap miss on an oversized record), not normal zoom-in. -3. **During work:** apply the write gate above before any create or body +4. **During work:** apply the write gate above before any create or body edit that adds claims. When outside material supports a 4/4 record, save large content with `WriteBlob` if needed, then create a `WriteCitation`, then create the Issue, Finding, Decision, Constraint, or Risk with @@ -243,10 +268,10 @@ server-side. Citation first. Open Issues for real gaps or blockers, and use `derives_from` on Decisions to link the Findings, Constraints, and Issues they respond to. -4. **On commitment:** preview `AcceptDecision` with `dryRun: true`, inspect +5. **On commitment:** preview `AcceptDecision` with `dryRun: true`, inspect its `rejectedIds`, then commit the acceptance. Use `ResolveIssue` with `resolvedBy` citing the closing Decision/Findings. These lifecycle transitions do not need a 4/4 score. Retract session noise with `Retract` rather than deleting files. -5. Maintenance: `flatbread proof cache prune` deletes digests older than +6. Maintenance: `flatbread proof cache prune` deletes digests older than 24h / over the 100 MiB ceiling. Prune does not need a 4/4 score. diff --git a/.agents/skills/proof/reference.md b/.agents/skills/proof/reference.md index 6adf5d9d..b484f68c 100644 --- a/.agents/skills/proof/reference.md +++ b/.agents/skills/proof/reference.md @@ -182,7 +182,7 @@ survivor (score 4/4 if it adds claims) plus `Retract` on the rest. `generation` is a durable, monotonic journal token — the input to strict reads. -## The 6 read queries +## The 7 read queries All reads execute through Flatbread's query engine (in-process GraphQL over the generated schema) and return a `ReadEnvelope`: @@ -209,6 +209,8 @@ lists), one-hop related records, and an edge table. Body policy: Blob payloads and Citation bodies (URLs / short notes). - **`proof brief`:** full body of the head compaction only. Every other record appears as one line. +- **`proof find`:** the hit cards (below) under `## Find hits`, and each hit's + record with the browse excerpt. The envelope also carries the cards inline. - **`list` / `records` / `relations` / `blocking-decisions`:** body excerpt capped at 600 chars / 12 lines (`[…truncated]`). **Blob bodies are omitted** from these digests — use `proof get` for the payload. Citation bodies @@ -231,7 +233,8 @@ order: error; do not retry the same page. 3. If `cap_reasons` is not empty, narrow the query or fail closed. It can hold several sorted, duplicate-free values from `primary_records`, - `displayed_edges`, and `bytes`. + `displayed_edges`, `bytes`, and `find_limit` (`proof find` matched more + records than it shows). 4. A hard cap alone leaves `page.has_more: false` and `next_cursor: null`. Paging and hard caps can occur together, but caps never create a cursor. @@ -247,13 +250,14 @@ flatbread proof records [--kinds k1,k2] [--state s1,s2] [--status s1, flatbread proof relations --relations r1,r2 [--limit n] [--cursor c] [consistency flags] flatbread proof blocking-decisions [consistency flags] flatbread proof brief [consistency flags] +flatbread proof find "" [--effort | --all] [--kinds k1,k2] [--as-of iso] [--retracted exclude|include|only] [--limit n] [consistency flags] flatbread proof cache prune flatbread proof serve [--socket .flatbread/proof/serve.sock] ``` -- `proof serve` keeps the graph loaded in one process and answers the six +- `proof serve` keeps the graph loaded in one process and answers the seven reads above (`list`, `records`, `get`, `relations`, `blocking-decisions`, - `brief`); it serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its + `brief`, `find`); it keeps the `find` index with the graph and serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its socket and those read commands go to it, in any folder whose graph root resolves to the same files. Each later read then takes a fraction of a second instead of rebuilding the graph. A write bumps the journal @@ -305,6 +309,72 @@ flatbread proof serve [--socket .flatbread/proof/serve.sock] stale head, or at least 5 changes. Programs read these fields, not the digest. +### `proof find` + +`find` locates records by what they say. Quoted or not, every word after +`find` is part of the query, locally and through `proof serve`. + +- **Scope:** `--effort ` searches one Effort. Without it, or with + `--all`, it searches every Effort. `--kinds` narrows to some of `issue`, + `finding`, `decision`, `constraint`, `risk`, `citation`, `blob` (default: + all seven; Efforts are never hits). `--retracted` works as on `records`: + retracted records are left out unless you pass `include` or `only`, and a + shown one is marked `RETRACTED`. Every version a card scores, quotes, or + shows passes this scope; other versions appear only on the chain line. +- **Ranking:** BM25 keyword scoring (k1 1.2, b 0.75) over the title (counted + twice), the full body, `kind`, `state`, `status`, `role`, `created_by`, and + Blob payloads. Rare words weigh more, measured over the whole graph. Words + are lowercased, apostrophes dropped, hyphenated words indexed joined and + split, and plurals, `-ed`, `-ing`, and a final `-e` folded (`added` meets + `add`, `using` meets `use`). Numbers with units + (`4.7 ms` is `4.7ms`), people (`created_by`), and code references (file + paths, `symbol()` names, commit SHAs, `#123`) count 1.5 times. Ties break by + id. The index is rebuilt from the record files; nothing derived is stored. +- **Chains:** all versions linked by `supersedes` / `superseded_by` form one + hit, scored by its best in-scope version. Versions are numbered `v1`, + `v2`, … in link order: each comes after the versions it supersedes, and + created time, then id, orders fork branches. The card shows the chain head, + or the newest in-scope version when the head is out of scope (for example, + retracted under the default `exclude`). If an older version matched, the + card says which (`matched: v1 `). +- **As of:** `--as-of ` takes `YYYY-MM-DD` (UTC midnight) or a + date-time with `Z` or an offset, such as `2026-03-01T00:00:00Z`; other + forms are refused, so the answer does not depend on the machine's time + zone. Only versions created at or before that moment are scored, quoted, + and shown. The card shows the last such version in link order, and chains + with no such version drop out. +- **Grouping:** without `--effort`, Efforts are ranked by the summed score + of their best 3 hits (`efforts by match`), and hits are grouped under each + Effort. Unless the top Effort's sum is at least 1.5 times the next one's, + at most 3 hits come from one Effort at first; slots still empty then go to + the next hits by score. +- **Caps:** `--limit` 1 to 10, default 5. Each card holds at most 4 chain + versions (the head and the shown and matched versions always appear), 3 + links per relation, and 6 code references. When five cards run past about + 4,000 characters, the cards shrink in this order: link titles, the + `efforts by match` line, the middle of each chain, then link entries. A + relation with no entry left shows its count, as `cited_by +4`. When more + records matched than are shown, the envelope has `complete: false` and + `cap_reasons: ["find_limit"]`; there is no cursor, so narrow the query or + raise `--limit`. + +Each card has these lines: + +```text +1. | [:] | [(head)] | | score + "" + matched: v<n> <id> (only when an older version matched) + "…<160-character passage with the most query words>…" [chars <from>-<to> of <length>] + chain: v1 <id> <standing> <date> -> (n more) -> v4 <id> <standing> <date> (head) + out: <relation> <id> "<title, 60 chars>" (<standing>), … +n more; <relation> +n + in: <derived_by|cited_by|resolves|rejects|mitigates|evidence_for|blob_of> <id> "<title>" (<standing>) + code: <paths, symbol() names, SHAs, #numbers the chain's records name> +``` + +The card header reads `find "<words>" in <effortId|all Efforts> (<searched> records searched, <matched> matched, <shown> shown)`, then `matched: … | missing: …`. The envelope adds `cards` (that Markdown) and `find`: +`{"query","effort_id","searched","matched","shown","terms":{"matched","missing"},"efforts"?,"hits":[{"id","matched_id"?,"score"}]}`. +`find` shows only the top hits; use `records` to count. + ### Consistency flags - `--strict-min-generation <token>`: serve at or after that journal diff --git a/packages/flatbread/bin/flatbread.js b/packages/flatbread/bin/flatbread.js index 6bdd1889..d436f3b0 100755 --- a/packages/flatbread/bin/flatbread.js +++ b/packages/flatbread/bin/flatbread.js @@ -34,6 +34,7 @@ const SERVED_READS = new Set([ 'relations', 'blocking-decisions', 'brief', + 'find', ]); /** diff --git a/packages/flatbread/src/cli/proof.ts b/packages/flatbread/src/cli/proof.ts index 0e51cfd5..78bc74ba 100644 --- a/packages/flatbread/src/cli/proof.ts +++ b/packages/flatbread/src/cli/proof.ts @@ -15,7 +15,9 @@ import { import { blockingDecisions, effortBrief, + effortFind, effortRecords, + type FindEnvelope, relations, getRecord, listEfforts, @@ -39,6 +41,9 @@ export interface EffortCliOptions { cursor?: string; relations?: string[]; retracted?: RetractedMode; + effort?: string; + all?: boolean; + asOf?: string; verify?: boolean; dryRun?: boolean; } @@ -74,6 +79,15 @@ export function mapEffortCliOptions( typeof options.retracted === 'string' ? (options.retracted as RetractedMode) : undefined, + effort: typeof options.effort === 'string' ? options.effort : undefined, + all: options.all === true || options.all === 'true' ? true : undefined, + // The option parser turns `--as-of 2026` into a number; keep it as + // text so the date check refuses it, as the read server does. + asOf: + typeof options['as-of'] === 'string' || + typeof options['as-of'] === 'number' + ? String(options['as-of']) + : undefined, verify: options.verify === true ? true : undefined, dryRun: options['dry-run'] === true || options.dryRun === true @@ -258,6 +272,33 @@ export async function handleEffortBrief( }); } +/** + * `proof find`: searches one Effort with `effort`, or every Effort when + * `effort` is left out or `all` is set. + */ +export async function handleEffortFind( + query: string, + options: EffortCliOptions = {} +): Promise<FindEnvelope> { + if (options.effort && options.all) + throw new ProofReadValidationError( + 'PROOF_INVALID_ARGUMENT', + 'Pass --effort <effortId> or --all, not both' + ); + const cwd = options.cwd ?? process.cwd(); + return effortFind(String(query ?? ''), { + cwd, + rootDir: await rootFor(cwd), + cacheRoot: resolve(cwd, '.flatbread/proof'), + consistency: consistency(options), + effort: options.effort, + kinds: options.kinds as PrimitiveKind[] | undefined, + asOf: options.asOf, + retracted: options.retracted, + limit: options.limit, + }); +} + export interface EffortBootstrapReport { readonly status: 'ready' | 'action_required'; readonly config_path: string | null; @@ -538,6 +579,35 @@ export function registerProofCommands(prog: any): void { ).action(async (effortId: string, options: Record<string, unknown>) => printResult(handleEffortRecords(effortId, mapEffortCliOptions(options))) ); + consistencyOptions( + prog.command( + 'proof find <words>', + 'Find records by what they say, ranked; one card per supersession chain' + ) + ) + .option('--effort <effortId>', 'Search one Effort') + .option( + '--all', + 'Search every Effort (the default without --effort)', + false + ) + .option('--kinds <values>', 'Comma-separated primitive kinds') + .option('--as-of <iso>', 'Show the version current at this date') + .option( + '--retracted <mode>', + 'exclude (default), include, or only retracted records' + ) + .option('--limit <n>', 'Hits to show, 1 to 10 (default 5)') + .action(async (words: unknown, options: Record<string, unknown>) => + printResult( + handleEffortFind( + // An unquoted query arrives as one word here and the rest in `_`. + // Join them, as the read server does, so every word is searched. + [words, ...(Array.isArray(options._) ? options._ : [])].join(' '), + mapEffortCliOptions(options) + ) + ) + ); paginationOptions( consistencyOptions( prog.command('proof list', 'List Efforts by lifecycle status') diff --git a/packages/flatbread/src/cli/serve.test.ts b/packages/flatbread/src/cli/serve.test.ts index ff7c7812..768b8f61 100644 --- a/packages/flatbread/src/cli/serve.test.ts +++ b/packages/flatbread/src/cli/serve.test.ts @@ -161,6 +161,126 @@ test.serial( } ); +test.serial( + 'proof find returns the chain head card locally and through proof serve', + async (t) => { + // Each local run loads the graph from scratch, and this test makes many. + t.timeout(180_000); + const cwd = await createProject(t); + const write = async (input: Record<string, unknown>) => + (await handleEffortWrite(JSON.stringify(input), { cwd })).artifacts[0].id; + const effortId = await write({ + type: 'CreateEffort', + title: 'Renderer', + body: '', + }); + await write({ + type: 'WriteFinding', + effort: effortId, + title: 'Build times', + body: 'The nightly build takes ten minutes.', + kind: 'measurement', + }); + const first = await write({ + type: 'WriteDecision', + effort: effortId, + title: 'Clamp the quokka history buffer', + body: 'Clamp history to the neighborhood to stop ghosting.', + }); + const head = await write({ + type: 'WriteDecision', + effort: effortId, + title: 'Clamp history only outside caustics', + body: 'Caustic light skips the clamp; everything else keeps it.', + supersedes: [first], + }); + const socket = join(cwd, '.flatbread/proof/serve.sock'); + const server = await serveProofReads(socket); + t.teardown(() => new Promise((done) => server.close(() => done()))); + const served = { FLATBREAD_PROOF_SERVER: socket }; + + for (const scope of [['--effort', effortId], ['--all']]) { + const args = ['proof', 'find', 'quokka history', ...scope]; + const local = await runCli(cwd, {}, ...args); + t.is(local.code, 0, local.stderr); + const remote = await runCli(cwd, served, ...args); + t.is(remote.code, 0, remote.stderr); + t.deepEqual(JSON.parse(remote.stdout), JSON.parse(local.stdout)); + const envelope = JSON.parse(local.stdout); + // One hit for the whole chain: the head, which says the old version matched. + t.deepEqual( + envelope.find.hits.map((hit: { id: string }) => hit.id), + [head] + ); + t.is(envelope.find.hits[0].matched_id, first); + t.deepEqual(envelope.find.terms, { + matched: ['quokka', 'history'], + missing: [], + }); + t.regex( + envelope.cards, + new RegExp(`1\\. ${head} \\| decision \\| proposed \\(head\\)`) + ); + t.regex(envelope.cards, new RegExp(`matched: v1 ${first}`)); + t.deepEqual(envelope.hints, [`getRecord("${head}")`]); + t.regex(await readFile(envelope.artifact_path, 'utf8'), /## Find hits\n/); + } + for (const env of [{}, served] as Record<string, string>[]) { + // An unquoted query searches every word, locally and served alike. + const unquoted = await runCli( + cwd, + env, + 'proof', + 'find', + 'quokka', + 'nightly', + '--effort', + effortId + ); + t.is(unquoted.code, 0, unquoted.stderr); + const envelope = JSON.parse(unquoted.stdout); + t.is(envelope.find.query, 'quokka nightly'); + t.deepEqual(envelope.find.terms.missing, []); + t.is(envelope.find.shown, 2); + t.true(envelope.complete); + // More matches than shown: the envelope says it is not complete. + const capped = JSON.parse( + ( + await runCli( + cwd, + env, + 'proof', + 'find', + 'quokka nightly', + '--limit', + '1' + ) + ).stdout + ); + t.false(capped.complete); + t.deepEqual(capped.cap_reasons, ['find_limit']); + t.false(capped.page.has_more); + // A bare year and a date-time with no offset are refused both ways. + for (const asOf of ['2026', '2026-03-01T00:00:00']) { + const bad = await runCli( + cwd, + env, + 'proof', + 'find', + 'x', + '--as-of', + asOf + ); + t.is(bad.code, 1, asOf); + t.is(JSON.parse(bad.stderr).error.code, 'PROOF_INVALID_ARGUMENT'); + } + const bad = await runCli(cwd, env, 'proof', 'find', 'x', '--limit', '11'); + t.is(bad.code, 1); + t.is(JSON.parse(bad.stderr).error.code, 'PROOF_INVALID_ARGUMENT'); + } + } +); + test.serial( 'reads fall back to the local CLI when the server is unreachable', async (t) => { diff --git a/packages/flatbread/src/cli/serve.ts b/packages/flatbread/src/cli/serve.ts index 7783f811..3f22924a 100644 --- a/packages/flatbread/src/cli/serve.ts +++ b/packages/flatbread/src/cli/serve.ts @@ -8,6 +8,7 @@ import { errorPayload, handleEffortBlockingDecisions, handleEffortBrief, + handleEffortFind, handleEffortGet, handleEffortList, handleEffortRecords, @@ -28,8 +29,17 @@ export const SERVED_READS: Readonly<Record<string, number>> = { relations: 2, 'blocking-decisions': 1, brief: 1, + find: 1, }; +/** Options that take no value. */ +const BOOLEAN_FLAGS = new Set(['all']); +/** + * Reads whose positionals are words of one query. They take one or more, and + * the words are joined with spaces, as the local CLI joins them. + */ +const JOINED_POSITIONALS = new Set(['find']); + /** What the server sends back: what the CLI would print, and its exit code. */ export interface ServedRead { readonly stdout: string; @@ -78,10 +88,14 @@ export function parseReadArgs(argv: readonly string[]): { continue; } const eq = arg.indexOf('='); - if (eq > 0) options[arg.slice(2, eq)] = arg.slice(eq + 1); + if (eq < 0 && BOOLEAN_FLAGS.has(arg.slice(2))) + options[arg.slice(2)] = 'true'; + else if (eq > 0) options[arg.slice(2, eq)] = arg.slice(eq + 1); else if (i + 1 < rest.length) options[arg.slice(2)] = rest[++i]; else invalid(`Option ${arg} needs a value`); } + if (JOINED_POSITIONALS.has(command) && positional.length) + return { command, positional: [positional.join(' ')], options }; if (positional.length !== SERVED_READS[command]) invalid( `proof ${command} takes ${SERVED_READS[command]} argument(s), got ${positional.length}` @@ -109,6 +123,8 @@ export async function runRead( ? await handleEffortRelations(first, second, mapped) : command === 'blocking-decisions' ? await handleEffortBlockingDecisions(first, mapped) + : command === 'find' + ? await handleEffortFind(first, mapped) : await handleEffortBrief(first, mapped); return { stdout: `${JSON.stringify(result)}\n`, stderr: '', code: 0 }; } catch (error) { diff --git a/packages/flatbread/src/proof/find.test.ts b/packages/flatbread/src/proof/find.test.ts new file mode 100644 index 00000000..e8840be9 --- /dev/null +++ b/packages/flatbread/src/proof/find.test.ts @@ -0,0 +1,510 @@ +import test from 'ava'; +import type { PrimitiveKind, ReadRecord } from '@flatbread/proof'; +import { + bestSnippet, + buildFindIndex, + codeReferences, + findRecords, + renderFindCards, + stem, + tokenize, +} from './find.js'; + +function rec( + id: string, + fields: { + title?: string; + body?: string; + effort?: string; + created_at?: string; + state?: string; + status?: string; + retracted?: boolean; + retracted_reason?: string; + created_by?: string; + relations?: ReadRecord['relations']; + } = {} +): ReadRecord { + const prefix = id.split('-', 1)[0]; + const kind = ( + { + eff: 'effort', + iss: 'issue', + fnd: 'finding', + dec: 'decision', + con: 'constraint', + rsk: 'risk', + cit: 'citation', + blb: 'blob', + } as Record<string, PrimitiveKind> + )[prefix]; + const { body, relations, ...frontmatter } = fields; + return { + id, + kind, + path: `${id}.md`, + frontmatter: { + title: id, + created_at: '2026-01-01T00:00:00.000Z', + ...(kind === 'effort' ? {} : { effort: 'eff-a' }), + ...frontmatter, + }, + body_excerpt: body ?? '', + relations: relations ?? {}, + }; +} + +test('tokenize keeps numbers, units, and code terms and splits hyphens', (t) => { + const terms = tokenize( + "Lantern's ray-marched forward+ pass costs 4.7 ms at 120Hz in r11g11b10" + ); + for (const term of [ + 'lantern', + 'raymarch', + 'ray', + 'march', + 'forward+', + 'cost', + '4.7', + 'ms', + '4.7ms', + '120hz', + 'r11g11b10', + ]) + t.true(terms.includes(term), term); + t.false(terms.includes('the')); + t.false(terms.includes('at')); + t.deepEqual(tokenize('Caches cached caching cache').slice(0, 4), [ + 'cach', + 'cach', + 'cach', + 'cach', + ]); + t.is(stem('running'), 'run'); + t.is(stem('queries'), 'query'); + t.is(stem('status'), 'status'); + t.deepEqual( + codeReferences( + 'See packages/render/tern/history.ts and resolveHistory() in a1b2c3d, PR #412.' + ), + ['packages/render/tern/history.ts', 'resolveHistory()', 'a1b2c3d', '#412'] + ); + const code = tokenize('packages/render/tern/history.ts'); + t.true(code.includes('packages/render/tern/history.ts')); + t.true(code.includes('history.ts')); +}); + +test('BM25 ranks the record about the query first, ties break by id', (t) => { + const index = buildFindIndex([ + rec('eff-a', { title: 'Renderer' }), + rec('fnd-b', { + title: 'Tern history buffer precision', + body: 'The history buffer keeps 16 bits per channel.', + }), + rec('fnd-c', { + title: 'Build times', + body: 'The build takes ten minutes.', + }), + rec('fnd-d', { + title: 'Shadow cascade', + body: 'Buffer sizes for shadows.', + }), + rec('fnd-e', { title: 'Same words', body: 'zebra' }), + rec('fnd-f', { title: 'Same words', body: 'zebra' }), + ]); + const result = findRecords(index, 'tern history buffer', { effort: 'eff-a' }); + t.is(result.hits[0].record.id, 'fnd-b'); + t.is(result.hits[1].record.id, 'fnd-d'); + t.is(result.matched, 2); + t.is(result.searched, 5); + t.deepEqual(result.missing, []); + const tie = findRecords(index, 'zebra', { effort: 'eff-a' }); + t.deepEqual( + tie.hits.map((hit) => hit.record.id), + ['fnd-e', 'fnd-f'] + ); + const none = findRecords(index, 'tern unicorn', { effort: 'eff-a' }); + t.deepEqual(none.found, ['tern']); + t.deepEqual(none.missing, ['unicorn']); +}); + +test('a supersession chain is one hit showing its head; as-of picks the version then current', (t) => { + const index = buildFindIndex([ + rec('eff-a', { title: 'Renderer' }), + rec('dec-v1', { + title: 'Use TAA for upscaling', + body: 'Temporal antialiasing with a sharpen pass.', + created_at: '2025-11-02T00:00:00.000Z', + state: 'accepted', + relations: { superseded_by: ['dec-v2'] }, + }), + rec('dec-v2', { + title: 'Use Tern for upscaling', + body: 'Tern replaces the old path.', + created_at: '2026-01-12T00:00:00.000Z', + state: 'accepted', + relations: { supersedes: ['dec-v1'], superseded_by: ['dec-v3'] }, + }), + rec('dec-v3', { + title: 'Use Tern with history clamp for upscaling', + body: 'Adds a clamp.', + created_at: '2026-03-01T00:00:00.000Z', + state: 'accepted', + relations: { supersedes: ['dec-v2'] }, + }), + ]); + const result = findRecords(index, 'temporal antialiasing sharpen', { + effort: 'eff-a', + }); + t.is(result.hits.length, 1); + const [hit] = result.hits; + t.is(hit.record.id, 'dec-v3'); + t.is(hit.head.id, 'dec-v3'); + t.is(hit.matched.id, 'dec-v1'); + t.deepEqual( + hit.chain.map((member) => member.id), + ['dec-v1', 'dec-v2', 'dec-v3'] + ); + const cards = renderFindCards(index, result); + t.regex(cards, /matched: v1 dec-v1/); + t.regex( + cards, + /chain: v1 dec-v1 superseded 2025-11-02 -> v2 dec-v2 superseded 2026-01-12 -> v3 dec-v3 accepted 2026-03-01 \(head\)/ + ); + const asOf = findRecords(index, 'upscaling', { + effort: 'eff-a', + asOf: '2026-02-01T00:00:00.000Z', + }); + t.is(asOf.hits[0].record.id, 'dec-v2'); + t.is(asOf.hits[0].head.id, 'dec-v3'); + t.regex( + renderFindCards(index, asOf), + /v2 dec-v2 superseded 2026-01-12 \(as of\)/ + ); + const before = findRecords(index, 'upscaling', { + effort: 'eff-a', + asOf: '2025-01-01T00:00:00.000Z', + }); + t.is(before.hits.length, 0); +}); + +test('the snippet is the window with the most matched terms, even past 600 characters', (t) => { + const filler = 'Nothing to see in this sentence at all. '.repeat(20); + const body = `${filler}The saltglass caustic pass clamps luminance to 4.7 ms per frame.`; + t.true(filler.length > 600); + const snippet = bestSnippet(body, new Set(['saltglass', 'caustic'])); + t.true(snippet.start > 600); + t.true(snippet.start <= body.indexOf('saltglass')); + t.is(snippet.length, body.length); + t.regex(snippet.text, /saltglass caustic pass/); + t.true(snippet.end - snippet.start <= 160); + const index = buildFindIndex([ + rec('eff-a'), + rec('fnd-long', { title: 'Frame costs', body }), + ]); + const cards = renderFindCards( + index, + findRecords(index, 'saltglass caustic', { effort: 'eff-a' }) + ); + t.regex( + cards, + new RegExp(`\\[chars ${snippet.start}-${snippet.end} of ${body.length}\\]`) + ); +}); + +test('cards list stored links and reverse links, capped per relation', (t) => { + const index = buildFindIndex([ + rec('eff-a'), + rec('iss-q', { + title: 'Which upscaler', + status: 'resolved', + relations: { resolved_by: ['dec-pick'] }, + }), + rec('fnd-cost', { title: 'Tern cost at 4K' }), + rec('dec-pick', { + title: 'Pick the zephyr upscaler', + state: 'accepted', + relations: { derives_from: ['fnd-cost'] }, + }), + ...['a', 'b', 'c', 'd', 'e'].map((x) => + rec(`dec-alt-${x}`, { + title: `Alternative ${x} with a long title that runs well past sixty characters`, + state: 'rejected', + relations: { rejected_by: ['dec-pick'] }, + }) + ), + ]); + const result = findRecords(index, 'zephyr', { effort: 'eff-a' }); + const [hit] = result.hits; + t.deepEqual( + hit.inbound.map((link) => link.relation), + ['resolves', 'rejects'] + ); + const cards = renderFindCards(index, result); + t.regex(cards, /out: derives_from fnd-cost "Tern cost at 4K" \(live\)/); + t.regex(cards, /in: resolves iss-q "Which upscaler" \(resolved\)/); + t.regex( + cards, + /rejects dec-alt-a "Alternative a with a long title that runs well past sixty c…" \(rejected\)/ + ); + t.regex(cards, /dec-alt-c [^;]+ \+2 more/); + t.false(cards.includes('dec-alt-d')); + // A record with no body gets no snippet line. + t.false(cards.includes('[chars')); +}); + +test('retracted records are left out unless asked for, and marked when shown', (t) => { + const index = buildFindIndex([ + rec('eff-a'), + rec('fnd-live', { title: 'Quokka live' }), + rec('fnd-gone', { + title: 'Quokka noise', + retracted: true, + retracted_reason: 'session noise', + }), + ]); + const ids = (mode?: 'exclude' | 'include' | 'only') => + findRecords(index, 'quokka', { effort: 'eff-a', retracted: mode }) + .hits.map((hit) => hit.record.id) + .sort(); + t.deepEqual(ids(), ['fnd-live']); + t.deepEqual(ids('include'), ['fnd-gone', 'fnd-live']); + t.deepEqual(ids('only'), ['fnd-gone']); + const cards = renderFindCards( + index, + findRecords(index, 'quokka', { effort: 'eff-a', retracted: 'only' }) + ); + t.regex(cards, /fnd-gone \| finding \| RETRACTED/); + t.regex(cards, /RETRACTED: session noise; not live/); +}); + +test('without an Effort, hits group by Effort, at most 3 each unless one Effort leads', (t) => { + const records = [ + rec('eff-a', { title: 'Render' }), + rec('eff-b', { title: 'World' }), + ]; + const add = (effort: string, id: string) => + records.push( + rec(id, { + effort, + title: `Shadow pool overflow ${id}`, + body: 'shadow pool overflow at low tide', + }) + ); + for (let i = 0; i < 6; i++) add('eff-a', `fnd-a${i}`); + for (let i = 0; i < 3; i++) add('eff-b', `fnd-b${i}`); + const index = buildFindIndex(records); + // Equal mass: the cap applies, and the hits group by Effort. + const result = findRecords(index, 'shadow pool overflow low tide'); + t.deepEqual( + result.efforts?.map((effort) => effort.id), + ['eff-a', 'eff-b'] + ); + t.deepEqual( + result.hits.map((hit) => hit.effort), + ['eff-a', 'eff-a', 'eff-a', 'eff-b', 'eff-b'] + ); + const cards = renderFindCards(index, result); + t.regex( + cards, + /efforts by match: eff-a "Render" [\d.]+ \| eff-b "World" [\d.]+/ + ); + t.regex(cards, /## eff-a "Render"[\s\S]*## eff-b "World"/); + // Slots the cap leaves empty go to the next hits by score. + const ten = findRecords(index, 'shadow pool overflow low tide', { + limit: 10, + }); + t.is(ten.hits.length, 9); + const scoped = findRecords(index, 'shadow pool overflow', { + effort: 'eff-a', + }); + t.is(scoped.hits.length, 5); + t.is(scoped.efforts, undefined); + // One Effort with at least 1.5 times the next one's mass is not capped. + const lead = buildFindIndex( + records.filter((record) => !['fnd-b1', 'fnd-b2'].includes(record.id)) + ); + t.deepEqual( + findRecords(lead, 'shadow pool overflow low tide').hits.map( + (hit) => hit.effort + ), + ['eff-a', 'eff-a', 'eff-a', 'eff-a', 'eff-a'] + ); +}); + +test('stemming meets added and add, used and use, and keeps real doubles', (t) => { + for (const [word, want] of [ + ['add', 'add'], + ['added', 'add'], + ['adding', 'add'], + ['buff', 'buff'], + ['buffed', 'buff'], + ['use', 'use'], + ['used', 'use'], + ['using', 'use'], + ['stopped', 'stop'], + ['seed', 'seed'], + ['thing', 'thing'], + ]) + t.is(stem(word), want, word); + const index = buildFindIndex([ + rec('eff-a'), + rec('fnd-retry', { body: 'Added retry to the uploader.' }), + rec('fnd-slice', { body: 'The vertical slice ships in May.' }), + ]); + t.deepEqual( + findRecords(index, 'add retry', { effort: 'eff-a' }).hits.map( + (hit) => hit.record.id + ), + ['fnd-retry'] + ); + // A hyphenated word whose parts both matched is not reported missing. + const slice = findRecords(index, 'vertical-slice', { effort: 'eff-a' }); + t.deepEqual(slice.found, ['vertical', 'slice']); + t.deepEqual(slice.missing, []); +}); + +test('chains follow supersession links, not created times', (t) => { + const index = buildFindIndex([ + rec('eff-a'), + rec('dec-c1', { + title: 'Walrus codec', + created_at: '2026-05-01T00:00:00.000Z', + relations: { superseded_by: ['dec-c2'] }, + }), + rec('dec-c2', { + title: 'Walrus codec, revised', + created_at: '2026-04-01T00:00:00.000Z', + relations: { supersedes: ['dec-c1'] }, + }), + ]); + const [hit] = findRecords(index, 'walrus', { effort: 'eff-a' }).hits; + t.deepEqual( + hit.chain.map((member) => member.id), + ['dec-c1', 'dec-c2'] + ); + t.is(hit.head.id, 'dec-c2'); + const [asOf] = findRecords(index, 'walrus', { + effort: 'eff-a', + asOf: '2026-05-15T00:00:00Z', + }).hits; + t.is(asOf.record.id, 'dec-c2'); +}); + +test('as-of scores and quotes only versions written by then', (t) => { + const index = buildFindIndex([ + rec('eff-a'), + rec('dec-v1', { + body: 'We use the alpha codec.', + created_at: '2026-01-01T00:00:00.000Z', + relations: { superseded_by: ['dec-v2'] }, + }), + rec('dec-v2', { + body: 'We switch to the zeta codec after the March capture.', + created_at: '2026-03-01T00:00:00.000Z', + relations: { supersedes: ['dec-v1'] }, + }), + ]); + const options = { effort: 'eff-a', asOf: '2026-02-01T00:00:00Z' }; + t.is(findRecords(index, 'zeta', options).hits.length, 0); + const result = findRecords(index, 'zeta codec', options); + const [hit] = result.hits; + t.is(hit.record.id, 'dec-v1'); + t.is(hit.matched.id, 'dec-v1'); + t.false(hit.snippet.text.includes('zeta')); + t.deepEqual(result.missing, ['zeta']); +}); + +test('the card shows a version in scope, never an out-of-scope head', (t) => { + const index = buildFindIndex([ + rec('eff-a'), + rec('eff-b'), + rec('dec-r1', { + title: 'Kelp farm layout', + relations: { superseded_by: ['dec-r2'] }, + }), + rec('dec-r2', { + title: 'Kelp farm layout v2', + created_at: '2026-02-01T00:00:00.000Z', + retracted: true, + relations: { supersedes: ['dec-r1'] }, + }), + rec('fnd-x1', { + title: 'Zebra crossing', + relations: { superseded_by: ['fnd-x2'] }, + }), + rec('fnd-x2', { + title: 'Zebra crossing moved', + effort: 'eff-b', + created_at: '2026-02-01T00:00:00.000Z', + relations: { supersedes: ['fnd-x1'] }, + }), + ]); + const [kelp] = findRecords(index, 'kelp').hits; + t.is(kelp.record.id, 'dec-r1'); + t.is(kelp.head.id, 'dec-r2'); + t.notRegex(renderFindCards(index, findRecords(index, 'kelp')), /RETRACTED/); + const [only] = findRecords(index, 'kelp', { retracted: 'only' }).hits; + t.is(only.record.id, 'dec-r2'); + const [zebra] = findRecords(index, 'zebra', { effort: 'eff-a' }).hits; + t.is(zebra.record.id, 'fnd-x1'); + t.is(zebra.effort, 'eff-a'); +}); + +test('long chains keep the head and the matched version, and snippets near the top start at 0', (t) => { + const ids = [1, 2, 3, 4, 5, 6].map((n) => `dec-l${n}`); + const index = buildFindIndex([ + rec('eff-a'), + ...ids.map((id, i) => + rec(id, { + title: i === 0 ? 'Narwhal budget' : `Budget ${i}`, + body: i === 0 ? 'As of beta the narwhal mode costs 4 ms.' : 'Budget.', + created_at: `2026-0${i + 1}-01T00:00:00.000Z`, + relations: { + ...(i ? { supersedes: [ids[i - 1]] } : {}), + ...(i < 5 ? { superseded_by: [ids[i + 1]] } : {}), + }, + }) + ), + ]); + const result = findRecords(index, 'narwhal beta', { effort: 'eff-a' }); + t.is(result.hits[0].snippet.start, 0); + const cards = renderFindCards(index, result); + t.regex(cards, /"As of beta the narwhal/); + t.regex( + cards, + /chain: v1 dec-l1 [^>]+ -> \(2 more\) -> v4 dec-l4 [^>]+ -> v5 dec-l5 [^>]+ -> v6 dec-l6 \S+ \S+ \(head\)/ + ); +}); + +test('five dense cards stay within the inline budget', (t) => { + const records: ReadRecord[] = [rec('eff-a', { title: 'Render' })]; + const long = 'x'.repeat(40); + for (let i = 0; i < 10; i++) + records.push( + rec(`dec-hit-${i}-${long}`, { + title: `Gloam upscaler choice ${i} with a fairly long descriptive title`, + body: `${'Context words here. '.repeat( + 40 + )} gloam upscaler packages/render/gloam/file${i}.ts`, + relations: { + derives_from: [0, 1, 2, 3, 4].map((j) => `fnd-dep-${j}-${long}`), + cites: [0, 1, 2].map((j) => `cit-src-${j}-${long}`), + }, + }) + ); + for (let j = 0; j < 5; j++) + records.push( + rec(`fnd-dep-${j}-${long}`, { + title: 'A dependency finding with a title long enough to cut', + }) + ); + const index = buildFindIndex(records); + const cards = renderFindCards( + index, + findRecords(index, 'gloam upscaler', { effort: 'eff-a' }) + ); + t.true(cards.length / 4 <= 1000, `${cards.length / 4} tokens`); + // A relation with no entry left still names itself, with its count. + t.false(cards.includes('in other relations')); +}); diff --git a/packages/flatbread/src/proof/find.ts b/packages/flatbread/src/proof/find.ts new file mode 100644 index 00000000..658ee2d8 --- /dev/null +++ b/packages/flatbread/src/proof/find.ts @@ -0,0 +1,994 @@ +import { + recordStanding, + type PrimitiveKind, + type ReadRecord, + type ReadRelation, +} from '@flatbread/proof'; + +/** + * `proof find`: ranked keyword search over the record files. Everything here + * is a pure function of the records passed in. There are no embeddings, no + * random choices, and no locale-aware comparisons, so the same files give the + * same hits on every machine. + */ + +export const FIND_DEFAULT_LIMIT = 5; +export const FIND_MAX_LIMIT = 10; +/** BM25 term-frequency saturation and length normalization. */ +const K1 = 1.2; +const B = 0.75; +const TITLE_WEIGHT = 2; +/** Extra weight for entity terms: numbers with units, people, code refs. */ +const ENTITY_BOOST = 1.5; +const SNIPPET_CHARS = 160; +const LINKS_PER_RELATION = 3; +const LINK_TITLE_CHARS = 60; +const HITS_PER_EFFORT = 3; +/** + * Without `--effort`, the per-Effort cap applies only when no Effort clearly + * leads: when the top Effort's score mass is under 1.5 times the next one's. + */ +const EFFORT_LEAD = 1.5; +const EFFORTS_SHOWN = 5; +const CHAIN_VERSIONS_SHOWN = 4; +const CODE_REFS_SHOWN = 6; +/** + * Characters the cards aim for: a header share plus a share per hit, so five + * hits aim for about 4,000 characters (near 1,000 tokens at 4 characters a + * token). The renderer shrinks link titles, the Effort ranking line, chain + * lines, and link lists to fit, in that order. Titles, snippets, and code + * lines are never cut further, so a card set can still run past the aim. On + * the 10x benchmark graph the whole JSON envelope for five hits ran about + * 5,700 characters at the median and up to about 6,900. + */ +const CARDS_HEADER_CHARS = 300; +const CARD_CHARS = 750; + +const STOPWORDS = new Set( + ( + 'a an the of to in on for and or nor but is are was were be been being ' + + 'by with what which who whom whose how why when where that this these ' + + 'those it its at as from into than then there their they them we our us ' + + 'you your i me my he she his her do does did done has have had not no ' + + 'so if any all each can could would should will shall may might must' + ).split(' ') +); + +/** + * A word: letters and digits, with inner apostrophes, hyphens, underscores, + * and dots between digits (4.7), plus trailing `+` signs (forward+). + */ +const WORD = + /[\p{L}\p{N}]+(?:(?:['’]|(?<=\p{N})\.(?=\p{N})|[-_])[\p{L}\p{N}]+)*\+{0,2}/gu; +const UNIT = + /(?<![\p{L}\p{N}.])(\d+(?:\.\d+)?)\s?(ms|µs|us|ns|s|hz|khz|mhz|ghz|fps|px|kb|mb|gb|tb|kib|mib|gib|k|x|%)(?![\p{L}\p{N}])/giu; +const CODE_EXTENSIONS = + 'ts|tsx|js|jsx|mjs|cjs|rs|go|py|rb|java|kt|swift|c|h|cc|cpp|hpp|cs|json|ya?ml|toml|md|css|scss|html|sql|sh|glsl|hlsl|wgsl|vue|svelte'; +const CODE_PATH = new RegExp( + `(?<![\\w/.-])((?:[\\w.-]+/)*[\\w-]+(?:\\.[\\w-]+)*\\.(?:${CODE_EXTENSIONS}))(?![\\w/])`, + 'g' +); +const SYMBOL = /(?<![\w$.])([A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)*)\(\)/g; +const SHA = /(?<![\p{L}\p{N}])[0-9a-f]{7,40}(?![\p{L}\p{N}])/giu; +const ISSUE_NUMBER = /(?<![\w&#])#(\d+)\b/g; +const PULL_URL = /github\.com\/[\w.-]+\/[\w.-]+\/(?:pull|issues)\/(\d+)/g; + +/** One term found in a text, with where it sits. */ +interface Span { + readonly term: string; + /** The word as written, lowercased, before stemming. */ + readonly surface: string; + readonly start: number; + readonly end: number; + /** For a hyphenated word's joined form, the terms of its parts. */ + readonly parts?: readonly string[]; +} + +/** A doubled final consonant that -ed and -ing add: stopped, running. */ +const DOUBLED = /([bdgmnprt])\1$/; +/** A two-letter root that lost a final e: us(e), ow(e), ag(e). */ +const SHORT_ROOT = /^[aeiou][b-df-hj-np-tv-z]$/; + +/** + * Light, deterministic stemming for plain words: plural -s and -ies, -ed, + * -ing, and a final -e, so cache, caches, cached, and caching meet, and so do + * add, added, and adding, and use, used, and using. + */ +export function stem(word: string): string { + if (!/^[a-z]+$/.test(word) || word.length <= 3) return word; + let w = word; + if (w.endsWith('ies') && w.length > 4) w = `${w.slice(0, -3)}y`; + else if ( + w.endsWith('s') && + !w.endsWith('ss') && + !w.endsWith('us') && + !w.endsWith('is') + ) + w = w.slice(0, -1); + const suffix = w.endsWith('ing') ? 3 : w.endsWith('ed') ? 2 : 0; + const root = w.slice(0, w.length - suffix); + if (suffix && root.length >= 3) + // Undouble only when 3 letters stay, so add and egg keep their pair. + w = root.length >= 4 && DOUBLED.test(root) ? root.slice(0, -1) : root; + else if (suffix && SHORT_ROOT.test(root)) w = `${root}e`; + if (w.endsWith('e') && w.length >= 4) w = w.slice(0, -1); + return w; +} + +function wordSpans(text: string): Span[] { + const spans: Span[] = []; + for (const match of text.matchAll(WORD)) { + const start = match.index ?? 0; + const end = start + match[0].length; + const raw = match[0].toLowerCase().replace(/['’]/g, ''); + const parts = raw.split(/[-_]/).filter(Boolean); + if (parts.length > 1) { + const joined = parts.join(''); + const kept = parts.filter((part) => !STOPWORDS.has(part)); + if (!STOPWORDS.has(joined)) + spans.push({ + term: stem(joined), + surface: joined, + start, + end, + parts: kept.map(stem), + }); + for (const surface of kept) + spans.push({ term: stem(surface), surface, start, end }); + } else if (!STOPWORDS.has(raw)) + spans.push({ term: stem(raw), surface: raw, start, end }); + } + return spans; +} + +function codeSpans(text: string): Span[] { + const spans: Span[] = []; + const add = (term: string, start: number, end: number) => + spans.push({ term, surface: term, start, end }); + for (const match of text.matchAll(CODE_PATH)) { + const start = match.index ?? 0; + const path = match[1].toLowerCase(); + add(path, start, start + match[0].length); + const base = path.slice(path.lastIndexOf('/') + 1); + if (base !== path) add(base, start, start + match[0].length); + } + for (const match of text.matchAll(SYMBOL)) { + const start = match.index ?? 0; + const name = match[1].toLowerCase(); + add(`${name}()`, start, start + match[0].length); + const last = name.slice(name.lastIndexOf('.') + 1); + if (last !== name) add(`${last}()`, start, start + match[0].length); + } + for (const match of text.matchAll(SHA)) + if (/\d/.test(match[0]) && /[a-f]/i.test(match[0])) + add( + match[0].toLowerCase(), + match.index ?? 0, + (match.index ?? 0) + match[0].length + ); + for (const pattern of [ISSUE_NUMBER, PULL_URL]) + for (const match of text.matchAll(pattern)) + add( + `#${match[1]}`, + match.index ?? 0, + (match.index ?? 0) + match[0].length + ); + return spans; +} + +function unitSpans(text: string): Span[] { + return [...text.matchAll(UNIT)].map((match) => { + const term = `${match[1]}${match[2].toLowerCase()}`; + const start = match.index ?? 0; + return { term, surface: term, start, end: start + match[0].length }; + }); +} + +/** Entity terms: numbers with units ('4.7 ms' becomes 4.7ms) and code refs. */ +function entitySpans(text: string): Span[] { + return [...unitSpans(text), ...codeSpans(text)]; +} + +/** The terms a text yields, words first, then entity terms. */ +export function tokenize(text: string): string[] { + return [...wordSpans(text), ...entitySpans(text)].map((span) => span.term); +} + +/** + * Code references a text names, as written: file paths, `symbol()` names, + * commit SHAs, and issue or pull request numbers. + */ +export function codeReferences(text: string): string[] { + const seen = new Set<string>(); + const refs: string[] = []; + const add = (ref: string) => { + if (seen.has(ref.toLowerCase())) return; + seen.add(ref.toLowerCase()); + refs.push(ref); + }; + for (const match of text.matchAll(CODE_PATH)) add(match[1]); + for (const match of text.matchAll(SYMBOL)) add(`${match[1]}()`); + for (const match of text.matchAll(SHA)) + if (/\d/.test(match[0]) && /[a-f]/i.test(match[0])) add(match[0]); + for (const pattern of [ISSUE_NUMBER, PULL_URL]) + for (const match of text.matchAll(pattern)) add(`#${match[1]}`); + return refs; +} + +function normalizeBody(body: string): string { + return body.replace(/\r\n?/g, '\n'); +} + +function text(value: unknown): string { + return typeof value === 'string' ? value : ''; +} + +function compareIds(a: string, b: string): number { + return a < b ? -1 : a > b ? 1 : 0; +} + +function time(record: ReadRecord): number { + const value = Date.parse(text(record.frontmatter.created_at)); + return Number.isNaN(value) ? 0 : value; +} + +/** Oldest first, then by id, so chains and ties always sort the same way. */ +function byCreated(a: ReadRecord, b: ReadRecord): number { + return time(a) - time(b) || compareIds(a.id, b.id); +} + +/** + * For each chain member, the members its links say are older: the ones it + * `supersedes`, and the ones that list it in `superseded_by`. + */ +function olderVersions( + members: readonly ReadRecord[] +): Map<string, Set<string>> { + const ids = new Set(members.map((member) => member.id)); + const older = new Map( + members.map((member) => [member.id, new Set<string>()]) + ); + for (const member of members) { + for (const id of member.relations.supersedes ?? []) + if (ids.has(id) && id !== member.id) older.get(member.id)?.add(id); + for (const id of member.relations.superseded_by ?? []) + if (ids.has(id) && id !== member.id) older.get(id)?.add(member.id); + } + return older; +} + +/** + * Puts a chain in link order: every version comes after the versions it + * supersedes, as `asof` in the benchmark's graph.mjs walks it. Created time, + * then id, breaks ties between fork branches. A cycle, which only a hand edit + * can make, releases its oldest member first. + */ +function linkOrder(members: readonly ReadRecord[]): ReadRecord[] { + const older = olderVersions(members); + const pending = [...members].sort(byCreated); + const placed = new Set<string>(); + const ordered: ReadRecord[] = []; + while (pending.length) { + const ready = pending.findIndex((member) => + [...(older.get(member.id) ?? [])].every((id) => placed.has(id)) + ); + const [next] = pending.splice(Math.max(ready, 0), 1); + placed.add(next.id); + ordered.push(next); + } + return ordered; +} + +/** The standing word for a record, as `brief` computes it. */ +export function standingOf(record: ReadRecord): string { + return recordStanding({ + id: record.id, + kind: record.kind, + frontmatter: { ...record.frontmatter, ...record.relations }, + }); +} + +interface FindDoc { + readonly record: ReadRecord; + readonly tf: ReadonlyMap<string, number>; + readonly length: number; +} + +/** Relations whose stored ids, read backward, give a record's inbound links. */ +const INBOUND_LABELS: Readonly<Record<string, string>> = { + derives_from: 'derived_by', + cites: 'cited_by', + resolved_by: 'resolves', + rejected_by: 'rejects', + mitigated_by: 'mitigates', + evidence: 'evidence_for', + blob: 'blob_of', +}; +/** Outbound links a card lists. Supersession shows on the chain line instead. */ +const OUTBOUND_RELATIONS: readonly ReadRelation[] = [ + 'derives_from', + 'invalidates', + 'invalidated_by', + 'resolved_by', + 'rejected_by', + 'mitigated_by', + 'evidence', + 'cites', +]; + +export interface FindIndex { + readonly records: ReadonlyMap<string, ReadRecord>; + readonly docs: ReadonlyMap<string, FindDoc>; + readonly df: ReadonlyMap<string, number>; + readonly averageLength: number; + /** Chain root for every record id; records with no supersession are their own root. */ + readonly chainOf: ReadonlyMap<string, string>; + /** Members of each chain in link order, oldest first. */ + readonly chains: ReadonlyMap<string, readonly ReadRecord[]>; + /** For each record id, the records that link to it, by inbound label. */ + readonly inbound: ReadonlyMap<string, ReadonlyMap<string, readonly string[]>>; +} + +function addTerms( + tf: Map<string, number>, + spans: readonly Span[], + weight: number +): number { + for (const span of spans) + tf.set(span.term, (tf.get(span.term) ?? 0) + weight); + return spans.length * weight; +} + +function indexDoc(record: ReadRecord): FindDoc { + const tf = new Map<string, number>(); + const title = text(record.frontmatter.title); + const body = normalizeBody(record.body_excerpt); + let length = 0; + length += addTerms(tf, wordSpans(title), TITLE_WEIGHT); + length += addTerms(tf, wordSpans(body), 1); + for (const field of ['kind', 'state', 'status', 'role']) + length += addTerms(tf, wordSpans(text(record.frontmatter[field])), 1); + length += addTerms( + tf, + wordSpans(text(record.frontmatter.created_by)), + ENTITY_BOOST + ); + length += addTerms(tf, entitySpans(`${title}\n${body}`), ENTITY_BOOST); + return { record, tf, length }; +} + +/** + * Builds the search index. Every non-Effort record is a document; Efforts are + * kept only for their titles. IDF (inverse document frequency, the BM25 + * weight that makes rare words count more) counts every record, whatever a + * later query filters to, so words common across the whole graph weigh + * little. + */ +export function buildFindIndex(records: readonly ReadRecord[]): FindIndex { + const byId = new Map<string, ReadRecord>(); + for (const record of [...records].sort(byCreated)) + byId.set(record.id, record); + const docs = new Map<string, FindDoc>(); + const df = new Map<string, number>(); + let total = 0; + for (const record of byId.values()) { + if (record.kind === 'effort') continue; + const doc = indexDoc(record); + docs.set(record.id, doc); + total += doc.length; + for (const term of doc.tf.keys()) df.set(term, (df.get(term) ?? 0) + 1); + } + // Union-find over supersession, so every version of a record is one chain. + const parent = new Map<string, string>(); + const root = (id: string): string => { + let current = id; + while (parent.get(current) !== current) + current = parent.get(current) as string; + parent.set(id, current); + return current; + }; + for (const id of docs.keys()) parent.set(id, id); + for (const doc of docs.values()) + for (const other of [ + ...(doc.record.relations.supersedes ?? []), + ...(doc.record.relations.superseded_by ?? []), + ]) { + if (!docs.has(other)) continue; + const [a, b] = [root(doc.record.id), root(other)]; + if (a !== b) parent.set(a < b ? b : a, a < b ? a : b); + } + const chainOf = new Map<string, string>(); + const chains = new Map<string, ReadRecord[]>(); + for (const doc of docs.values()) { + const id = root(doc.record.id); + chainOf.set(doc.record.id, id); + if (!chains.has(id)) chains.set(id, []); + (chains.get(id) as ReadRecord[]).push(doc.record); + } + for (const [id, members] of chains) chains.set(id, linkOrder(members)); + const inbound = new Map<string, Map<string, string[]>>(); + for (const record of byId.values()) + for (const [relation, label] of Object.entries(INBOUND_LABELS)) { + const targets = + relation === 'blob' + ? [record.frontmatter.blob].filter( + (id): id is string => typeof id === 'string' + ) + : record.relations[relation as ReadRelation] ?? []; + for (const target of targets) { + if (!inbound.has(target)) inbound.set(target, new Map()); + const links = inbound.get(target) as Map<string, string[]>; + if (!links.has(label)) links.set(label, []); + (links.get(label) as string[]).push(record.id); + } + } + return { + records: byId, + docs, + df, + averageLength: docs.size ? total / docs.size : 0, + chainOf, + chains, + inbound, + }; +} + +export interface FindOptions { + /** Search one Effort; leave it out to search every Effort. */ + readonly effort?: string; + readonly kinds?: readonly PrimitiveKind[]; + /** ISO date: show the version of each chain that was current then. */ + readonly asOf?: string; + readonly retracted?: 'exclude' | 'include' | 'only'; + readonly limit?: number; +} + +export interface FindLink { + readonly relation: string; + readonly ids: readonly string[]; +} + +export interface FindHit { + /** The version the card shows: the chain head, or the as-of version. */ + readonly record: ReadRecord; + readonly head: ReadRecord; + /** The version whose text scored best. */ + readonly matched: ReadRecord; + readonly chain: readonly ReadRecord[]; + readonly score: number; + readonly effort: string | null; + readonly snippet: { + readonly text: string; + readonly start: number; + readonly end: number; + readonly length: number; + }; + readonly outbound: readonly FindLink[]; + readonly inbound: readonly FindLink[]; + readonly code: readonly string[]; +} + +export interface FindEffortMass { + readonly id: string; + readonly title: string; + readonly mass: number; +} + +export interface FindResult { + readonly query: string; + readonly effort: string | null; + readonly searched: number; + readonly matched: number; + readonly hits: readonly FindHit[]; + /** + * Only when searching every Effort: the 5 Efforts with the highest score + * mass, the summed scores of each one's best 3 hits. + */ + readonly efforts?: readonly FindEffortMass[]; + /** Query words found in at least one shown hit, as written. */ + readonly found: readonly string[]; + /** Query words that no shown hit contains. */ + readonly missing: readonly string[]; +} + +export interface QueryTerm { + readonly term: string; + /** The word the term came from, as written, lowercased. */ + readonly surface: string; + /** For a hyphenated word's joined form, the terms of its parts. */ + readonly parts?: readonly string[]; +} + +/** Distinct query terms with the word each came from, in query order. */ +export function queryTerms(query: string): QueryTerm[] { + const seen = new Set<string>(); + const terms: QueryTerm[] = []; + for (const span of [...wordSpans(query), ...entitySpans(query)]) + if (!seen.has(span.term)) { + seen.add(span.term); + terms.push({ + term: span.term, + surface: span.surface, + ...(span.parts ? { parts: span.parts } : {}), + }); + } + return terms; +} + +function bm25( + index: FindIndex, + doc: FindDoc, + terms: readonly string[] +): number { + let score = 0; + const n = index.docs.size; + for (const term of terms) { + const tf = doc.tf.get(term); + if (!tf) continue; + const df = index.df.get(term) ?? 0; + const idf = Math.log(1 + (n - df + 0.5) / (df + 0.5)); + score += + (idf * tf * (K1 + 1)) / + (tf + K1 * (1 - B + (B * doc.length) / (index.averageLength || 1))); + } + return score; +} + +/** + * The 160-character window of `body` that holds the most distinct matched + * terms (then the most matches, then the earliest), with its offsets. + */ +export function bestSnippet( + body: string, + terms: ReadonlySet<string> +): FindHit['snippet'] { + const source = normalizeBody(body); + const hits = [...wordSpans(source), ...entitySpans(source)] + .filter((span) => terms.has(span.term)) + .sort((a, b) => a.start - b.start || a.end - b.end); + let start = 0; + if (hits.length) { + let best = { distinct: -1, count: -1, start: 0 }; + for (const anchor of hits) { + const from = Math.max(0, anchor.start - 20); + const inside = hits.filter( + (span) => span.start >= from && span.end <= from + SNIPPET_CHARS + ); + const distinct = new Set(inside.map((span) => span.term)).size; + if ( + distinct > best.distinct || + (distinct === best.distinct && inside.length > best.count) + ) + best = { distinct, count: inside.length, start: from }; + } + // Start at the top when that drops only a word or two. + start = best.start <= 20 ? 0 : best.start; + // Start on a word boundary, without skipping past the first match. + const first = hits.find((span) => span.start >= start)?.start ?? start; + while (start > 0 && start < first && /\S/.test(source[start - 1])) start++; + } + let end = Math.min(source.length, start + SNIPPET_CHARS); + if (end < source.length) { + const space = source.lastIndexOf(' ', end); + if (space > start + SNIPPET_CHARS * 0.75) end = space; + } + const window = source.slice(start, end).replace(/\s+/g, ' ').trim(); + return { + text: `${start > 0 ? '…' : ''}${window}${end < source.length ? '…' : ''}`, + start, + end, + length: source.length, + }; +} + +function inScope(record: ReadRecord, options: FindOptions): boolean { + const retracted = record.frontmatter.retracted === true; + const mode = options.retracted ?? 'exclude'; + return ( + (!options.effort || record.frontmatter.effort === options.effort) && + (!options.kinds?.length || options.kinds.includes(record.kind)) && + (mode === 'include' || (mode === 'only') === retracted) + ); +} + +/** Chain members that no other member supersedes, in link order. */ +function headsOf(chain: readonly ReadRecord[]): ReadRecord[] { + const superseded = new Set( + [...olderVersions(chain).values()].flatMap((ids) => [...ids]) + ); + return chain.filter((member) => !superseded.has(member.id)); +} + +function outboundLinks(record: ReadRecord): FindLink[] { + const links: FindLink[] = OUTBOUND_RELATIONS.flatMap((relation) => { + const ids = record.relations[relation] ?? []; + return ids.length ? [{ relation, ids: [...ids].sort(compareIds) }] : []; + }); + const blob = record.frontmatter.blob; + if (typeof blob === 'string') links.push({ relation: 'blob', ids: [blob] }); + return links; +} + +function inboundLinks(index: FindIndex, record: ReadRecord): FindLink[] { + const links = index.inbound.get(record.id); + if (!links) return []; + return Object.values(INBOUND_LABELS).flatMap((relation) => { + const ids = links.get(relation) ?? []; + return ids.length ? [{ relation, ids: [...ids].sort(compareIds) }] : []; + }); +} + +/** + * Runs a query. Each supersession chain is one hit, scored by its best + * member. Only versions that pass the query's scope count: the Effort, the + * kinds, the retracted mode, and, with `asOf`, a created time at or before + * it. They alone are scored, quoted, and shown. Without an Effort, when no + * Effort clearly leads, at most 3 hits come from any one Effort, so an + * ambiguous query shows its candidate Efforts side by side; slots the cap + * leaves empty go to the next hits by score. + */ +export function findRecords( + index: FindIndex, + query: string, + options: FindOptions = {} +): FindResult { + const terms = queryTerms(query); + const termList = terms.map(({ term }) => term); + const limit = Math.min( + FIND_MAX_LIMIT, + Math.max(1, options.limit ?? FIND_DEFAULT_LIMIT) + ); + const asOf = options.asOf ? Date.parse(options.asOf) : undefined; + const eligible = (record: ReadRecord) => + inScope(record, options) && (asOf === undefined || time(record) <= asOf); + let searched = 0; + const best = new Map<string, { record: ReadRecord; score: number }>(); + for (const doc of index.docs.values()) { + if (!eligible(doc.record)) continue; + searched++; + const score = bm25(index, doc, termList); + if (score <= 0) continue; + const chainId = index.chainOf.get(doc.record.id) as string; + const current = best.get(chainId); + if ( + !current || + score > current.score || + (score === current.score && + compareIds(doc.record.id, current.record.id) < 0) + ) + best.set(chainId, { record: doc.record, score }); + } + const ranked = [...best.entries()] + .flatMap(([chainId, { record: matched, score }]) => { + const chain = index.chains.get(chainId) as readonly ReadRecord[]; + const heads = headsOf(chain); + const head = heads.at(-1) ?? (chain.at(-1) as ReadRecord); + const versions = chain.filter(eligible); + // The card shows an eligible version: the newest head that is in + // scope, else the newest version that is. With `asOf`, the last version + // in link order created by then. `matched` is always eligible. + const record = + (asOf === undefined ? heads.filter(eligible).at(-1) : undefined) ?? + versions.at(-1) ?? + matched; + return [{ chain, versions, head, record, matched, score }]; + }) + .sort((a, b) => b.score - a.score || compareIds(a.head.id, b.head.id)); + const effortOf = (record: ReadRecord) => + typeof record.frontmatter.effort === 'string' + ? record.frontmatter.effort + : null; + let selected = ranked; + let efforts: FindEffortMass[] | undefined; + if (!options.effort) { + const perEffort = new Map<string, number[]>(); + for (const hit of ranked) { + const effort = effortOf(hit.record) ?? ''; + if (!perEffort.has(effort)) perEffort.set(effort, []); + (perEffort.get(effort) as number[]).push(hit.score); + } + efforts = [...perEffort.entries()] + .map(([id, scores]) => ({ + id, + title: text(index.records.get(id)?.frontmatter.title), + mass: scores + .slice(0, HITS_PER_EFFORT) + .reduce((sum, score) => sum + score, 0), + })) + .sort((a, b) => b.mass - a.mass || compareIds(a.id, b.id)); + const [top, next] = efforts; + const cap = + next && top.mass < EFFORT_LEAD * next.mass ? HITS_PER_EFFORT : limit; + const counts = new Map<string, number>(); + const capped: typeof ranked = []; + const over: typeof ranked = []; + for (const hit of ranked) { + const effort = effortOf(hit.record) ?? ''; + const count = counts.get(effort) ?? 0; + (count < cap ? capped : over).push(hit); + counts.set(effort, count + 1); + } + selected = [...capped, ...over].slice(0, limit); + const order = new Map(efforts.map((effort, i) => [effort.id, i])); + selected = [...selected].sort( + (a, b) => + (order.get(effortOf(a.record) ?? '') ?? 0) - + (order.get(effortOf(b.record) ?? '') ?? 0) || + b.score - a.score || + compareIds(a.head.id, b.head.id) + ); + } else selected = selected.slice(0, limit); + const querySet = new Set(termList); + const foundTerms = new Set<string>(); + const hits: FindHit[] = selected.map((hit) => { + for (const member of hit.versions) + for (const term of termList) + if (index.docs.get(member.id)?.tf.has(term)) foundTerms.add(term); + const code: string[] = []; + for (const member of [...hit.versions].reverse()) + for (const ref of codeReferences( + `${text(member.frontmatter.title)}\n${member.body_excerpt}` + )) + if (!code.some((seen) => seen.toLowerCase() === ref.toLowerCase())) + code.push(ref); + return { + record: hit.record, + head: hit.head, + matched: hit.matched, + chain: hit.chain, + score: hit.score, + effort: effortOf(hit.record), + snippet: bestSnippet(hit.matched.body_excerpt, querySet), + outbound: outboundLinks(hit.record), + inbound: inboundLinks(index, hit.record), + code, + }; + }); + // A hyphenated query word is searched joined and split. When every part + // was found, the joined form is not reported missing. + const missed = (entry: QueryTerm) => + !foundTerms.has(entry.term) && + !(entry.parts?.length && entry.parts.every((part) => foundTerms.has(part))); + const surfaces = (wanted: (entry: QueryTerm) => boolean) => [ + ...new Set(terms.filter(wanted).map((entry) => entry.surface)), + ]; + return { + query, + effort: options.effort ?? null, + searched, + matched: ranked.length, + hits, + ...(efforts ? { efforts: efforts.slice(0, EFFORTS_SHOWN) } : {}), + found: surfaces((entry) => foundTerms.has(entry.term)), + missing: surfaces(missed), + }; +} + +function cut(value: string, max: number): string { + const flat = value.replace(/\s+/g, ' ').trim(); + return [...flat].length > max + ? `${[...flat].slice(0, max - 1).join('')}…` + : flat; +} + +function date(record: ReadRecord): string { + return text(record.frontmatter.created_at).slice(0, 10) || 'undated'; +} + +function kindLabel(record: ReadRecord): string { + const sub = record.frontmatter.kind; + return typeof sub === 'string' && sub ? `${record.kind}:${sub}` : record.kind; +} + +/** How much each card may show; the renderer shrinks these to fit. */ +interface CardShape { + /** Link entries per `out:` or `in:` line. */ + readonly links: number; + /** Whether link entries carry their target's title. */ + readonly titles: boolean; + /** + * Most chain versions shown. The head and the shown and matched versions + * always show, even past it. + */ + readonly chain: number; +} + +function linkEntry(index: FindIndex, id: string, titles: boolean): string { + const target = index.records.get(id); + if (!target) return `${id} (missing)`; + return `${id}${ + titles ? ` "${cut(text(target.frontmatter.title), LINK_TITLE_CHARS)}"` : '' + } (${standingOf(target)})`; +} + +/** + * Renders link groups on one line, at most 3 per relation, and at most + * `shape.links` entries in all. A relation with no entry left shows only its + * count, as `cited_by +4`. + */ +function linkLine( + index: FindIndex, + label: string, + links: readonly FindLink[], + shape: CardShape +): string | undefined { + if (!links.length) return undefined; + let left = shape.links; + const parts = links.map((link) => { + const shown = link.ids.slice(0, Math.min(LINKS_PER_RELATION, left)); + left -= shown.length; + const more = link.ids.length - shown.length; + if (!shown.length) return `${link.relation} +${more}`; + return `${link.relation} ${shown + .map((id) => linkEntry(index, id, shape.titles)) + .join(', ')}${more ? ` +${more} more` : ''}`; + }); + return ` ${label}: ${parts.join('; ')}`; +} + +/** + * The chain line. It keeps the head and the shown and matched versions, + * fills up to `shape.chain` versions newest first, and counts the versions it + * leaves out, as `(2 more)`. + */ +function chainLine(hit: FindHit, shape: CardShape): string | undefined { + if (hit.chain.length < 2) return undefined; + const kept = new Set( + [hit.head, hit.record, hit.matched].map(({ id }) => + hit.chain.findIndex((member) => member.id === id) + ) + ); + for (let i = hit.chain.length - 1; i >= 0 && kept.size < shape.chain; i--) + kept.add(i); + const parts: string[] = []; + let skipped = 0; + hit.chain.forEach((member, i) => { + if (!kept.has(i)) { + skipped++; + return; + } + if (skipped) parts.push(`(${skipped} more)`); + skipped = 0; + const marks = [ + member.id === hit.head.id ? 'head' : '', + member.id === hit.record.id && member.id !== hit.head.id ? 'as of' : '', + ].filter(Boolean); + const killer = [ + ...(member.relations.invalidated_by ?? []).map( + (id) => `invalidated_by ${id}` + ), + ...(member.relations.rejected_by ?? []).map((id) => `rejected_by ${id}`), + ]; + parts.push( + `v${i + 1} ${member.id} ${standingOf(member)} ${date(member)}${ + killer.length ? ` ${killer.join(', ')}` : '' + }${marks.length ? ` (${marks.join(', ')})` : ''}` + ); + }); + if (skipped) parts.push(`(${skipped} more)`); + return ` chain: ${parts.join(' -> ')}`; +} + +function renderCard( + index: FindIndex, + hit: FindHit, + n: number, + shape: CardShape +): string[] { + const { record } = hit; + const standing = standingOf(record); + const retracted = record.frontmatter.retracted === true; + const version = hit.chain.findIndex((member) => member.id === hit.matched.id); + const lines = [ + `${n}. ${record.id} | ${kindLabel(record)} | ${ + retracted ? 'RETRACTED' : standing + }${ + record.id === hit.head.id && hit.chain.length > 1 ? ' (head)' : '' + } | ${date(record)} | score ${hit.score.toFixed(1)}`, + ` "${cut(text(record.frontmatter.title), 100)}"`, + ...(retracted + ? [ + ` RETRACTED${ + typeof record.frontmatter.retracted_at === 'string' + ? ` ${record.frontmatter.retracted_at.slice(0, 10)}` + : '' + }: ${cut( + text(record.frontmatter.retracted_reason), + 80 + )}; not live, kept for audit only`, + ] + : []), + ...(hit.matched.id !== record.id + ? [` matched: v${version + 1} ${hit.matched.id}`] + : []), + ...(hit.snippet.length + ? [ + ` "${hit.snippet.text}" [chars ${hit.snippet.start}-${hit.snippet.end} of ${hit.snippet.length}]`, + ] + : []), + ]; + for (const line of [ + chainLine(hit, shape), + linkLine(index, 'out', hit.outbound, shape), + linkLine(index, 'in', hit.inbound, shape), + ]) + if (line) lines.push(line); + if (hit.code.length) + lines.push( + ` code: ${hit.code.slice(0, CODE_REFS_SHOWN).join(', ')}${ + hit.code.length > CODE_REFS_SHOWN + ? ` +${hit.code.length - CODE_REFS_SHOWN} more` + : '' + }` + ); + return lines; +} + +/** + * Shapes to try, fullest first. Each step gives up one thing: link titles, + * then the Effort ranking line (the `##` headings and the JSON `find.efforts` + * still carry it), then the middle of each chain, then link entries. + */ +const SHRINK_STEPS: readonly (CardShape & { efforts: boolean })[] = [ + { links: 4, titles: true, chain: CHAIN_VERSIONS_SHOWN, efforts: true }, + { links: 4, titles: false, chain: CHAIN_VERSIONS_SHOWN, efforts: true }, + { links: 4, titles: false, chain: CHAIN_VERSIONS_SHOWN, efforts: false }, + { links: 4, titles: false, chain: 0, efforts: false }, + { links: 3, titles: false, chain: 0, efforts: false }, + { links: 2, titles: false, chain: 0, efforts: false }, + { links: 1, titles: false, chain: 0, efforts: false }, + { links: 0, titles: false, chain: 0, efforts: false }, +]; + +/** + * The cards as Markdown. The renderer measures the whole text and takes the + * first shape in SHRINK_STEPS that fits the character aim. + */ +export function renderFindCards(index: FindIndex, result: FindResult): string { + const scope = result.effort ? `in ${result.effort}` : 'in all Efforts'; + const render = (shape: typeof SHRINK_STEPS[number]) => { + const lines = [ + `find "${cut(result.query, 120)}" ${scope} (${ + result.searched + } records searched, ${result.matched} matched, ${ + result.hits.length + } shown)`, + `matched: ${result.found.join(' ') || 'none'} | missing: ${ + result.missing.join(' ') || 'none' + }`, + ]; + if (shape.efforts && result.efforts?.length) + lines.push( + `efforts by match: ${result.efforts + .map( + (effort) => + `${effort.id} "${cut(effort.title, 30)}" ${effort.mass.toFixed( + 1 + )}` + ) + .join(' | ')}` + ); + let effort: string | null | undefined; + result.hits.forEach((hit, i) => { + if (result.efforts && hit.effort !== effort) { + effort = hit.effort; + lines.push( + `## ${effort ?? 'no Effort'} "${cut( + text(index.records.get(effort ?? '')?.frontmatter.title), + 60 + )}"` + ); + } + lines.push(...renderCard(index, hit, i + 1, shape)); + }); + if (!result.hits.length) lines.push('no records matched'); + return lines.join('\n'); + }; + const limit = CARDS_HEADER_CHARS + CARD_CHARS * result.hits.length; + let markdown = ''; + for (const shape of SHRINK_STEPS) { + markdown = render(shape); + if (markdown.length <= limit) break; + } + return markdown; +} diff --git a/packages/flatbread/src/proof/read.ts b/packages/flatbread/src/proof/read.ts index 5eba5d10..217da941 100644 --- a/packages/flatbread/src/proof/read.ts +++ b/packages/flatbread/src/proof/read.ts @@ -17,6 +17,7 @@ import { READ_RELATIONS, readQueryHash, renderDigest, + type ReadCapReason, type ReadEdge, type ReadEnvelope, type ReadRecord, @@ -25,6 +26,14 @@ import { type PrimitiveKind, } from '@flatbread/proof'; import { loadConfig } from '@flatbread/config'; +import { + buildFindIndex, + FIND_DEFAULT_LIMIT, + FIND_MAX_LIMIT, + findRecords, + renderFindCards, + type FindIndex, +} from './find.js'; import { createHash } from 'node:crypto'; import { readdir, realpath, stat } from 'node:fs/promises'; import { join, relative, resolve } from 'node:path'; @@ -343,6 +352,7 @@ function configForCwd( class EngineProjection { readonly provider: FlatbreadProvider; private fieldsPromise?: Promise<Map<Collection, Set<string>>>; + private findIndexPromise?: Promise<FindIndex>; constructor(config: LoadedFlatbreadConfig) { this.provider = new FlatbreadProvider(config); @@ -427,6 +437,20 @@ class EngineProjection { return nodes.map((node) => toRecord(node, collection)); } + /** + * The `find` index over every record. It lives on the projection, so a + * cached projection keeps its index and a rebuilt one builds a new one. + */ + findIndex(): Promise<FindIndex> { + if (!this.findIndexPromise) { + this.findIndexPromise = Promise.all( + COLLECTIONS.map((collection) => this.query(collection)) + ).then((records) => buildFindIndex(records.flat())); + this.findIndexPromise.catch(() => (this.findIndexPromise = undefined)); + } + return this.findIndexPromise; + } + async one( collection: Collection, id: string @@ -516,6 +540,7 @@ async function render( fullBody?: boolean; sections?: DigestSection[]; compaction?: CompactionStatus; + capReasons?: ReadCapReason[]; } = {} ): Promise<ReadEnvelope> { const served = await servedGeneration(options.rootDir, options.consistency); @@ -1113,3 +1138,161 @@ export async function effortBrief( } ); } + +/** A `find` envelope: the usual read fields plus the hit cards inline. */ +export interface FindEnvelope extends ReadEnvelope { + find: { + query: string; + effort_id: string | null; + searched: number; + matched: number; + shown: number; + terms: { matched: string[]; missing: string[] }; + /** Searching every Effort: the Efforts whose best 3 hits score highest. */ + efforts?: { id: string; mass: number }[]; + /** In card order; `matched_id` names an older version when it matched. */ + hits: { id: string; matched_id?: string; score: number }[]; + }; + /** The hit cards as Markdown, so the agent needs no second Read. */ + cards: string; +} + +const FIND_KINDS: readonly PrimitiveKind[] = [ + 'issue', + 'finding', + 'decision', + 'constraint', + 'risk', + 'citation', + 'blob', +]; + +/** + * An as-of date: `YYYY-MM-DD` (read as UTC midnight) or a date-time with `Z` + * or an offset. Other forms are refused, because JavaScript reads them in the + * machine's own time zone and the hits would differ between machines. + */ +const AS_OF = + /^\d{4}-\d{2}-\d{2}(?:T\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:Z|[+-]\d{2}:\d{2}))?$/; + +function roundScore(score: number): number { + return Math.round(score * 1000) / 1000; +} + +/** + * Finds records by what they say: BM25 over titles, full bodies, a few + * frontmatter fields, and Blob payloads, one hit per supersession chain. + * The envelope carries the cards inline and writes them, with each hit's + * record, to the usual digest. + */ +export async function effortFind( + query: string, + options: EffortProjectionOptions & { + effort?: string; + kinds?: PrimitiveKind[]; + asOf?: string; + retracted?: RetractedMode; + limit?: number; + } +): Promise<FindEnvelope> { + const invalid = (message: string): never => { + throw new ProofReadValidationError('PROOF_INVALID_ARGUMENT', message); + }; + const retracted = options.retracted ?? 'exclude'; + if (!RETRACTED_MODES.includes(retracted)) + invalid( + `Invalid retracted mode: ${String( + retracted + )}. Expected exclude, include, or only.` + ); + const limit = options.limit ?? FIND_DEFAULT_LIMIT; + if (!Number.isInteger(limit) || limit < 1 || limit > FIND_MAX_LIMIT) + invalid(`limit must be an integer between 1 and ${FIND_MAX_LIMIT}`); + if ( + options.asOf !== undefined && + (!AS_OF.test(options.asOf) || Number.isNaN(Date.parse(options.asOf))) + ) + invalid( + `Invalid as-of date: ${options.asOf}. Expected YYYY-MM-DD or an ISO date-time with Z or an offset.` + ); + const kinds = options.kinds?.length + ? [...new Set(options.kinds)].sort() + : undefined; + const badKinds = (kinds ?? []).filter((kind) => !FIND_KINDS.includes(kind)); + if (badKinds.length) + invalid( + `Invalid kinds for find: ${badKinds.join( + ', ' + )}. Expected ${FIND_KINDS.join(', ')}.` + ); + const text = String(query ?? '').trim(); + if (!text) invalid('find needs at least one word to search for'); + const projection = await makeProjection(options); + const index = await projection.findIndex(); + if (options.effort && index.records.get(options.effort)?.kind !== 'effort') + invalid(`Effort ${options.effort} does not exist`); + const result = findRecords(index, text, { + effort: options.effort, + kinds, + asOf: options.asOf, + retracted, + limit, + }); + const cards = renderFindCards(index, result); + const envelope = await render( + options, + { + type: 'effortFind', + query: text, + effort_id: options.effort ?? null, + ...(kinds ? { kinds } : {}), + ...(options.asOf ? { as_of: options.asOf } : {}), + ...(retracted === 'exclude' ? {} : { retracted }), + limit, + }, + result.hits.map((hit) => hit.record), + [], + result.hits.map((hit) => `getRecord("${hit.record.id}")`), + { + sections: [{ heading: 'Find hits', lines: cards.split('\n') }], + // A ranked search that shows fewer records than matched is not complete. + capReasons: result.matched > result.hits.length ? ['find_limit'] : [], + } + ); + const summary = `${result.hits.length} shown of ${result.matched} matching (${ + result.searched + } searched); matched: ${result.found.join(' ') || 'none'}; missing: ${ + result.missing.join(' ') || 'none' + }`; + return { + ...envelope, + summary: + [...summary].length <= 640 + ? summary + : `${[...summary].slice(0, 639).join('')}…`, + find: { + query: text, + effort_id: result.effort, + searched: result.searched, + matched: result.matched, + shown: result.hits.length, + terms: { matched: [...result.found], missing: [...result.missing] }, + ...(result.efforts + ? { + efforts: result.efforts.map((effort) => ({ + id: effort.id, + mass: roundScore(effort.mass), + })), + } + : {}), + hits: result.hits.map((hit) => ({ + id: hit.record.id, + ...(hit.matched.id !== hit.record.id + ? { matched_id: hit.matched.id } + : {}), + score: roundScore(hit.score), + })), + }, + cards, + }; +} diff --git a/packages/proof/skills/proof/SKILL.md b/packages/proof/skills/proof/SKILL.md index f4d75e3c..65b8e104 100644 --- a/packages/proof/skills/proof/SKILL.md +++ b/packages/proof/skills/proof/SKILL.md @@ -11,7 +11,7 @@ repository. It has eight record types: **Effort**, **Issue**, **Finding**, **Citation** stores a source or reference; and **Blob** stores attached content such as a document, JSON, or image. Every record belongs to one Effort. Create and update records through 18 typed mutations, and read them -through 6 bounded queries. Do not hand-edit record frontmatter, although you +through 7 bounded queries. Do not hand-edit record frontmatter, although you may edit record bodies freely. Read [glossary.md](./glossary.md) for the primitive and edge semantics before @@ -169,6 +169,23 @@ an `artifact_path` to a rendered markdown digest (the evidence — spend one Read on it, or grep it), `served_generation`, page info, and ≤10 executable `hints`. Digests cap at 25 records / one-hop expansion / 50 edges / 64 KiB. +Pick the read by what you need, in this order: + +1. **Locate a record by what it says: `proof find "<words>"`.** It ranks + records by their title, full body, and Blob payload, and prints the hit + cards inline in the JSON, so you need no second Read. Each card shows the + newest version of the record (or, with `--as-of`, the version current + then), its standing, the passage that matched, its supersession chain + with dates, and its links in both directions. Leave out `--effort` when + you do not know the Effort: hits then group by Effort, at most 3 each + unless one Effort clearly leads. The `missing` terms name query words no + hit contains. Ordinary words often land there; a missing distinctive + term, such as a name, a number, or a code reference, is the cue that the + answer may not be recorded. +2. **Read one record in full: `proof get <id>`.** +3. **Count or page through records: `proof records`.** `find` shows only + the top hits, so never count from it. + Browse digests (`list`, `records`, `relations`, `blocking-decisions`) excerpt each body at 600 chars / 12 lines (`[…truncated]`). **`proof get` digests always include the full record body** (still subject to the 64 KiB digest @@ -176,6 +193,10 @@ byte cap). Zoom in with `get`, then Read/grep that digest — do not open `.flatbread-proof/**/*.md` for normal full-body recall. ```bash +# Find records by what they say (one Effort, or all when --effort is left out) +flatbread proof find "tern history buffer precision" --effort <effortId> +flatbread proof find "shadow pool overflow" --as-of 2026-03-01T00:00:00Z + # What's gating this effort? (proposed Decisions deriving from open blocker Issues) flatbread proof blocking-decisions <effortId> @@ -201,7 +222,8 @@ flatbread proof brief <effortId> Flags shared by reads: `--strict-min-generation <token>` (with optional `--timeout-ms <ms>`, default 3000) and, on `list`/`records`/`relations`, `--limit` (≤25) and `--cursor` (opaque `next_cursor` from a prior page; only valid for -the same query at the same generation). +the same query at the same generation). `find` takes `--limit` 1 to 10 +(default 5) and has no cursor. `proof list` is bounded Effort discovery. It defaults to `active`; valid statuses are exactly `active`, `paused`, `completed`, and `abandoned`. @@ -230,12 +252,15 @@ server-side. families showed this roughly halves recall tool calls with no loss of answer quality (Decision `dec-adopt-a-bounded-status-briefing-fast-path-for-ef--kcw0rw39g3b2ym2h`). -2. **When a browse digest shows `[…truncated]` and you need the body:** run +2. **When you need a particular record or fact:** run `proof find` first, + then `proof get` the hit whose card fits. Do not page through `records` + to look for it. +3. **When a browse digest shows `[…truncated]` and you need the body:** run `flatbread proof get <id>`, then Read/grep that digest (`artifact_path`) for the full body. Reserve opening `.flatbread-proof/**/*.md` for rare cases (e.g. digest byte-cap miss on an oversized record), not normal zoom-in. -3. **During work:** apply the write gate above before any create or body +4. **During work:** apply the write gate above before any create or body edit that adds claims. When outside material supports a 4/4 record, save large content with `WriteBlob` if needed, then create a `WriteCitation`, then create the Issue, Finding, Decision, Constraint, or Risk with @@ -243,10 +268,10 @@ server-side. Citation first. Open Issues for real gaps or blockers, and use `derives_from` on Decisions to link the Findings, Constraints, and Issues they respond to. -4. **On commitment:** preview `AcceptDecision` with `dryRun: true`, inspect +5. **On commitment:** preview `AcceptDecision` with `dryRun: true`, inspect its `rejectedIds`, then commit the acceptance. Use `ResolveIssue` with `resolvedBy` citing the closing Decision/Findings. These lifecycle transitions do not need a 4/4 score. Retract session noise with `Retract` rather than deleting files. -5. Maintenance: `flatbread proof cache prune` deletes digests older than +6. Maintenance: `flatbread proof cache prune` deletes digests older than 24h / over the 100 MiB ceiling. Prune does not need a 4/4 score. diff --git a/packages/proof/skills/proof/reference.md b/packages/proof/skills/proof/reference.md index 6adf5d9d..b484f68c 100644 --- a/packages/proof/skills/proof/reference.md +++ b/packages/proof/skills/proof/reference.md @@ -182,7 +182,7 @@ survivor (score 4/4 if it adds claims) plus `Retract` on the rest. `generation` is a durable, monotonic journal token — the input to strict reads. -## The 6 read queries +## The 7 read queries All reads execute through Flatbread's query engine (in-process GraphQL over the generated schema) and return a `ReadEnvelope`: @@ -209,6 +209,8 @@ lists), one-hop related records, and an edge table. Body policy: Blob payloads and Citation bodies (URLs / short notes). - **`proof brief`:** full body of the head compaction only. Every other record appears as one line. +- **`proof find`:** the hit cards (below) under `## Find hits`, and each hit's + record with the browse excerpt. The envelope also carries the cards inline. - **`list` / `records` / `relations` / `blocking-decisions`:** body excerpt capped at 600 chars / 12 lines (`[…truncated]`). **Blob bodies are omitted** from these digests — use `proof get` for the payload. Citation bodies @@ -231,7 +233,8 @@ order: error; do not retry the same page. 3. If `cap_reasons` is not empty, narrow the query or fail closed. It can hold several sorted, duplicate-free values from `primary_records`, - `displayed_edges`, and `bytes`. + `displayed_edges`, `bytes`, and `find_limit` (`proof find` matched more + records than it shows). 4. A hard cap alone leaves `page.has_more: false` and `next_cursor: null`. Paging and hard caps can occur together, but caps never create a cursor. @@ -247,13 +250,14 @@ flatbread proof records <effortId> [--kinds k1,k2] [--state s1,s2] [--status s1, flatbread proof relations <effortId> <fromId> --relations r1,r2 [--limit n] [--cursor c] [consistency flags] flatbread proof blocking-decisions <effortId> [consistency flags] flatbread proof brief <effortId> [consistency flags] +flatbread proof find "<words>" [--effort <effortId> | --all] [--kinds k1,k2] [--as-of iso] [--retracted exclude|include|only] [--limit n] [consistency flags] flatbread proof cache prune flatbread proof serve [--socket .flatbread/proof/serve.sock] ``` -- `proof serve` keeps the graph loaded in one process and answers the six +- `proof serve` keeps the graph loaded in one process and answers the seven reads above (`list`, `records`, `get`, `relations`, `blocking-decisions`, - `brief`); it serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its + `brief`, `find`); it keeps the `find` index with the graph and serves nothing that writes. Set `FLATBREAD_PROOF_SERVER` to its socket and those read commands go to it, in any folder whose graph root resolves to the same files. Each later read then takes a fraction of a second instead of rebuilding the graph. A write bumps the journal @@ -305,6 +309,72 @@ flatbread proof serve [--socket .flatbread/proof/serve.sock] stale head, or at least 5 changes. Programs read these fields, not the digest. +### `proof find` + +`find` locates records by what they say. Quoted or not, every word after +`find` is part of the query, locally and through `proof serve`. + +- **Scope:** `--effort <effortId>` searches one Effort. Without it, or with + `--all`, it searches every Effort. `--kinds` narrows to some of `issue`, + `finding`, `decision`, `constraint`, `risk`, `citation`, `blob` (default: + all seven; Efforts are never hits). `--retracted` works as on `records`: + retracted records are left out unless you pass `include` or `only`, and a + shown one is marked `RETRACTED`. Every version a card scores, quotes, or + shows passes this scope; other versions appear only on the chain line. +- **Ranking:** BM25 keyword scoring (k1 1.2, b 0.75) over the title (counted + twice), the full body, `kind`, `state`, `status`, `role`, `created_by`, and + Blob payloads. Rare words weigh more, measured over the whole graph. Words + are lowercased, apostrophes dropped, hyphenated words indexed joined and + split, and plurals, `-ed`, `-ing`, and a final `-e` folded (`added` meets + `add`, `using` meets `use`). Numbers with units + (`4.7 ms` is `4.7ms`), people (`created_by`), and code references (file + paths, `symbol()` names, commit SHAs, `#123`) count 1.5 times. Ties break by + id. The index is rebuilt from the record files; nothing derived is stored. +- **Chains:** all versions linked by `supersedes` / `superseded_by` form one + hit, scored by its best in-scope version. Versions are numbered `v1`, + `v2`, … in link order: each comes after the versions it supersedes, and + created time, then id, orders fork branches. The card shows the chain head, + or the newest in-scope version when the head is out of scope (for example, + retracted under the default `exclude`). If an older version matched, the + card says which (`matched: v1 <id>`). +- **As of:** `--as-of <date>` takes `YYYY-MM-DD` (UTC midnight) or a + date-time with `Z` or an offset, such as `2026-03-01T00:00:00Z`; other + forms are refused, so the answer does not depend on the machine's time + zone. Only versions created at or before that moment are scored, quoted, + and shown. The card shows the last such version in link order, and chains + with no such version drop out. +- **Grouping:** without `--effort`, Efforts are ranked by the summed score + of their best 3 hits (`efforts by match`), and hits are grouped under each + Effort. Unless the top Effort's sum is at least 1.5 times the next one's, + at most 3 hits come from one Effort at first; slots still empty then go to + the next hits by score. +- **Caps:** `--limit` 1 to 10, default 5. Each card holds at most 4 chain + versions (the head and the shown and matched versions always appear), 3 + links per relation, and 6 code references. When five cards run past about + 4,000 characters, the cards shrink in this order: link titles, the + `efforts by match` line, the middle of each chain, then link entries. A + relation with no entry left shows its count, as `cited_by +4`. When more + records matched than are shown, the envelope has `complete: false` and + `cap_reasons: ["find_limit"]`; there is no cursor, so narrow the query or + raise `--limit`. + +Each card has these lines: + +```text +1. <id> | <kind>[:<kind field>] | <standing> [(head)] | <created date> | score <n> + "<title>" + matched: v<n> <id> (only when an older version matched) + "…<160-character passage with the most query words>…" [chars <from>-<to> of <length>] + chain: v1 <id> <standing> <date> -> (n more) -> v4 <id> <standing> <date> (head) + out: <relation> <id> "<title, 60 chars>" (<standing>), … +n more; <relation> +n + in: <derived_by|cited_by|resolves|rejects|mitigates|evidence_for|blob_of> <id> "<title>" (<standing>) + code: <paths, symbol() names, SHAs, #numbers the chain's records name> +``` + +The card header reads `find "<words>" in <effortId|all Efforts> (<searched> records searched, <matched> matched, <shown> shown)`, then `matched: … | missing: …`. The envelope adds `cards` (that Markdown) and `find`: +`{"query","effort_id","searched","matched","shown","terms":{"matched","missing"},"efforts"?,"hits":[{"id","matched_id"?,"score"}]}`. +`find` shows only the top hits; use `records` to count. + ### Consistency flags - `--strict-min-generation <token>`: serve at or after that journal diff --git a/packages/proof/src/digest.ts b/packages/proof/src/digest.ts index abffd4af..e7cd0528 100644 --- a/packages/proof/src/digest.ts +++ b/packages/proof/src/digest.ts @@ -31,7 +31,15 @@ export interface ReadEdge { to_id: string; } -export type ReadCapReason = 'primary_records' | 'displayed_edges' | 'bytes'; +/** + * Why a read is not complete. `find_limit` means `proof find` matched more + * records than it shows; it pages no further, so narrow the query instead. + */ +export type ReadCapReason = + | 'primary_records' + | 'displayed_edges' + | 'bytes' + | 'find_limit'; /** Machine-readable state of an Effort's head compaction (`brief` only). */ export interface CompactionStatus { @@ -85,6 +93,8 @@ export interface DigestInput { /** Extra line-per-record sections, rendered after checkpoints. */ sections?: readonly DigestSection[]; compaction?: CompactionStatus; + /** Caps the caller applied before the digest, such as `find_limit`. */ + capReasons?: readonly ReadCapReason[]; } const CAP_RECORDS = 25; @@ -346,7 +356,7 @@ export async function renderDigest(input: DigestInput): Promise<ReadEnvelope> { ) ) .slice(0, CAP_EDGES); - const reasons: ReadCapReason[] = []; + const reasons: ReadCapReason[] = [...(input.capReasons ?? [])]; if (records.length > CAP_RECORDS) reasons.push('primary_records'); if (input.edges.length > CAP_EDGES) reasons.push('displayed_edges'); let completeness = digestCompleteness(hasMore, reasons);