diff --git a/benchmarks/proof-recall/README.md b/benchmarks/proof-recall/README.md index 887db35c..8c3cef29 100644 --- a/benchmarks/proof-recall/README.md +++ b/benchmarks/proof-recall/README.md @@ -18,7 +18,12 @@ Each question has a known answer and a stated path through the graph. | `questions/.json` | That Effort's questions and answers | | `questions/cross.json` | Questions that need two or more Efforts | | `questions/ambiguous.json` | Questions that name no Effort, asked only on request | +| `questions/code.json` | Which records govern a code path, on the git test bed | +| `questions/branch.json` | What open branches propose, on the git test bed | +| `spec/repo.mjs` | The git test bed: code, commits, anchors, branches, PRs | | `scripts/build.mjs` | Replays every spec through the Proof writer | +| `scripts/build-repo.mjs` | Builds the git test bed from `spec/repo.mjs` | +| `scripts/testbed.mjs` | Checks that read the test bed's refs, log, and PRs | | `scripts/check.mjs` | Proves each answer against the built graph | | `scripts/grade.mjs` | Scores one run | | `scripts/run.mjs` | Runs one headless agent session per question | @@ -354,11 +359,14 @@ A benchmark agent must never see `spec/`, `questions/`, or `keymap.json`. `run.mjs` gives every question its own folder and its own fresh, headless `claude -p` session, once per strategy. `--strategies` picks them from `brief` (start from `proof brief`), `nobrief`, `find` - (start from `proof find`), and `findstate` (the `find` text plus one line + (start from `proof find`), `findstate` (the `find` text plus one line that says what `proof state` holds: exact counts, open Issues and blockers, Risks, Constraints in force, contested topics, Issues closed without a fix, dead evidence, other dated changes, and retractions, with - long lists cut); the default is `brief,nobrief`. A server started before + long lists cut), and `findcode` (the `find` text plus two lines: run + `proof find --path` before opening code, and `proof find --refs all` + before building on a Decision; for the test bed below); the default is + `brief,nobrief`. A server started before `proof state` existed refuses it, so stop the server and let the run start a new one after rebuilding. Each folder holds the graph without the key map, the config, a `./flatbread` @@ -460,6 +468,96 @@ Three rules shaped this protocol, each learned from a pilot below: `scripts/prepare-run.mjs` and `scripts/collect.mjs` remain for runs driven by hand. +**The git test bed: code and branch questions.** Two sets ask what the +graph says about code and about unmerged work. They need a git repository, +which `scripts/build-repo.mjs` builds from `spec/repo.mjs`: + +```bash +node scripts/build-repo.mjs --out /tmp/p2-repo +node scripts/check.mjs --only code --repo /tmp/p2-repo +node scripts/check.mjs --only branch --repo /tmp/p2-repo +``` + +The repository holds about 35 source files for the studio's subsystems +(`engine/kestrel/`, `engine/lantern/`, `platform/skiff/`, `net/coop/`, +`tools/shadercache/`). Main has 29 commits. Most code commits carry one +`Proof: ` trailer per record they implement, written when that record +was current, so some trailers name records that were superseded later. One +commit adds the committed 1x graph. A later one writes code anchors through +the real Proof writer: Citations with `role: code` and a body of the form +`#@<12-character commit hash>`, each cited by an +implementation-note Finding or a Risk that derives from the governing +record. A few records name a file path in their body instead. Four branches +start at the tip of main, each with a draft PR in the gitignored +`.flatbread/proof/prs.json` (the shape `gh pr list --json number,headRefName,isDraft,author,title` prints): + +| Branch | PR | What it does to the graph | +| ------------------ | --- | ----------------------------------------------------------------------------------------------------------------------------------- | +| `ines/vsm-pool` | #41 | Supersedes the sun-shadow Decision and accepts the new one, resolves the spring-low-tide blocker, writes a new Lantern compaction | +| `tomasz/vsm-evict` | #44 | Supersedes the same Decision with a different, proposed one | +| `kenji/tide-lod` | #46 | Accepts the proposed answer to main's open far-cell tide question, resolves it, writes a new world compaction | +| `priya/skiff-cpu` | #48 | Retracts the storm capture main's Skiff blocker rests on, accepts the particle cap, and accepts a rival answer to the tide question | + +So merging the two VSM branches supersedes one record twice, and merging +the two tide branches leaves one question Issue with two accepted Decisions +and one Decision accepted on one side and rejected on the other. Commit +authors, dates, and messages and record ids and times are fixed, so every +build gives the same commit hashes. The builder writes the key map to +`.keymap.json`, beside the repository rather than in it. Branch +records have keys like main's, and the spec says which branch adds each. + +`code` and `branch` are opt-in sets, loaded only by name (`--only code`, +`--only branch`, or `--ids`), and multi-Effort sets like `cross`, so every +key is `.`. A plain `node scripts/check.mjs` skips them. Their +questions name code paths, behaviors, and branch names the way an engineer +would; `check.mjs` refuses one that contains a record id, a whole +four-word title, or any run of five words from a longer title, on main or +a branch. A question whose `path` runs +through branch records names that branch in `pathRef`. With `--repo`, +`check.mjs` reads the graph on main from the test bed and checks only +these two sets (both, when `--only` is left out), and these check types +become available (`scripts/testbed.mjs` documents every field): + +| `type` | Returns | +| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `atRef` | Any other check's result on the graph at a git ref, written out with `git archive` into a temporary folder. `ref: "*"` returns the branches where it gives `equals`, or their PR numbers or authors | +| `trailer` | Records named by `Proof:` trailers on commits that touch a file or folder: all commits, the latest, or the one that introduced a symbol | +| `codeAnchor` | Records that cite a `role: code` Citation for a file or folder. Each anchor must be one `spec/repo.mjs` declares, and git must show that its commit is on the ref, touches the file, and holds the symbol | +| `pr` | A branch's draft PR number, author, login, or draft flag | +| `refDiff` | Records a branch adds, retracts, restands, or changes since it left main | +| `changedPaths` | Code files a branch changes since it left main | +| `conflicts` | What merging two refs would do: two accepted Decisions on one question Issue, a record superseded twice, a standing each side changed differently, a retraction that a record still live after the merge relies on, or a compaction fork | + +`trailer` and `codeAnchor` can step along a link field (`follow: "derives_from"`), resolve to the chain head (`resolve: "head"`), and filter +like `select` (`kinds`, `where`, `live`, `titleIncludes`, `bodyIncludes`). + +To run them, pass the test bed to `run.mjs`: + +```bash +node scripts/run.mjs /tmp/recall-code --name --only code,branch \ + --repo /tmp/p2-repo --strategies findcode,find,nobrief +node scripts/grade.mjs runs/-findcode.json --only code,branch --repo /tmp/p2-repo +``` + +With `--repo`, each session's folder is its own `git clone --no-hardlinks` +of the test bed: main checked out, every branch as `origin/`, the +full log, and a copy of `prs.json`. The clone's `origin` points at a URL +that leads nowhere and its reflogs are deleted, since the real path sits +beside the key map; `prepareClone` fails if any file under `.git` still +names it. Every arm's `TASK.md` then adds a section that allows read-only +git (`git log`, `git show :`, `git diff main...origin/`, +`git branch -a`) and points at `prs.json`, so the baselines can reach +branch records, trailers, and PRs too; the four texts are unchanged for +runs without `--repo`. The folder +holds no key map, spec, or questions file, and the files a session adds +are listed in `.git/info/exclude`. These runs use neither the shared graph +copy nor the read server, since every clone is its own graph root. +`run.mjs` refuses to mix these sets with the others in one run, since the +test bed's graph holds records that could change their answers. +`--prepare-only` builds every session folder and its `TASK.md`, then +stops, without starting an agent, a server, or a shared graph copy; use it +to see what a session would see. + ## Pilot results The first pilot (`runs/pilot-1-*.json`) drew a stratified sample of 46 diff --git a/benchmarks/proof-recall/questions/branch.json b/benchmarks/proof-recall/questions/branch.json new file mode 100644 index 00000000..5d99ab4d --- /dev/null +++ b/benchmarks/proof-recall/questions/branch.json @@ -0,0 +1,848 @@ +{ + "effort": "branch", + "questions": [ + { + "id": "branch-001", + "tier": 1, + "hops": 1, + "pathRef": "ines/vsm-pool", + "categories": ["branch", "supersession"], + "question": "Does any open branch propose replacing the accepted sun-shadow Decision by growing the shadow page pool? Give its draft PR number.", + "answer": { + "kind": "value", + "value": "41", + "accept": ["#41"] + }, + "evidence": [ + "render.dec-vsm", + "render.dec-vsm-grow", + "render.dec-vsm-coarse" + ], + "path": [["render.dec-vsm", "superseded_by", "render.dec-vsm-grow"]], + "checks": [ + { + "type": "atRef", + "ref": "*", + "check": { + "type": "head", + "start": "render.dec-vsm" + }, + "equals": "render.dec-vsm-grow", + "report": "pr" + }, + { + "type": "pr", + "branch": "ines/vsm-pool", + "field": "number", + "expect": 41 + } + ], + "traps": [ + "tomasz/vsm-evict (draft #44) also replaces it, but with coarser pages on wet sand and the same pool size." + ] + }, + { + "id": "branch-002", + "tier": 4, + "hops": 0, + "categories": ["branch", "supersession"], + "question": "Two open branches each rewrite how engine/lantern/shadows/vsm_pool.rs copes with spring low tide. If both merged, which record would end up superseded by two different records?", + "answer": { + "kind": "record", + "value": "render.dec-vsm" + }, + "evidence": [ + "render.dec-vsm", + "render.dec-vsm-grow", + "render.dec-vsm-coarse" + ], + "checks": [ + { + "type": "conflicts", + "refs": ["ines/vsm-pool", "tomasz/vsm-evict"], + "kind": "double-supersede" + }, + { + "type": "changedPaths", + "ref": "*", + "under": "engine/lantern/shadows", + "expect": ["engine/lantern/shadows/vsm_pool.rs"] + }, + { + "type": "conflicts", + "refs": ["ines/vsm-pool", "tomasz/vsm-evict"], + "kind": "state-differs", + "expect": [] + } + ], + "traps": [ + "The Lantern summary is replaced on only one of the two branches, so it is not superseded twice." + ] + }, + { + "id": "branch-003", + "tier": 0, + "hops": 0, + "categories": ["branch", "current-state"], + "question": "On the branch that draws exposed wet sand with coarser shadow pages, what state is its new sun-shadow Decision in?", + "answer": { + "kind": "value", + "value": "proposed" + }, + "evidence": ["render.dec-vsm-coarse", "render.dec-vsm-grow"], + "checks": [ + { + "type": "atRef", + "ref": "tomasz/vsm-evict", + "check": { + "type": "field", + "record": "render.dec-vsm-coarse", + "field": "state" + } + } + ], + "traps": ["The rival replacement on ines/vsm-pool is accepted there."] + }, + { + "id": "branch-004", + "tier": 1, + "hops": 1, + "pathRef": "ines/vsm-pool", + "categories": ["branch", "blocker", "lifecycle"], + "question": "Which open branch closes the blocker about the shadow pool overflowing when the tide drains the flats? Give the branch name.", + "answer": { + "kind": "value", + "value": "ines/vsm-pool" + }, + "evidence": [ + "render.iss-vsm-tide", + "render.dec-vsm-grow", + "render.dec-vsm-coarse" + ], + "path": [["render.iss-vsm-tide", "resolved_by", "render.dec-vsm-grow"]], + "checks": [ + { + "type": "atRef", + "ref": "*", + "check": { + "type": "field", + "record": "render.iss-vsm-tide", + "field": "status" + }, + "equals": "resolved" + }, + { + "type": "atRef", + "ref": "main", + "check": { + "type": "field", + "record": "render.iss-vsm-tide", + "field": "status" + }, + "expect": "open" + } + ], + "traps": [ + "tomasz/vsm-evict builds its Decision on the blocker but leaves it open." + ] + }, + { + "id": "branch-005", + "tier": 4, + "hops": 0, + "categories": ["branch", "lifecycle"], + "question": "Two open branches each settle how the tide simulation should spend less time on cells far from players. If both merged, which question would be left with two accepted answers?", + "answer": { + "kind": "record", + "value": "world.iss-tide-lod-q" + }, + "evidence": [ + "world.iss-tide-lod-q", + "world.dec-tide-lod-quarter", + "world.dec-tide-lod-freeze" + ], + "checks": [ + { + "type": "conflicts", + "refs": ["kenji/tide-lod", "priya/skiff-cpu"], + "kind": "double-accept" + } + ], + "traps": [ + "On main the question is open with one proposed answer, so main alone shows no conflict." + ] + }, + { + "id": "branch-006", + "tier": 4, + "hops": 0, + "categories": ["branch", "rejected-alternative"], + "question": "Two open branches pick different ways to make far-off tide cells cheaper. Which Decision is accepted on one of them and rejected on the other?", + "answer": { + "kind": "record", + "value": "world.dec-tide-lod-quarter" + }, + "evidence": [ + "world.dec-tide-lod-quarter", + "world.dec-tide-lod-freeze", + "world.iss-tide-lod-q" + ], + "checks": [ + { + "type": "conflicts", + "refs": ["kenji/tide-lod", "priya/skiff-cpu"], + "kind": "state-differs" + }, + { + "type": "atRef", + "ref": "kenji/tide-lod", + "check": { + "type": "field", + "record": "world.dec-tide-lod-quarter", + "field": "standing" + }, + "expect": "accepted" + }, + { + "type": "atRef", + "ref": "priya/skiff-cpu", + "check": { + "type": "field", + "record": "world.dec-tide-lod-quarter", + "field": "standing" + }, + "expect": "rejected" + } + ], + "traps": [ + "The freeze Decision exists only on priya/skiff-cpu, so its state cannot differ between branches." + ] + }, + { + "id": "branch-007", + "tier": 1, + "hops": 1, + "pathRef": "priya/skiff-cpu", + "categories": ["branch", "rejected-alternative"], + "question": "Main has an open question about cheaper tide steps for distant cells, with one proposed answer. Which open branch adds a rival answer instead of accepting that one? Give its draft PR number.", + "answer": { + "kind": "value", + "value": "48", + "accept": ["#48"] + }, + "evidence": [ + "world.iss-tide-lod-q", + "world.dec-tide-lod-freeze", + "world.dec-tide-lod-quarter" + ], + "path": [ + ["world.dec-tide-lod-freeze", "derives_from", "world.iss-tide-lod-q"] + ], + "checks": [ + { + "type": "atRef", + "ref": "*", + "check": { + "type": "select", + "effort": "world", + "kinds": ["decision"], + "derivesFrom": "world.iss-tide-lod-q", + "where": { + "state": "accepted" + } + }, + "equals": ["world.dec-tide-lod-freeze"], + "report": "pr" + } + ], + "traps": [ + "kenji/tide-lod (#46) accepts the answer main already proposed." + ] + }, + { + "id": "branch-008", + "tier": 1, + "hops": 1, + "categories": ["branch", "retraction"], + "question": "An open branch retracts a measurement that main's Skiff tide-storm blocker rests on. Which record is it?", + "answer": { + "kind": "record", + "value": "ports.fnd-storm-capture" + }, + "evidence": [ + "ports.fnd-storm-capture", + "ports.iss-blk-storm", + "ports.fnd-storm-recapture" + ], + "path": [ + ["ports.iss-blk-storm", "derives_from", "ports.fnd-storm-capture"] + ], + "checks": [ + { + "type": "refDiff", + "ref": "*", + "change": "retracted" + }, + { + "type": "edge", + "edge": [ + "ports.iss-blk-storm", + "derives_from", + "ports.fnd-storm-capture" + ], + "expect": true + } + ], + "traps": [ + "The retail recapture that replaces it exists only on the branch and is not retracted." + ] + }, + { + "id": "branch-009", + "tier": 0, + "hops": 0, + "categories": ["branch", "lookup", "retraction"], + "question": "On the branch that caps tide-storm particles on Skiff, what peak CPU frame time does the new storm capture measure in the 40 Hz mode?", + "answer": { + "kind": "value", + "value": "27.0 ms", + "accept": ["27 ms"] + }, + "evidence": ["ports.fnd-storm-recapture", "ports.fnd-storm-capture"], + "checks": [ + { + "type": "atRef", + "ref": "priya/skiff-cpu", + "check": { + "type": "select", + "effort": "ports", + "kinds": ["finding"], + "live": true, + "bodyIncludes": "27.0 ms" + }, + "expect": ["ports.fnd-storm-recapture"] + }, + { + "type": "atRef", + "ref": "priya/skiff-cpu", + "check": { + "type": "field", + "record": "ports.fnd-storm-capture", + "field": "standing" + }, + "expect": "retracted" + } + ], + "traps": [ + "Main's capture says 28.4 ms; the branch retracts it because the particle debug overlay was on." + ] + }, + { + "id": "branch-010", + "tier": 4, + "hops": 0, + "categories": ["branch", "compaction", "aggregate"], + "question": "Which records are new summaries (compactions) written on open branches?", + "answer": { + "kind": "records", + "value": ["render.cmp-5", "world.cmp-5"] + }, + "evidence": [ + "render.cmp-5", + "world.cmp-5", + "render.cmp-4", + "world.cmp-4" + ], + "checks": [ + { + "type": "refDiff", + "ref": "*", + "change": "added", + "where": { + "kind": "compaction" + } + }, + { + "type": "conflicts", + "refs": ["ines/vsm-pool", "kenji/tide-lod"], + "kind": "compaction-fork", + "expect": [] + } + ], + "traps": [ + "Main's latest summaries are older; each branch supersedes its own Effort's head." + ] + }, + { + "id": "branch-011", + "tier": 4, + "hops": 0, + "categories": ["branch", "negative"], + "question": "Does any open branch change how the streaming pool picks cells to evict?", + "answer": { + "kind": "boolean", + "value": false + }, + "evidence": ["world.dec-pool-forecast", "world.fnd-impl-pool-score-2"], + "checks": [ + { + "type": "changedPaths", + "ref": "*", + "under": "engine/kestrel/streaming", + "count": true + }, + { + "type": "atRef", + "ref": "*", + "check": { + "type": "field", + "record": "world.dec-pool-forecast", + "field": "standing" + }, + "equals": "accepted", + "expect": [ + "ines/vsm-pool", + "kenji/tide-lod", + "priya/skiff-cpu", + "tomasz/vsm-evict" + ] + } + ], + "traps": [ + "The one-turn eviction fix is already on main, not on a branch." + ] + }, + { + "id": "branch-012", + "tier": 1, + "hops": 1, + "pathRef": "ines/vsm-pool", + "categories": ["branch", "supersession"], + "question": "If ines/vsm-pool merged as it stands, what would the state of main's current sun-shadow Decision become?", + "answer": { + "kind": "value", + "value": "superseded" + }, + "evidence": ["render.dec-vsm", "render.dec-vsm-grow"], + "path": [["render.dec-vsm", "superseded_by", "render.dec-vsm-grow"]], + "checks": [ + { + "type": "atRef", + "ref": "ines/vsm-pool", + "check": { + "type": "field", + "record": "render.dec-vsm", + "field": "state" + } + }, + { + "type": "atRef", + "ref": "main", + "check": { + "type": "field", + "record": "render.dec-vsm", + "field": "state" + }, + "expect": "accepted" + } + ], + "traps": ["It is accepted on main today."] + }, + { + "id": "branch-013", + "tier": 0, + "hops": 0, + "categories": ["branch", "lifecycle"], + "question": "Which open branch accepts the far-cell tide Decision that main already proposed? Give the branch name.", + "answer": { + "kind": "value", + "value": "kenji/tide-lod" + }, + "evidence": ["world.dec-tide-lod-quarter", "world.iss-tide-lod-q"], + "checks": [ + { + "type": "atRef", + "ref": "*", + "check": { + "type": "field", + "record": "world.dec-tide-lod-quarter", + "field": "standing" + }, + "equals": "accepted" + } + ], + "traps": ["priya/skiff-cpu rejects that Decision by accepting a rival."] + }, + { + "id": "branch-014", + "tier": 0, + "hops": 0, + "categories": ["branch", "lookup"], + "question": "Who opened the draft PR that freezes far-off tide cells instead of stepping them?", + "answer": { + "kind": "value", + "value": "Priya Natarajan", + "accept": ["priya-natarajan"] + }, + "evidence": ["world.dec-tide-lod-freeze"], + "checks": [ + { + "type": "atRef", + "ref": "*", + "check": { + "type": "field", + "record": "world.dec-tide-lod-freeze", + "field": "state" + }, + "equals": "accepted", + "report": "author" + } + ], + "traps": [ + "The question and the proposal on main came from Rhea Valdés and Kenji Arakawa." + ] + }, + { + "id": "branch-015", + "tier": 4, + "hops": 0, + "categories": ["branch", "aggregate"], + "question": "How many records that already exist on main does kenji/tide-lod change the standing of?", + "answer": { + "kind": "count", + "value": 3 + }, + "evidence": [ + "world.dec-tide-lod-quarter", + "world.iss-tide-lod-q", + "world.cmp-4", + "world.cmp-5" + ], + "checks": [ + { + "type": "refDiff", + "ref": "kenji/tide-lod", + "change": "restanded", + "count": true + }, + { + "type": "refDiff", + "ref": "kenji/tide-lod", + "change": "restanded", + "expect": [ + "world.cmp-4", + "world.dec-tide-lod-quarter", + "world.iss-tide-lod-q" + ] + } + ], + "traps": [ + "The branch's new summary supersedes main's world summary, which counts.", + "The new summary itself is added, not changed." + ] + }, + { + "id": "branch-016", + "tier": 0, + "hops": 0, + "categories": ["branch", "current-state"], + "question": "On the branch that caps tide-storm particles on Skiff, what state is the proposal to drop to 30 Hz during storms left in?", + "answer": { + "kind": "value", + "value": "proposed" + }, + "evidence": [ + "ports.dec-storm-30", + "ports.dec-storm-cap", + "ports.iss-blk-storm" + ], + "checks": [ + { + "type": "atRef", + "ref": "priya/skiff-cpu", + "check": { + "type": "field", + "record": "ports.dec-storm-30", + "field": "state" + } + }, + { + "type": "atRef", + "ref": "priya/skiff-cpu", + "check": { + "type": "field", + "record": "ports.dec-storm-cap", + "field": "state" + }, + "expect": "accepted" + } + ], + "traps": [ + "The storm blocker is not a question Issue, so accepting the particle cap did not reject the 30 Hz option." + ] + }, + { + "id": "branch-017", + "tier": 0, + "hops": 0, + "categories": ["branch", "lookup"], + "question": "Two open branches both add engine/kestrel/tides/lod.rs. Which one keeps far cells stepping instead of freezing them? Give its draft PR number.", + "answer": { + "kind": "value", + "value": "46", + "accept": ["#46"] + }, + "evidence": ["world.dec-tide-lod-quarter", "world.dec-tide-lod-freeze"], + "checks": [ + { + "type": "atRef", + "ref": "*", + "check": { + "type": "select", + "effort": "world", + "kinds": ["decision"], + "where": { + "state": "accepted" + }, + "titleIncludes": "quarter" + }, + "equals": ["world.dec-tide-lod-quarter"], + "report": "pr" + }, + { + "type": "changedPaths", + "ref": "*", + "under": "engine/kestrel/tides", + "expect": [ + "engine/kestrel/tides/lod.rs", + "engine/kestrel/tides/mod.rs" + ] + } + ], + "traps": ["priya/skiff-cpu (#48) freezes them."] + }, + { + "id": "branch-018", + "tier": 1, + "hops": 1, + "pathRef": "ines/vsm-pool", + "categories": ["branch", "compaction"], + "question": "Which accepted sun-shadow Decision does the new Lantern summary on an open branch rest on?", + "answer": { + "kind": "record", + "value": "render.dec-vsm-grow" + }, + "evidence": [ + "render.cmp-5", + "render.dec-vsm-grow", + "render.cmp-4", + "render.dec-vsm" + ], + "path": [["render.cmp-5", "derives_from", "render.dec-vsm-grow"]], + "checks": [ + { + "type": "atRef", + "ref": "ines/vsm-pool", + "check": { + "type": "select", + "effort": "render", + "kinds": ["decision"], + "live": true, + "titleIncludes": "virtual shadow maps for the sun" + } + }, + { + "type": "atRef", + "ref": "ines/vsm-pool", + "check": { + "type": "select", + "effort": "render", + "kinds": ["decision"], + "where": { "state": "accepted" }, + "titleIncludes": ["sun", "shadow"] + }, + "expect": ["render.dec-vsm-grow"] + }, + { + "type": "atRef", + "ref": "ines/vsm-pool", + "check": { + "type": "edge", + "edge": ["render.cmp-5", "derives_from", "render.dec-vsm-grow"] + }, + "expect": true + } + ], + "traps": [ + "Main's Lantern summary lists the original sun-shadow Decision." + ] + }, + { + "id": "branch-019", + "tier": 4, + "hops": 0, + "categories": ["branch", "negative"], + "question": "Does any open branch change code under tools/shadercache/?", + "answer": { + "kind": "boolean", + "value": false + }, + "evidence": ["tools.dec-key-v3"], + "checks": [ + { + "type": "changedPaths", + "ref": "*", + "under": "tools/shadercache", + "count": true + }, + { + "type": "refDiff", + "ref": "*", + "change": "changed", + "effort": "tools", + "expect": [] + }, + { + "type": "refDiff", + "ref": "*", + "change": "added", + "effort": "tools", + "expect": [] + } + ], + "traps": [ + "Both shader caches are called the shader cache; the Skiff pipeline cache is not under tools/ and no branch changes it either." + ] + }, + { + "id": "branch-020", + "tier": 4, + "hops": 0, + "categories": ["branch", "supersession", "aggregate"], + "question": "I'm about to build on main's accepted sun-shadow Decision. Which open branches would replace it? Give the branch names.", + "answer": { + "kind": "text", + "rubric": [["ines/vsm-pool"], ["tomasz/vsm-evict"]] + }, + "evidence": [ + "render.dec-vsm", + "render.dec-vsm-grow", + "render.dec-vsm-coarse" + ], + "checks": [ + { + "type": "atRef", + "ref": "*", + "check": { + "type": "field", + "record": "render.dec-vsm", + "field": "standing" + }, + "equals": "superseded", + "expect": ["ines/vsm-pool", "tomasz/vsm-evict"] + } + ], + "traps": [ + "Only one of the two accepts its replacement; both supersede the Decision." + ] + }, + { + "id": "branch-021", + "tier": 1, + "hops": 1, + "pathRef": "priya/skiff-cpu", + "categories": ["branch", "blocker"], + "question": "If priya/skiff-cpu merged, which Decision would resolve main's open Skiff tide-storm blocker?", + "answer": { + "kind": "record", + "value": "ports.dec-storm-cap" + }, + "evidence": [ + "ports.iss-blk-storm", + "ports.dec-storm-cap", + "ports.dec-storm-30" + ], + "path": [["ports.iss-blk-storm", "resolved_by", "ports.dec-storm-cap"]], + "checks": [ + { + "type": "atRef", + "ref": "priya/skiff-cpu", + "check": { + "type": "select", + "effort": "ports", + "kinds": ["decision"], + "where": { + "state": "accepted" + }, + "titleIncludes": "storm" + } + }, + { + "type": "atRef", + "ref": "priya/skiff-cpu", + "check": { + "type": "edge", + "edge": [ + "ports.iss-blk-storm", + "resolved_by", + "ports.dec-storm-cap" + ] + }, + "expect": true + } + ], + "traps": ["The 30 Hz storm fallback stays proposed on the branch."] + }, + { + "id": "branch-022", + "tier": 4, + "hops": 0, + "categories": ["branch", "aggregate"], + "question": "Which open branch changes the standing of the most records that already exist on main? Give the branch name.", + "answer": { + "kind": "value", + "value": "priya/skiff-cpu" + }, + "evidence": [ + "ports.fnd-storm-capture", + "ports.dec-storm-cap", + "ports.iss-blk-storm", + "world.dec-tide-lod-quarter", + "world.iss-tide-lod-q" + ], + "checks": [ + { + "type": "refDiff", + "ref": "priya/skiff-cpu", + "change": "restanded", + "count": true, + "expect": 5 + }, + { + "type": "refDiff", + "ref": "kenji/tide-lod", + "change": "restanded", + "count": true, + "expect": 3 + }, + { + "type": "refDiff", + "ref": "ines/vsm-pool", + "change": "restanded", + "count": true, + "expect": 3 + }, + { + "type": "refDiff", + "ref": "tomasz/vsm-evict", + "change": "restanded", + "count": true, + "expect": 1 + } + ], + "traps": [ + "A record the branch adds does not count; its retraction of the storm capture does." + ] + } + ] +} diff --git a/benchmarks/proof-recall/questions/code.json b/benchmarks/proof-recall/questions/code.json new file mode 100644 index 00000000..e2b09394 --- /dev/null +++ b/benchmarks/proof-recall/questions/code.json @@ -0,0 +1,777 @@ +{ + "effort": "code", + "questions": [ + { + "id": "code-001", + "tier": 0, + "hops": 0, + "categories": ["code-context", "current-state"], + "question": "Before I edit engine/lantern/shadows/vsm_pool.rs, which live Decision on main governs how the shadow page pool is sized and evicted?", + "answer": { + "kind": "record", + "value": "render.dec-vsm" + }, + "evidence": [ + "render.dec-vsm", + "render.fnd-impl-vsm-evict", + "render.cit-code-vsm-evict" + ], + "checks": [ + { + "type": "trailer", + "path": "engine/lantern/shadows/vsm_pool.rs", + "kinds": ["decision"], + "live": true + }, + { + "type": "codeAnchor", + "path": "engine/lantern/shadows/vsm_pool.rs", + "follow": "derives_from", + "kinds": ["decision"], + "live": true + } + ], + "traps": [ + "Two open branches, ines/vsm-pool and tomasz/vsm-evict, each supersede this Decision, but neither is merged; on main it still holds.", + "The cascaded shadow map Decision it replaced is superseded." + ] + }, + { + "id": "code-002", + "tier": 1, + "hops": 1, + "categories": ["code-context", "supersession", "current-state"], + "question": "Before I change how tools/shadercache/key.py builds the cache key, which Decision about the key is in force?", + "answer": { + "kind": "record", + "value": "tools.dec-key-v3" + }, + "evidence": [ + "tools.dec-key-v2", + "tools.dec-key-v3", + "tools.dec-key-v1", + "tools.fnd-impl-cache-key", + "tools.cit-code-cache-key" + ], + "path": [["tools.dec-key-v2", "superseded_by", "tools.dec-key-v3"]], + "checks": [ + { + "type": "trailer", + "path": "tools/shadercache/key.py", + "resolve": "head" + }, + { + "type": "trailer", + "path": "tools/shadercache/key.py", + "expect": ["tools.dec-key-v1", "tools.dec-key-v2"] + }, + { + "type": "trailer", + "path": "tools/shadercache/key.py", + "commit": "latest", + "expect": [] + }, + { + "type": "codeAnchor", + "path": "tools/shadercache/key.py", + "follow": "derives_from", + "kinds": ["decision"], + "live": true + } + ], + "traps": [ + "The trailers on this file name only the first two key Decisions, both superseded. The latest commit, which hashes the compiler binary, has no trailer.", + "The rejected modification-time key was never in force." + ] + }, + { + "id": "code-003", + "tier": 4, + "hops": 1, + "categories": ["code-context", "aggregate"], + "question": "Which Findings or Risks rest on the code anchor for the commit that added forecast-based eviction to engine/kestrel/streaming/pool.rs?", + "answer": { + "kind": "records", + "value": ["world.fnd-impl-pool-score", "world.rsk-pool-score-cost"] + }, + "evidence": [ + "world.cit-code-pool-score", + "world.fnd-impl-pool-score", + "world.rsk-pool-score-cost", + "world.cit-code-pool-score-2", + "world.fnd-impl-pool-score-2" + ], + "checks": [ + { + "type": "codeAnchor", + "path": "engine/kestrel/streaming/pool.rs", + "symbol": "evict_score", + "commit": "introducing" + }, + { + "type": "codeAnchor", + "path": "engine/kestrel/streaming/pool.rs", + "symbol": "evict_score", + "commit": "introducing", + "citations": true, + "expect": ["world.cit-code-pool-score"] + } + ], + "traps": [ + "The later one-turn fix has its own code anchor and note; those cite the fix commit, not the one that added forecast eviction.", + "The commit that first added the file brought least-recently-used eviction, not the forecast score." + ] + }, + { + "id": "code-004", + "tier": 1, + "hops": 1, + "categories": ["code-context", "supersession"], + "question": "Of the records that cite the commit that added forecast-based eviction to engine/kestrel/streaming/pool.rs, which one has since been superseded?", + "answer": { + "kind": "record", + "value": "world.fnd-impl-pool-score" + }, + "evidence": [ + "world.cit-code-pool-score", + "world.fnd-impl-pool-score", + "world.fnd-impl-pool-score-2", + "world.rsk-pool-score-cost" + ], + "path": [ + ["world.cit-code-pool-score", "cited_by", "world.fnd-impl-pool-score"] + ], + "checks": [ + { + "type": "codeAnchor", + "path": "engine/kestrel/streaming/pool.rs", + "symbol": "evict_score", + "commit": "introducing", + "where": { + "standing": "superseded" + } + }, + { + "type": "head", + "start": "world.fnd-impl-pool-score", + "expect": "world.fnd-impl-pool-score-2" + } + ], + "traps": [ + "The eviction-cost Risk cites the same commit and is still open.", + "The note that replaced it cites the fix commit and is live." + ] + }, + { + "id": "code-005", + "tier": 0, + "hops": 0, + "categories": ["code-context", "invalidation"], + "question": "Before I edit engine/kestrel/streaming/io_queue.rs, is the Decision behind push_tide_read, which lets tide reads jump the queue, still sound?", + "answer": { + "kind": "boolean", + "value": false + }, + "evidence": [ + "world.dec-io-tide-priority", + "world.fnd-io-starve", + "world.fnd-impl-tide-push", + "world.cit-code-tide-push" + ], + "checks": [ + { + "type": "trailer", + "path": "engine/kestrel/streaming/io_queue.rs", + "symbol": "push_tide_read", + "commit": "introducing", + "expect": ["world.dec-io-tide-priority"] + }, + { + "type": "codeAnchor", + "path": "engine/kestrel/streaming/io_queue.rs", + "symbol": "push_tide_read", + "follow": "derives_from", + "expect": ["world.dec-io-tide-priority"] + }, + { + "type": "field", + "record": "world.dec-io-tide-priority", + "field": "standing", + "expect": "invalidated" + } + ], + "traps": [ + "The single-request-queue Decision the file started from is still accepted, but it is not the one behind push_tide_read." + ] + }, + { + "id": "code-006", + "tier": 1, + "hops": 1, + "categories": ["code-context", "invalidation"], + "question": "Which Finding showed that the tide-read ordering in engine/kestrel/streaming/io_queue.rs is wrong?", + "answer": { + "kind": "record", + "value": "world.fnd-io-starve" + }, + "evidence": [ + "world.dec-io-tide-priority", + "world.fnd-io-starve", + "world.fnd-impl-tide-push" + ], + "path": [ + ["world.dec-io-tide-priority", "invalidated_by", "world.fnd-io-starve"] + ], + "checks": [ + { + "type": "select", + "effort": "world", + "kinds": ["finding"], + "invalidates": "world.dec-io-tide-priority" + }, + { + "type": "trailer", + "path": "engine/kestrel/streaming/io_queue.rs", + "symbol": "push_tide_read", + "expect": ["world.dec-io-tide-priority"] + } + ], + "traps": [ + "The implementation note on push_tide_read describes the code; it does not show the rule wrong." + ] + }, + { + "id": "code-007", + "tier": 0, + "hops": 0, + "categories": ["code-context", "lookup"], + "question": "Before I change how many shader compiles tools/shadercache/remote.py runs at once on one farm agent, which Decision sets that limit?", + "answer": { + "kind": "record", + "value": "tools.dec-cap-jobs" + }, + "evidence": ["tools.dec-cap-jobs", "tools.dec-remote"], + "checks": [ + { + "type": "trailer", + "path": "tools/shadercache/remote.py", + "titleIncludes": "parallel" + }, + { + "type": "trailer", + "path": "tools/shadercache/remote.py", + "expect": ["tools.dec-cap-jobs", "tools.dec-remote"] + } + ], + "traps": [ + "The same commit also names the remote-execution Decision, which fans misses out but sets no per-agent limit." + ] + }, + { + "id": "code-008", + "tier": 0, + "hops": 0, + "categories": ["code-context", "lookup"], + "question": "Which open defect names tools/shadercache/remote.py?", + "answer": { + "kind": "record", + "value": "tools.iss-remote-retry" + }, + "evidence": ["tools.iss-remote-retry", "tools.iss-flaky"], + "checks": [ + { + "type": "select", + "effort": "*", + "kinds": ["issue"], + "where": { + "status": "open" + }, + "bodyIncludes": "tools/shadercache/remote.py" + } + ], + "traps": ["The nightly timeout defect is resolved and names no file."] + }, + { + "id": "code-009", + "tier": 0, + "hops": 0, + "categories": ["code-context", "current-state"], + "question": "Before I touch platform/skiff/dynres.rs: by the Decision in force, below what resolution does Skiff's 30 Hz mode rarely drop?", + "answer": { + "kind": "value", + "value": "600p", + "accept": ["600"] + }, + "evidence": [ + "ports.dec-dynres-3", + "ports.dec-dynres-2", + "ports.dec-dynres-1" + ], + "checks": [ + { + "type": "trailer", + "path": "platform/skiff/dynres.rs", + "expect": ["ports.dec-dynres-3"] + }, + { + "type": "body", + "record": "ports.dec-dynres-3", + "includes": "The 30 Hz mode rarely drops below 600p.", + "expect": true + } + ], + "traps": [ + "The code and the commit subject give only the 504p floor and the 720p ceiling; only the Decision says how low the 30 Hz mode usually goes.", + "Earlier versions set 540p and then 576p floors; both are superseded." + ] + }, + { + "id": "code-010", + "tier": 0, + "hops": 0, + "categories": ["code-context", "lookup"], + "question": "Is there an open defect about how platform/skiff/dynres.rs recovers after a single slow frame? Which record is it?", + "answer": { + "kind": "record", + "value": "ports.iss-dynres-hold" + }, + "evidence": ["ports.iss-dynres-hold", "ports.dec-dynres-3"], + "checks": [ + { + "type": "select", + "effort": "*", + "kinds": ["issue"], + "live": true, + "bodyIncludes": "platform/skiff/dynres.rs" + } + ], + "traps": [ + "The dynamic resolution question Issue is resolved and is not a defect." + ] + }, + { + "id": "code-011", + "tier": 2, + "hops": 3, + "categories": ["code-context", "supersession", "current-state"], + "question": "The commits on engine/lantern/vb/resolve.rs name one visibility-buffer Decision. Which version of that Decision is in force now?", + "answer": { + "kind": "record", + "value": "render.dec-vb-6" + }, + "evidence": [ + "render.dec-vb-3", + "render.dec-vb-4", + "render.dec-vb-5", + "render.dec-vb-6" + ], + "path": [ + ["render.dec-vb-3", "superseded_by", "render.dec-vb-4"], + ["render.dec-vb-4", "superseded_by", "render.dec-vb-5"], + ["render.dec-vb-5", "superseded_by", "render.dec-vb-6"] + ], + "checks": [ + { + "type": "trailer", + "path": "engine/lantern/vb/resolve.rs", + "resolve": "head" + }, + { + "type": "trailer", + "path": "engine/lantern/vb/resolve.rs", + "expect": ["render.dec-vb-3"] + }, + { + "type": "trailer", + "path": "engine/lantern/vb/resolve.rs", + "commit": "latest", + "expect": [] + } + ], + "traps": [ + "Only the first commit has a trailer, and it names the third version, which is three replacements old.", + "The uber-shader resolve Decision governs how materials are shaded, not which version of the visibility buffer is in force." + ] + }, + { + "id": "code-012", + "tier": 0, + "hops": 0, + "categories": ["code-context", "supersession"], + "question": "Before I edit engine/lantern/tern/presets.rs, which Decision sets the presets it must offer?", + "answer": { + "kind": "record", + "value": "render.dec-presets-4b" + }, + "evidence": [ + "render.dec-presets-4b", + "render.dec-presets-3", + "render.dec-presets-4" + ], + "checks": [ + { + "type": "trailer", + "path": "engine/lantern/tern/presets.rs", + "resolve": "head" + }, + { + "type": "trailer", + "path": "engine/lantern/tern/presets.rs", + "expect": ["render.dec-presets-3", "render.dec-presets-4b"] + } + ], + "traps": [ + "The first commit names the three-preset Decision, which is superseded.", + "A rejected four-preset Decision has nearly the same title." + ] + }, + { + "id": "code-013", + "tier": 4, + "hops": 0, + "categories": ["code-context", "aggregate"], + "question": "Which Decisions do the commits on main under engine/kestrel/tides/ give as their reason?", + "answer": { + "kind": "records", + "value": [ + "world.dec-tide-fixed", + "world.dec-tide-harmonic", + "world.dec-tide-wrap" + ] + }, + "evidence": [ + "world.dec-tide-fixed", + "world.dec-tide-harmonic", + "world.dec-tide-wrap", + "world.con-tide-step-alloc", + "world.dec-tide-lod-quarter" + ], + "checks": [ + { + "type": "trailer", + "path": "engine/kestrel/tides", + "kinds": ["decision"] + }, + { + "type": "trailer", + "path": "engine/kestrel/tides", + "expect": [ + "world.dec-tide-fixed", + "world.dec-tide-harmonic", + "world.dec-tide-wrap" + ] + } + ], + "traps": [ + "A Constraint names engine/kestrel/tides/fixed.rs in its body, but no commit names it.", + "The far-cell tide Decision is only proposed on main, and only branches add code for it." + ] + }, + { + "id": "code-014", + "tier": 0, + "hops": 0, + "categories": ["code-context", "lookup"], + "question": "Which hard Constraint names engine/kestrel/tides/fixed.rs?", + "answer": { + "kind": "record", + "value": "world.con-tide-step-alloc" + }, + "evidence": ["world.con-tide-step-alloc", "world.con-tide-bitexact"], + "checks": [ + { + "type": "select", + "effort": "*", + "kinds": ["constraint"], + "where": { + "kind": "hard" + }, + "bodyIncludes": "engine/kestrel/tides/fixed.rs" + } + ], + "traps": [ + "The bit-identical tide Constraint governs the same code but names no file." + ] + }, + { + "id": "code-015", + "tier": 1, + "hops": 1, + "categories": ["code-context", "lookup"], + "question": "Before I change how engine/kestrel/save/upgrade.rs adds upgrade steps, which hard Constraint must the change keep?", + "answer": { + "kind": "record", + "value": "world.con-save-compat" + }, + "evidence": [ + "world.dec-save-version", + "world.con-save-compat", + "world.fnd-impl-save-upgrade" + ], + "path": [ + ["world.dec-save-version", "derives_from", "world.con-save-compat"] + ], + "checks": [ + { + "type": "codeAnchor", + "path": "engine/kestrel/save/upgrade.rs", + "follow": "derives_from", + "kinds": ["constraint"] + }, + { + "type": "trailer", + "path": "engine/kestrel/save/upgrade.rs", + "follow": "derives_from", + "kinds": ["constraint"] + } + ], + "traps": [ + "The no-allocation Constraint is about the tide step, not saves." + ] + }, + { + "id": "code-016", + "tier": 0, + "hops": 0, + "categories": ["code-context", "lookup"], + "question": "According to its Proof notes, does the reactive-mask code in engine/lantern/tern/history.rs do anything for the caustic flicker blocker?", + "answer": { + "kind": "boolean", + "value": false + }, + "evidence": [ + "render.fnd-impl-tern-reactive", + "render.cit-code-tern-reactive", + "render.iss-caustic-flicker" + ], + "checks": [ + { + "type": "codeAnchor", + "path": "engine/lantern/tern/history.rs", + "expect": ["render.fnd-impl-tern-reactive"] + }, + { + "type": "body", + "record": "render.fnd-impl-tern-reactive", + "includes": "does nothing for the caustic flicker blocker", + "expect": true + } + ], + "traps": [ + "The reactive mask fixed Tern ghosting on saltglass, which makes it look like a fix for caustics too." + ] + }, + { + "id": "code-017", + "tier": 0, + "hops": 0, + "categories": ["code-context", "lookup"], + "question": "Before I edit net/coop/checksum.rs: according to the Decision behind it, what does a checksum mismatch tell you?", + "answer": { + "kind": "text", + "rubric": [ + ["block"], + ["frame window", "window of frames", "frames it happened"] + ] + }, + "evidence": [ + "netcode.dec-checksum", + "netcode.dec-host", + "netcode.fnd-impl-checksum" + ], + "checks": [ + { + "type": "trailer", + "path": "net/coop/checksum.rs", + "titleIncludes": "Hash", + "expect": ["netcode.dec-checksum"] + }, + { + "type": "trailer", + "path": "net/coop/checksum.rs", + "expect": ["netcode.dec-checksum", "netcode.dec-host"] + }, + { + "type": "body", + "record": "netcode.dec-checksum", + "includes": "A mismatch names the block that diverged and the frame window it happened in.", + "expect": true + } + ], + "traps": [ + "The code and the commit subject say how often state is hashed, but not what a mismatch reports.", + "The same commit names the host-authoritative sync Decision." + ] + }, + { + "id": "code-018", + "tier": 0, + "hops": 0, + "categories": ["code-context", "negative"], + "question": "Before I edit engine/kestrel/ecs/names.rs, which Decision governs it?", + "answer": { + "kind": "value", + "value": "unknown" + }, + "evidence": ["world.iss-ecs-names", "world.dec-ecs-hybrid"], + "checks": [ + { + "type": "trailer", + "path": "engine/kestrel/ecs/names.rs", + "expect": [] + }, + { + "type": "codeAnchor", + "path": "engine/kestrel/ecs/names.rs", + "expect": [] + }, + { + "type": "select", + "effort": "*", + "bodyIncludes": "ecs/names.rs", + "expect": [] + } + ], + "traps": [ + "An Issue about missing entity debug names exists, but it is an Issue closed as wontfix, not a Decision, and names no file.", + "The ECS storage Decision governs component storage, not debug names." + ] + }, + { + "id": "code-019", + "tier": 1, + "hops": 1, + "categories": ["code-context", "supersession"], + "question": "Before I change engine/lantern/tonemap/brine.rs, at how many nits should the HDR shoulder start?", + "answer": { + "kind": "value", + "value": "800 nits", + "accept": ["800"] + }, + "evidence": ["render.dec-brine-2", "render.dec-brine-3"], + "path": [["render.dec-brine-2", "superseded_by", "render.dec-brine-3"]], + "checks": [ + { + "type": "trailer", + "path": "engine/lantern/tonemap/brine.rs", + "resolve": "head", + "expect": ["render.dec-brine-3"] + }, + { + "type": "trailer", + "path": "engine/lantern/tonemap/brine.rs", + "expect": ["render.dec-brine-2"] + }, + { + "type": "body", + "record": "render.dec-brine-3", + "includes": "800 nits", + "expect": true + } + ], + "traps": [ + "The code still starts the shoulder at 1,000 nits, and so does the only commit with a trailer, which names the superseded 1,000-nit Decision. The newest commit on the file has no trailer and leaves the shoulder alone." + ] + }, + { + "id": "code-020", + "tier": 1, + "hops": 1, + "categories": ["code-context", "supersession"], + "question": "The cache in platform/skiff/pso_cache.rs replaced an earlier plan for avoiding first-use pipeline compiles. Which superseded Decision was that plan?", + "answer": { + "kind": "record", + "value": "ports.dec-pso-ship" + }, + "evidence": [ + "ports.dec-pso-ondevice", + "ports.dec-pso-ship", + "ports.dec-pso-uber", + "ports.fnd-impl-pso-warm" + ], + "path": [["ports.dec-pso-ondevice", "supersedes", "ports.dec-pso-ship"]], + "checks": [ + { + "type": "trailer", + "path": "platform/skiff/pso_cache.rs", + "follow": "supersedes" + }, + { + "type": "codeAnchor", + "path": "platform/skiff/pso_cache.rs", + "follow": "derives_from", + "expect": ["ports.dec-pso-ondevice"] + } + ], + "traps": [ + "The ubershader option was rejected, not superseded.", + "The build farm shader compile cache is a different cache." + ] + }, + { + "id": "code-021", + "tier": 3, + "hops": 3, + "categories": ["code-context", "blocker"], + "question": "Before I edit engine/lantern/shadows/vsm_pool.rs, which open blocker on main is about the pool that file manages?", + "answer": { + "kind": "record", + "value": "render.iss-vsm-tide" + }, + "evidence": [ + "render.dec-vsm", + "render.rsk-vsm-overflow", + "render.fnd-vsm-overflow", + "render.iss-vsm-tide", + "render.fnd-impl-vsm-evict" + ], + "path": [ + ["render.dec-vsm", "supports", "render.rsk-vsm-overflow"], + ["render.rsk-vsm-overflow", "supports", "render.fnd-vsm-overflow"], + ["render.fnd-vsm-overflow", "supports", "render.iss-vsm-tide"] + ], + "checks": [ + { + "type": "select", + "effort": "render", + "kinds": ["issue"], + "where": { + "status": "open", + "kind": "blocker" + }, + "derivesFrom": "render.fnd-vsm-overflow" + }, + { + "type": "trailer", + "path": "engine/lantern/shadows/vsm_pool.rs", + "expect": ["render.dec-vsm"] + } + ], + "traps": [ + "ines/vsm-pool resolves this blocker, but only on that branch; on main it is open." + ] + }, + { + "id": "code-022", + "tier": 0, + "hops": 0, + "categories": ["code-context", "lookup"], + "question": "Which record says what evict_pages in engine/lantern/shadows/vsm_pool.rs never evicts?", + "answer": { + "kind": "record", + "value": "render.fnd-impl-vsm-evict" + }, + "evidence": ["render.fnd-impl-vsm-evict", "render.cit-code-vsm-evict"], + "checks": [ + { + "type": "codeAnchor", + "path": "engine/lantern/shadows/vsm_pool.rs", + "symbol": "evict_pages" + } + ], + "traps": [ + "The sun-shadow Decision describes the moving-object page set, but not what eviction skips." + ] + } + ] +} diff --git a/benchmarks/proof-recall/scripts/build-repo.mjs b/benchmarks/proof-recall/scripts/build-repo.mjs new file mode 100644 index 00000000..40ab97c7 --- /dev/null +++ b/benchmarks/proof-recall/scripts/build-repo.mjs @@ -0,0 +1,293 @@ +#!/usr/bin/env node +// Builds the git test bed for the code-context and branch questions from +// spec/repo.mjs: a small codebase on main with `Proof:` commit trailers, the +// committed 1x graph, code anchors written through the real Proof writer, +// and four engineer branches, each with a draft PR. +// +// node scripts/build-repo.mjs --out /tmp/p2-repo +// +// It writes: +// / the repository, with main checked out +// /.flatbread/proof/prs.json +// gitignored; each branch's draft PR, in the +// shape `gh pr list --json` prints +// .keymap.json every benchmark key to its record id, main and +// branch records alike. It sits outside the +// repository so no agent session sees it. +// +// Every commit has a fixed author, date and message, and every record a +// fixed id and time, so two builds give the same commit hashes. +import { execFileSync } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import { cp, mkdir, readFile, rm, writeFile } from 'node:fs/promises'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import { createProofWriter, ProofMutationSchema } from '@flatbread/proof'; +import { benchId } from './build.mjs'; +import spec, { PEOPLE, specCommits } from '../spec/repo.mjs'; + +const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +const REPO_ROOT = resolve(BENCH, '..', '..'); +const GRAPH = join(BENCH, '.flatbread-proof'); +const PROOF_DIR = '.flatbread-proof'; + +const KIND_BY_TYPE = { + WriteIssue: 'issue', + WriteFinding: 'finding', + WriteCompaction: 'finding', + WriteDecision: 'decision', + WriteConstraint: 'constraint', + WriteRisk: 'risk', + WriteCitation: 'citation', + WriteBlob: 'blob', +}; +// Fields whose values name records; the spec writes `.`. +const REF_FIELDS = [ + 'derives_from', + 'supersedes', + 'invalidates', + 'cites', + 'blob', + 'supersederId', + 'targetId', + 'findingId', + 'issueId', + 'resolvedBy', + 'decisionId', + 'rejects', + 'riskId', + 'evidence', + 'recordId', +]; + +function args(argv) { + const out = { out: null }; + for (let i = 0; i < argv.length; i++) { + if (argv[i] === '--out') out.out = resolve(argv[++i]); + else throw new Error(`Unknown argument ${argv[i]}`); + } + if (!out.out) throw new Error('usage: build-repo.mjs --out '); + // The key map is written next to the repository, so both must stay out + // of this one, where an agent working here could find them. + if (`${out.out}/`.startsWith(`${REPO_ROOT}/`)) + throw new Error('Build the test bed outside this repository'); + return out; +} + +/** Commit hash abbreviation used in anchors and record bodies. */ +const short = (sha) => sha.slice(0, 12); + +/** + * Runs git in `dir` with no user or system config, so a build depends on + * nothing but the spec. `env` adds author and committer fields. + */ +function git(dir, argv, env = {}) { + return execFileSync('git', argv, { + cwd: dir, + encoding: 'utf8', + env: { + PATH: process.env.PATH, + GIT_CONFIG_NOSYSTEM: '1', + GIT_CONFIG_GLOBAL: '/dev/null', + TZ: 'UTC', + ...env, + }, + }).trim(); +} + +export function email(person) { + return `${PEOPLE[person].login}@tidewater.example`; +} + +function checkSpec() { + const commits = specCommits(); + const keys = new Set(); + const subjects = new Set(); + for (const c of commits) { + if (keys.has(c.key)) throw new Error(`duplicate commit key ${c.key}`); + if (subjects.has(c.subject)) + throw new Error(`duplicate commit subject "${c.subject}"`); + if (!PEOPLE[c.author]) throw new Error(`${c.key}: unknown author`); + keys.add(c.key); + subjects.add(c.subject); + } + // Each line of history must move forward in time. + const lines = [ + spec.main, + ...spec.branches.map((b) => [spec.main.at(-1), ...b.commits]), + ]; + for (const line of lines) + for (let i = 1; i < line.length; i++) + if (line[i].at < line[i - 1].at) + throw new Error(`${line[i].key}: commits must be in time order`); +} + +/** Writes or deletes the commit's files in the working tree. */ +async function writeFiles(dir, files = {}) { + for (const [path, text] of Object.entries(files)) { + const target = join(dir, path); + if (text === null) await rm(target, { force: true }); + else { + await mkdir(dirname(target), { recursive: true }); + await writeFile(target, text); + } + } +} + +/** Replays one commit's events through the Proof writer. */ +async function replay(dir, commit, keymap, shas) { + const fill = (text) => + text.replace(/\{\{sha:([\w-]+)\}\}/g, (_, key) => { + if (!shas.has(key)) + throw new Error(`${commit.key}: no commit ${key} yet`); + return short(shas.get(key)); + }); + const resolveRef = (event, key) => { + const id = keymap[key]; + if (!id) + throw new Error( + `${commit.key} ${event.key ?? event.type}: unknown key "${key}"` + ); + return id; + }; + let now = new Date(commit.at); + const writer = createProofWriter({ + rootDir: join(dir, PROOF_DIR), + clock: () => now, + }); + for (const [i, event] of (commit.events ?? []).entries()) { + // Events in one commit are a minute apart, so the records keep order. + now = new Date(new Date(commit.at).getTime() + i * 60_000); + let { key, effort, ...payload } = event; + if (payload.type === 'Anchor') { + const { path, symbol, commit: of, ...rest } = payload; + if (!shas.has(of)) + throw new Error(`${commit.key}: anchor names ${of}, not yet committed`); + payload = { + ...rest, + type: 'WriteCitation', + role: 'code', + body: `${path}#${symbol}@${short(shas.get(of))}`, + }; + } + const kind = KIND_BY_TYPE[payload.type]; + if (payload.body) payload.body = fill(payload.body); + for (const field of REF_FIELDS) { + if (payload[field] === undefined) continue; + payload[field] = Array.isArray(payload[field]) + ? payload[field].map((ref) => resolveRef(event, ref)) + : resolveRef(event, payload[field]); + } + if (kind) { + if (!key || !effort) + throw new Error(`${commit.key}: create events need key and effort`); + const qualified = `${effort}.${key}`; + if (keymap[qualified]) + throw new Error(`${commit.key}: duplicate key ${qualified}`); + payload.effort = resolveRef(event, `${effort}.effort`); + payload.created_at = now.toISOString().replace('.000Z', 'Z'); + payload.id = benchId(kind, payload.title, qualified); + keymap[qualified] = payload.id; + } else if (key || effort) + throw new Error(`${commit.key}: only create events take key and effort`); + try { + await writer.mutate(ProofMutationSchema.parse(payload)); + } catch (error) { + throw new Error(`${commit.key} ${key ?? payload.type}: ${error.message}`); + } + } + // The journal is writer state, not content; it is gitignored as well. + await rm(join(dir, PROOF_DIR, '.journal'), { recursive: true, force: true }); +} + +async function commitOne(dir, commit, keymap, shas) { + await writeFiles(dir, commit.files); + if (commit.graph) + await cp(GRAPH, join(dir, PROOF_DIR), { + recursive: true, + filter: (src) => !src.endsWith('keymap.json'), + }); + await replay(dir, commit, keymap, shas); + const trailers = (commit.proof ?? []).map((key) => { + if (!keymap[key]) + throw new Error(`${commit.key}: trailer names unknown key ${key}`); + return `Proof: ${keymap[key]}`; + }); + const message = [commit.subject, commit.body, trailers.join('\n')] + .filter(Boolean) + .join('\n\n'); + const person = PEOPLE[commit.author]; + git(dir, ['add', '-A']); + git(dir, ['commit', '-q', '--no-verify', '--allow-empty', '-m', message], { + GIT_AUTHOR_NAME: person.name, + GIT_AUTHOR_EMAIL: email(commit.author), + GIT_AUTHOR_DATE: commit.at, + GIT_COMMITTER_NAME: person.name, + GIT_COMMITTER_EMAIL: email(commit.author), + GIT_COMMITTER_DATE: commit.at, + }); + const status = git(dir, ['status', '--porcelain']); + if (status) + throw new Error(`${commit.key}: files left uncommitted:\n${status}`); + const sha = git(dir, ['rev-parse', 'HEAD']); + shas.set(commit.key, sha); + return sha; +} + +async function main() { + const options = args(process.argv.slice(2)); + checkSpec(); + const dir = options.out; + await rm(dir, { recursive: true, force: true }); + await mkdir(dir, { recursive: true }); + git(dir, ['init', '-q', '-b', 'main']); + git(dir, ['config', 'commit.gpgsign', 'false']); + git(dir, ['config', 'core.autocrlf', 'false']); + const keymap = JSON.parse(await readFile(join(GRAPH, 'keymap.json'), 'utf8')); + const shas = new Map(); + for (const commit of spec.main) await commitOne(dir, commit, keymap, shas); + const tip = shas.get(spec.main.at(-1).key); + console.log(`main: ${spec.main.length} commits, tip ${short(tip)}`); + for (const branch of spec.branches) { + git(dir, ['checkout', '-q', '-b', branch.name, tip]); + for (const commit of branch.commits) + await commitOne(dir, commit, keymap, shas); + console.log( + `${branch.name} (draft #${branch.pr}): ${ + branch.commits.length + } commits, tip ${short(shas.get(branch.commits.at(-1).key))}` + ); + git(dir, ['checkout', '-q', 'main']); + } + // What `gh pr list --json number,headRefName,isDraft,author,title` gives. + const prs = spec.branches.map((b) => ({ + number: b.pr, + headRefName: b.name, + isDraft: true, + title: b.title, + author: { login: PEOPLE[b.author].login, name: PEOPLE[b.author].name }, + })); + await mkdir(join(dir, '.flatbread', 'proof'), { recursive: true }); + await writeFile( + join(dir, '.flatbread', 'proof', 'prs.json'), + `${JSON.stringify(prs, null, 2)}\n` + ); + if (git(dir, ['status', '--porcelain'])) + throw new Error('prs.json must be gitignored'); + const sorted = Object.fromEntries( + Object.entries(keymap).sort(([a], [b]) => a.localeCompare(b)) + ); + await writeFile(`${dir}.keymap.json`, `${JSON.stringify(sorted, null, 2)}\n`); + const added = + Object.keys(sorted).length - + Object.keys(JSON.parse(await readFile(join(GRAPH, 'keymap.json'), 'utf8'))) + .length; + console.log(`${added} records added; key map in ${dir}.keymap.json`); + if (!existsSync(join(dir, PROOF_DIR))) throw new Error('no graph on main'); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) + main().catch((error) => { + console.error(error.message); + process.exitCode = 1; + }); diff --git a/benchmarks/proof-recall/scripts/build.mjs b/benchmarks/proof-recall/scripts/build.mjs index 3e1ef4d2..2231a61f 100644 --- a/benchmarks/proof-recall/scripts/build.mjs +++ b/benchmarks/proof-recall/scripts/build.mjs @@ -31,6 +31,7 @@ import { mergeEvents, overlapTerms, } from '../spec/_filler.mjs'; +import { REPO_SETS } from './check.mjs'; const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); const SPEC_DIR = join(BENCH, 'spec'); @@ -103,9 +104,24 @@ function args(argv) { return out; } +/** + * Question files about the built graph. The test-bed sets (REPO_SETS) are + * asked only of build-repo.mjs output, never of a filler graph, so they + * neither constrain filler nor change its redraws. + */ +async function graphQuestionFiles() { + const dir = join(BENCH, 'questions'); + const out = []; + for (const name of (await readdir(dir)).filter((n) => n.endsWith('.json'))) { + const set = JSON.parse(await readFile(join(dir, name), 'utf8')); + if (!REPO_SETS.includes(set.effort)) out.push(name); + } + return out; +} + /** Filler may never name a subsystem a spec or question talks about. */ async function assertFillerIsDisjoint(specs) { - const questions = await readdir(join(BENCH, 'questions')); + const questions = await graphQuestionFiles(); const text = [ ...specs.map((spec) => JSON.stringify(spec)), ...(await Promise.all( @@ -123,7 +139,7 @@ async function assertFillerIsDisjoint(specs) { /** The question sets, and each spec as text, for the overlap checks. */ async function readContent(specs) { const dir = join(BENCH, 'questions'); - const files = (await readdir(dir)).filter((name) => name.endsWith('.json')); + const files = await graphQuestionFiles(); const sets = await Promise.all( files.map(async (name) => JSON.parse(await readFile(join(dir, name), 'utf8')) @@ -242,8 +258,11 @@ export function benchId(kind, title, qualifiedKey) { } export async function loadSpecs(only = []) { + // repo.mjs describes the git test bed (scripts/build-repo.mjs), not an + // Effort. const names = (await readdir(SPEC_DIR)).filter( - (name) => !name.startsWith('_') && !name.startsWith('.') + (name) => + !name.startsWith('_') && !name.startsWith('.') && name !== 'repo.mjs' ); const specs = []; for (const name of names.sort()) { diff --git a/benchmarks/proof-recall/scripts/check.mjs b/benchmarks/proof-recall/scripts/check.mjs index eb820135..eb10770b 100644 --- a/benchmarks/proof-recall/scripts/check.mjs +++ b/benchmarks/proof-recall/scripts/check.mjs @@ -5,13 +5,18 @@ // node scripts/check.mjs all question files // node scripts/check.mjs --only render one question set // node scripts/check.mjs --graph /tmp/x a graph built elsewhere +// node scripts/check.mjs --only code,branch --repo /tmp/p2-repo +// the sets about the git test +// bed from build-repo.mjs // -// This script checks every file, including the opt-in sets in OPT_IN_SETS. -// Other scripts load those only when asked for them by name. +// This script checks every file, including the opt-in sets in OPT_IN_SETS, +// except the sets in REPO_SETS, which need --repo and load only when named. +// Other scripts load opt-in sets only when asked for them by name. import { readdir, readFile } from 'node:fs/promises'; import { dirname, join, resolve } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { isCompaction } from '@flatbread/proof'; +import { openTestBed } from './testbed.mjs'; import { hasEdge, idFor, @@ -60,13 +65,23 @@ export const CATEGORIES = [ // Rewritten so it names no Effort and uses words several Efforts share; // the reader must find the right Effort first. Only in ambiguous.json. 'ambiguous-effort', + // The record that governs a code path, through commit trailers, code + // anchors, or a path named in a body. Only in code.json. + 'code-context', + // The answer exists only on an unmerged branch, or comes from merging two. + // Only in branch.json. + 'branch', ]; // Question files whose `effort` names no Effort: their questions span // several, so every key is qualified as `.`. -export const MULTI_EFFORT_SETS = ['cross', 'ambiguous']; +export const MULTI_EFFORT_SETS = ['cross', 'ambiguous', 'code', 'branch']; // Sets that load only when a script names them (`--only ambiguous`) or asks // for one of their ids, so every default keeps the original 246 questions. -export const OPT_IN_SETS = ['ambiguous']; +export const OPT_IN_SETS = ['ambiguous', 'code', 'branch']; +// Sets about the git test bed from build-repo.mjs. Their checks read git +// history and branches, so they need `--repo `, and check.mjs checks +// them only when --only names them. +export const REPO_SETS = ['code', 'branch']; // Names that point at one Effort without its title: the engine and renderer // code names that Effort titles use. const EFFORT_NAMES = ['Kestrel', 'Lantern']; @@ -74,12 +89,18 @@ const ANSWER_KINDS = ['record', 'records', 'value', 'boolean', 'count', 'text']; const NEEDS_CHECK = new Set(['record', 'records', 'count']); function args(argv) { - const out = { only: [], graph: join(BENCH, '.flatbread-proof') }; + const out = { only: [], graph: join(BENCH, '.flatbread-proof'), repo: null }; for (let i = 0; i < argv.length; i++) { if (argv[i] === '--only') out.only.push(...argv[++i].split(',')); else if (argv[i] === '--graph') out.graph = resolve(argv[++i]); + else if (argv[i] === '--repo') out.repo = resolve(argv[++i]); else throw new Error(`Unknown argument ${argv[i]}`); } + // The test bed's main graph adds records that change other sets' + // answers, so --repo checks only the sets about it. + if (out.repo && !out.only.length) out.only.push(...REPO_SETS); + if (out.repo && out.only.some((name) => !REPO_SETS.includes(name))) + throw new Error('--repo runs only the code and branch sets'); return out; } @@ -204,6 +225,40 @@ function qualifiedAnswer({ effort, q }) { return q.answer; } +// Words in the order they appear, lowercased, with punctuation dropped. +const words = (text) => + text.toLowerCase().match(/[a-z0-9]+(?:['.-][a-z0-9]+)*/g) ?? []; + +/** + * A test-bed question must not name a record: no id, no title of four + * words, and no run of five words from a longer title, on main or any + * branch, since a reader could search for it. + */ +function leakErrors(graph, q) { + const text = (q.question ?? '').toLowerCase(); + const padded = ` ${words(text).join(' ')} `; + const out = []; + const refs = graph.repo ? ['main', ...graph.repo.branches()] : []; + const seen = new Set(); + const report = (name) => { + if (seen.has(name)) return; + seen.add(name); + out.push(`names "${name}"`); + }; + for (const ref of refs) + for (const rec of graph.repo.graphAt(ref).records.values()) { + if (text.includes(rec.id)) report(rec.id); + const title = words(String(rec.frontmatter.title ?? '')); + if (title.length < 4) continue; + const run = Math.min(5, title.length); + for (let i = 0; i + run <= title.length; i++) { + const phrase = title.slice(i, i + run).join(' '); + if (padded.includes(` ${phrase} `)) report(phrase); + } + } + return out; +} + function validate( graph, set, @@ -264,6 +319,15 @@ function validate( if (answer.kind === 'record') keysOk([answer.value], 'answer'); if (answer.kind === 'records') keysOk(answer.value, 'answer'); const path = q.path ?? []; + // A test-bed question's path may run through records that exist only on + // one branch; `pathRef` names the ref whose graph holds its edges. + let pathGraph = graph; + if (q.pathRef) { + if (!graph.repo) err('`pathRef` needs --repo'); + else pathGraph = graph.repo.graphAt(q.pathRef); + } + if (REPO_SETS.includes(set.effort)) + for (const message of leakErrors(graph, q)) err(message); if (q.tier >= 1 && q.tier <= 3) { if (!path.length) err('tiers 1-3 need a path'); else if (path.length !== q.hops) @@ -344,7 +408,7 @@ function validate( // would otherwise end on the record before it. const end = answer.kind === 'record' ? answer.value : path.at(-1)[2]; try { - const shortest = shortestHops(graph, set.effort, start, end); + const shortest = shortestHops(pathGraph, set.effort, start, end); if (start !== end && shortest < q.hops) warnings.push( `${set.file} ${q.id}: a ${shortest}-edge route joins ${start} and ${end}; the path has ${q.hops}` @@ -355,7 +419,8 @@ function validate( } for (const edge of path) { try { - if (!hasEdge(graph, set.effort, edge)) err(`no edge ${edge.join(' ')}`); + if (!hasEdge(pathGraph, set.effort, edge)) + err(`no edge ${edge.join(' ')}`); } catch (error) { err(`edge ${edge.join(' ')}: ${error.message}`); } @@ -452,8 +517,18 @@ export function graphStats(graph, effortKey) { async function main() { const options = args(process.argv.slice(2)); - const graph = await loadGraph(options.graph); - const sets = await loadQuestions(options.only, { optIn: true }); + const graph = options.repo + ? openTestBed(options.repo).graphAt('main') + : await loadGraph(options.graph); + const sets = (await loadQuestions(options.only, { optIn: true })).filter( + (set) => + !REPO_SETS.includes(set.effort) || options.only.includes(set.effort) + ); + for (const set of sets) + if (REPO_SETS.includes(set.effort) && !options.repo) + throw new Error( + `${set.file} needs --repo , a test bed from build-repo.mjs` + ); // Ambiguous questions are checked against their sources, so load every // Effort file even under --only. const sources = new Map(); diff --git a/benchmarks/proof-recall/scripts/compare.mjs b/benchmarks/proof-recall/scripts/compare.mjs index 9a99c197..64867494 100644 --- a/benchmarks/proof-recall/scripts/compare.mjs +++ b/benchmarks/proof-recall/scripts/compare.mjs @@ -2,7 +2,7 @@ // Compares two runs over the questions both answered, pair by pair. // // node scripts/compare.mjs runs/.json runs/.json [--skip id1,id2] -// [--only ambiguous] [--split tune|holdout|all] +// [--only ambiguous] [--split tune|holdout|all] [--repo ] // node scripts/compare.mjs runs/a-r1.json,runs/a-r2.json \ // runs/b-r1.json,runs/b-r2.json // @@ -31,7 +31,7 @@ import { fileURLToPath, pathToFileURL } from 'node:url'; import { loadQuestions } from './check.mjs'; import { splitOf } from './find-recall.mjs'; import { score } from './grade.mjs'; -import { loadGraph } from './graph.mjs'; +import { loadGraph, readGraph } from './graph.mjs'; const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); @@ -67,10 +67,14 @@ async function main() { const split = atSplit >= 0 ? argv.splice(atSplit, 2)[1] : 'all'; if (!['tune', 'holdout', 'all'].includes(split)) throw new Error('--split must be one of tune, holdout, all'); - const [pathA, pathB] = argv; - if (!pathB) + // The code and branch sets name records that exist only in the git test + // bed, so they are scored against it, as grade.mjs does. + const atRepo = argv.indexOf('--repo'); + const repo = atRepo >= 0 ? resolve(argv.splice(atRepo, 2)[1]) : null; + const [pathA, pathB, ...extra] = argv; + if (!pathB || extra.length) throw new Error( - 'usage: compare.mjs [,more] [,more] [--skip ids] [--only sets] [--split tune|holdout|all]' + 'usage: compare.mjs [,more] [,more] [--skip ids] [--only sets] [--split tune|holdout|all] [--repo ]' ); const load = (paths) => Promise.all( @@ -100,7 +104,12 @@ async function main() { pooled ? runs.map((run, i) => run.run?.name ?? paths.split(',')[i]).join(',') : runs[0].run?.name ?? paths; - const graph = await loadGraph(join(BENCH, '.flatbread-proof')); + const graph = repo + ? readGraph( + join(repo, '.flatbread-proof'), + JSON.parse(await readFile(`${repo}.keymap.json`, 'utf8')) + ) + : await loadGraph(join(BENCH, '.flatbread-proof')); const sets = await loadQuestions(only); const rows = []; for (const [i, a] of runsA.entries()) diff --git a/benchmarks/proof-recall/scripts/grade.mjs b/benchmarks/proof-recall/scripts/grade.mjs index 882ec05e..8f7e90a9 100644 --- a/benchmarks/proof-recall/scripts/grade.mjs +++ b/benchmarks/proof-recall/scripts/grade.mjs @@ -2,7 +2,7 @@ // Scores one benchmark run. // // node scripts/grade.mjs runs/.json [--only ambiguous] -// [--split tune|holdout|all] +// [--split tune|holdout|all] [--repo ] // // It scores the original 246 questions that the run answered. `--only` // names the sets to score instead, as run.mjs takes them: `ambiguous` for @@ -10,7 +10,9 @@ // scores only the tune or holdout questions, split by a hash of each id as // in find-recall.mjs. A read that was changed after reading tune misses, // such as `proof state`, should be judged on holdout. `bySplit` always -// reports both halves. +// reports both halves. The code and branch sets name records that exist +// only in the git test bed, so score them with `--repo `, the folder +// build-repo.mjs wrote; its key map sits beside it. // // A run file holds one entry per question id: // { "run": { "strategy": "brief", "model": "..." }, @@ -23,7 +25,7 @@ import { dirname, join, resolve } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { loadQuestions } from './check.mjs'; import { splitOf } from './find-recall.mjs'; -import { loadGraph, qualify, sameAnswer } from './graph.mjs'; +import { loadGraph, qualify, readGraph, sameAnswer } from './graph.mjs'; const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); @@ -130,6 +132,8 @@ async function main() { const only = at >= 0 ? argv.splice(at, 2)[1].split(',') : []; const atSplit = argv.indexOf('--split'); const split = atSplit >= 0 ? argv.splice(atSplit, 2)[1] : 'all'; + const atRepo = argv.indexOf('--repo'); + const repo = atRepo >= 0 ? resolve(argv.splice(atRepo, 2)[1]) : null; if (!['tune', 'holdout', 'all'].includes(split)) throw new Error('--split must be one of tune, holdout, all'); const [runPath] = argv; @@ -138,7 +142,12 @@ async function main() { 'usage: grade.mjs [--only sets] [--split tune|holdout|all]' ); const run = JSON.parse(await readFile(resolve(runPath), 'utf8')); - const graph = await loadGraph(join(BENCH, '.flatbread-proof')); + const graph = repo + ? readGraph( + join(repo, '.flatbread-proof'), + JSON.parse(await readFile(`${repo}.keymap.json`, 'utf8')) + ) + : await loadGraph(join(BENCH, '.flatbread-proof')); const rows = []; for (const set of await loadQuestions(only)) for (const q of set.questions) { diff --git a/benchmarks/proof-recall/scripts/graph.mjs b/benchmarks/proof-recall/scripts/graph.mjs index 7938417f..0634d4fb 100644 --- a/benchmarks/proof-recall/scripts/graph.mjs +++ b/benchmarks/proof-recall/scripts/graph.mjs @@ -1,6 +1,7 @@ // Loads a built benchmark graph and evaluates question checks against it. // Shared by check.mjs (authoring) and grade.mjs (scoring runs). -import { readdir, readFile } from 'node:fs/promises'; +import { readdirSync, readFileSync } from 'node:fs'; +import { readFile } from 'node:fs/promises'; import { join } from 'node:path'; import { compactionChanges, @@ -19,17 +20,26 @@ const EXCERPT_LINES = 12; export async function loadGraph(root) { const keymap = JSON.parse(await readFile(join(root, 'keymap.json'), 'utf8')); + return readGraph(root, keymap); +} + +/** + * The records under `root`, read synchronously, with a key map from + * elsewhere. The test bed keeps its key map outside the repository, and + * checks on a git ref read a graph that git has just written out. + */ +export function readGraph(root, keymap) { const keyById = new Map(Object.entries(keymap).map(([k, id]) => [id, k])); const records = new Map(); for (const [kind, dir] of Object.entries(KIND_DIRECTORY)) { let files = []; try { - files = await readdir(join(root, dir)); + files = readdirSync(join(root, dir)); } catch { continue; } for (const file of files.filter((name) => name.endsWith('.md'))) { - const bytes = await readFile(join(root, dir, file)); + const bytes = readFileSync(join(root, dir, file)); const { frontmatter, body } = parseDocument(bytes, kind); records.set(frontmatter.id, { id: frontmatter.id, @@ -162,7 +172,7 @@ export function hasEdge(graph, effort, [fromKey, relation, toKey]) { return list(from.frontmatter[relation]).includes(to.id); } -function head(graph, rec) { +export function headRecord(graph, rec) { const seen = new Set(); let current = rec; while (current && list(current.frontmatter.superseded_by).length) { @@ -184,7 +194,7 @@ function chain(graph, rec) { return out; } -function matches(rec, where = {}) { +export function matches(rec, where = {}) { return Object.entries(where).every(([field, want]) => { const value = field === 'standing' ? standing(rec) : rec.frontmatter[field]; return list(want).map(String).includes(String(value)); @@ -241,7 +251,7 @@ export function runCheck(graph, effort, check) { return rec.frontmatter[check.field] ?? null; } case 'head': - return keyOf(head(graph, record(graph, effort, check.start))); + return keyOf(headRecord(graph, record(graph, effort, check.start))); case 'chain': return chain(graph, record(graph, effort, check.start)).map(keyOf); case 'asof': { @@ -352,6 +362,18 @@ export function runCheck(graph, effort, check) { } case 'edge': return hasEdge(graph, effort, check.edge); + // Checks on the git test bed (scripts/testbed.mjs), for the code and + // branch question sets. + case 'atRef': + case 'trailer': + case 'codeAnchor': + case 'pr': + case 'refDiff': + case 'changedPaths': + case 'conflicts': + if (!graph.repo) + throw new Error(`check ${check.type} needs --repo `); + return graph.repo.runCheck(graph, effort, check, runCheck); default: throw new Error(`bad check type ${check.type}`); } diff --git a/benchmarks/proof-recall/scripts/prepare-run.mjs b/benchmarks/proof-recall/scripts/prepare-run.mjs index c49f1830..1f546350 100644 --- a/benchmarks/proof-recall/scripts/prepare-run.mjs +++ b/benchmarks/proof-recall/scripts/prepare-run.mjs @@ -7,7 +7,18 @@ // // The answering agent works only inside , running // `./flatbread proof …` there. -import { cp, mkdir, rm, symlink, writeFile } from 'node:fs/promises'; +import { execFileSync } from 'node:child_process'; +import { + appendFile, + cp, + mkdir, + readdir, + readFile, + rm, + stat, + symlink, + writeFile, +} from 'node:fs/promises'; import { dirname, join, resolve } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { pickQuestions } from './check.mjs'; @@ -70,6 +81,18 @@ export async function prepareFolder( join(BENCH, 'flatbread.config.js'), join(out, 'flatbread.config.js') ); + await linkTools(out, server); + await writeFile( + join(out, 'questions.json'), + `${JSON.stringify(questions, null, 2)}\n` + ); +} + +/** + * Links the workspace's node_modules into `out` and writes its `./flatbread` + * wrapper. With `server`, the wrapper sends reads to that socket. + */ +async function linkTools(out, server) { await symlink( join(REPO, 'node_modules'), join(out, 'node_modules'), @@ -88,9 +111,61 @@ export async function prepareFolder( )}" "$@"\n`, { mode: 0o755 } ); - await writeFile( - join(out, 'questions.json'), - `${JSON.stringify(questions, null, 2)}\n` +} + +// Files a session adds to its clone, kept out of `git status`. +const SESSION_FILES = [ + '/flatbread', + '/node_modules', + '/TASK.md', + '/answer.json', + '/transcript.jsonl', + '/stderr.txt', +]; + +/** + * Empties `out` and makes it a `git clone --no-hardlinks` of the test bed + * `repo` (from build-repo.mjs), so a session has main checked out, every + * branch as `origin/`, and the full log. The clone gets the + * gitignored draft-PR file, the tool links, and nothing else: no key map, + * spec, or questions. Its `origin` is renamed to a URL that leads nowhere + * and its reflogs are deleted, since the real path sits beside the key map. + */ +export async function prepareClone(out, repo, { server } = {}) { + await rm(out, { recursive: true, force: true }); + await mkdir(dirname(out), { recursive: true }); + const git = (cwd, ...argv) => + execFileSync('git', argv, { + cwd, + stdio: ['ignore', 'ignore', 'pipe'], + env: { + ...process.env, + GIT_CONFIG_NOSYSTEM: '1', + GIT_CONFIG_GLOBAL: '/dev/null', + }, + }); + git(dirname(out), 'clone', '--quiet', '--no-hardlinks', repo, out); + git( + out, + 'remote', + 'set-url', + 'origin', + 'https://git.tidewater.example/saltglass-reach.git' + ); + // The reflogs record "clone: from ", so they go too. + await rm(join(out, '.git', 'logs'), { recursive: true, force: true }); + for (const file of await readdir(join(out, '.git'), { recursive: true })) { + const path = join(out, '.git', file); + if ((await stat(path)).isFile() && (await readFile(path)).includes(repo)) + throw new Error(`${path} still names the test bed path ${repo}`); + } + const prs = join('.flatbread', 'proof', 'prs.json'); + await mkdir(join(out, dirname(prs)), { recursive: true }); + await cp(join(repo, prs), join(out, prs)); + await linkTools(out, server); + await appendFile( + join(out, '.git', 'info', 'exclude'), + `${SESSION_FILES.join('\n')}\n` ); } diff --git a/benchmarks/proof-recall/scripts/run.mjs b/benchmarks/proof-recall/scripts/run.mjs index 729fbc3e..4c8ed83f 100644 --- a/benchmarks/proof-recall/scripts/run.mjs +++ b/benchmarks/proof-recall/scripts/run.mjs @@ -5,7 +5,7 @@ // node scripts/run.mjs --name [--model haiku] // [--strategies brief,nobrief,find,findstate] [--only render] [--ids a,b] // [--jobs 8] [--timeout 600] [--graph ] -// [--server ] [--no-serve] +// [--server ] [--no-serve] [--repo ] [--prepare-only] // node scripts/run.mjs --stop-server [--server ] // // --strategies names the task texts in task.mjs to run; the default is @@ -34,6 +34,18 @@ // with --stop-server, or restart it after rebuilding the CLI so it runs // the new code. --no-serve makes every read run locally instead. // +// --repo runs the code and branch sets (`--only code`, `--only +// branch`) on the git test bed from build-repo.mjs. Each session's folder is +// then its own `git clone --no-hardlinks` of the test bed, so git log, the +// branches, and the draft-PR file are there for every strategy. It holds no +// key map, spec, or questions file. These runs use neither the shared graph +// copy nor the read server: every clone is a different graph root, so +// reads run locally. +// +// --prepare-only builds every session folder and TASK.md, then stops. It +// starts no agent session and no server, and makes no shared graph copy, so +// it is the way to inspect what sessions would see. +// // Sessions get only the Bash, Read, and Write tools, no MCP servers, and no // settings files, so an agent cannot reach other sessions or services. import { spawn } from 'node:child_process'; @@ -56,10 +68,11 @@ import { fileURLToPath } from 'node:url'; import { collect, findHash, readHash, stateHash } from './collect.mjs'; import { copyPublicGraph, + prepareClone, prepareFolder, publicQuestions, } from './prepare-run.mjs'; -import { RUNS_DIR } from './check.mjs'; +import { pickQuestions, REPO_SETS, RUNS_DIR } from './check.mjs'; import { STRATEGIES, taskText } from './task.mjs'; const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); @@ -105,6 +118,8 @@ function args(argv) { serve: true, server: join(SHARED, 'proof.sock'), stopServer: false, + repo: null, + prepareOnly: false, }; for (let i = 0; i < argv.length; i++) { const flag = argv[i]; @@ -120,6 +135,8 @@ function args(argv) { else if (flag === '--no-serve') out.serve = false; else if (flag === '--server') out.server = resolve(next()); else if (flag === '--stop-server') out.stopServer = true; + else if (flag === '--repo') out.repo = resolve(next()); + else if (flag === '--prepare-only') out.prepareOnly = true; else if (!flag.startsWith('--') && !out.dir) out.dir = resolve(flag); else throw new Error(`Unknown argument ${flag}`); } @@ -133,6 +150,12 @@ function args(argv) { throw new Error('usage: run.mjs --name [options]'); if (out.dir.startsWith(resolve(BENCH, '..', '..') + '/')) throw new Error('Keep outside the repository'); + if (out.repo && out.dir.startsWith(`${out.repo}/`)) + throw new Error('Keep outside the test bed'); + if (out.repo && out.graph) + throw new Error( + '--repo sessions read the graph in their clone; drop --graph' + ); return out; } @@ -336,6 +359,15 @@ function runSession(folder, model, timeoutSeconds) { async function main() { const options = args(process.argv.slice(2)); if (options.stopServer) return stopServer(options.server); + // The test-bed sets need the test bed, and its graph holds records that + // could change other sets' answers, so the two never share a run. + const onBed = (await pickQuestions(options)).map((set) => + REPO_SETS.includes(set.effort) + ); + if (options.repo && onBed.includes(false)) + throw new Error('--repo runs only the code and branch sets'); + if (!options.repo && onBed.includes(true)) + throw new Error('The code and branch sets need --repo '); const questions = await publicQuestions(options); const find_hash = await findHash(); const state_hash = await stateHash(); @@ -360,23 +392,42 @@ async function main() { `${sessions.length} sessions, ${todo.length} to run, ${options.jobs} at a time` ); // Copying a large graph per session would fill the disk, so every session - // links to one read-only copy. - const { dir: shared, hash } = await sharedGraph(options.graph); - const socket = options.serve ? options.server : undefined; + // links to one read-only copy. A test-bed session reads the graph in its + // own clone instead, and --prepare-only touches nothing shared. + const shareGraph = !options.repo && !options.prepareOnly; + const { dir: shared, hash } = shareGraph + ? await sharedGraph(options.graph) + : {}; + const socket = options.serve && shareGraph ? options.server : undefined; if (socket) console.log(`proof serve on ${socket}: ${await ensureServer(socket)}`); - let done = 0; - const failures = []; - const worker = async () => { - for (let s = todo.shift(); s; s = todo.shift()) { + const prepare = async (s) => { + if (options.repo) await prepareClone(s.folder, options.repo); + else await prepareFolder(s.folder, [s.q], options.graph, { linkTo: shared, server: socket, }); - await writeFile( - join(s.folder, 'TASK.md'), - taskText({ folder: s.folder, strategy: s.strategy, ...s.q }) - ); + await writeFile( + join(s.folder, 'TASK.md'), + taskText({ + folder: s.folder, + strategy: s.strategy, + repo: Boolean(options.repo), + ...s.q, + }) + ); + }; + if (options.prepareOnly) { + for (const s of todo) await prepare(s); + console.log(`Prepared ${todo.length} session folders; started no sessions`); + return; + } + let done = 0; + const failures = []; + const worker = async () => { + for (let s = todo.shift(); s; s = todo.shift()) { + await prepare(s); const result = await runSession(s.folder, options.model, options.timeout); done++; if (result !== 0) failures.push(`${s.strategy}/${s.id}: ${result}`); @@ -386,7 +437,7 @@ async function main() { await Promise.all(Array.from({ length: options.jobs }, worker)); if (failures.length) console.warn(`Failed:\n${failures.join('\n')}`); // Root ignores file modes, so check that no session changed the graph. - if ((await treeHash(shared)) !== hash) + if (shared && (await treeHash(shared)) !== hash) console.warn(`The shared graph in ${shared} changed during the run`); await mkdir(RUNS_DIR, { recursive: true }); for (const strategy of options.strategies) diff --git a/benchmarks/proof-recall/scripts/task.mjs b/benchmarks/proof-recall/scripts/task.mjs index cbcded5e..a83bd44a 100644 --- a/benchmarks/proof-recall/scripts/task.mjs +++ b/benchmarks/proof-recall/scripts/task.mjs @@ -2,7 +2,7 @@ // commands its strategy allows, and one question. Never the answer. import { join } from 'node:path'; -export const STRATEGIES = ['brief', 'nobrief', 'find', 'findstate']; +export const STRATEGIES = ['brief', 'nobrief', 'find', 'findstate', 'findcode']; const BRIEF_LINE = '- `./flatbread proof brief `: an Effort\'s latest summary (a "compaction") in full, plus every record that changed since. Its envelope\'s `compaction.stale` is true when a record the summary relies on has changed. Start each Effort you need here.'; @@ -16,21 +16,40 @@ const FIND_FIRST_STATE = '\nStart with `find`, or with `state` for counts and the lists it holds. Pass `--all` when the question names no Effort, and `--effort ` when it does (`list` gives Effort ids). Run `get` on a hit to read its full text. Use `records` to page through records. Do not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n'; const FIND_FIRST = '\nStart with `find`. Pass `--all` when the question names no Effort, and `--effort ` when it does (`list` gives Effort ids). Run `get` on a hit to read its full text. Use `records` only to count records or to page through them. Do not use `proof brief`. Findings with `kind: compaction` are summaries; check what they claim.\n'; +// `findcode` adds these to the `find` text, for runs on the git test bed. +const FIND_CODE = + 'Before opening code, run `proof find --path ` to see the records that govern it; add `--refs all` when the file may exist only on a branch.\nBefore building on a Decision, run `proof find --refs all` to see what open branches propose and where they conflict.\n'; +// Every arm of a test-bed run (`repo`) gets this section, so that what an +// arm can answer depends on its read commands, not on a ban on git. +const GIT_SECTION = ` +## This folder is a git clone +Main is checked out, and the read commands above see only main's records. Each open branch exists only as \`origin/\`. Draft PRs for them are listed in \`.flatbread/proof/prs.json\` (number, branch, title, author). Commits name the record they follow in a \`Proof:\` trailer. +- You may read code files and run read-only git: \`git log\`, \`git show :\` (records on a branch included), \`git diff main...origin/\`, and \`git branch -a\`. Never change the clone: no checkout, switch, commit, merge, fetch, or reset. +`; -/** Markdown for TASK.md in `folder`, for one question and one strategy. */ -export function taskText({ folder, question, format, strategy }) { +/** + * Markdown for TASK.md in `folder`, for one question and one strategy. + * With `repo`, the folder is a clone of the git test bed and the text also + * allows read-only git; without it the text is unchanged. + */ +export function taskText({ folder, question, format, strategy, repo = false }) { if (!STRATEGIES.includes(strategy)) throw new Error(`Unknown strategy ${strategy}`); const brief = strategy === 'brief'; - // `findstate` is the `find` text plus the `state` line. - const find = strategy === 'find' || strategy === 'findstate'; + // `findstate` is the `find` text plus the `state` line, and `findcode` + // the `find` text plus the FIND_CODE lines. + const find = ['find', 'findstate', 'findcode'].includes(strategy); const state = strategy === 'findstate'; return `# One-question recall test -Answer one question about a project-memory graph, using only its read commands. Every command you run is counted, so be efficient, but a right answer matters most. +Answer one question about a project-memory graph, using only its read commands${ + repo ? ' and read-only git' : '' + }. Every command you run is counted, so be efficient, but a right answer matters most. ## Rules -- Work only inside this folder: \`${folder}\`. Never read, list, or search anything outside it. Never open files under \`.flatbread-proof/\` directly. +- Work only inside this folder: \`${folder}\`. Never read, list, or search anything outside it. Never open files under \`.flatbread-proof/\` directly${ + repo ? ', except through `git show :`' : '' + }. - Run one graph command per tool call. No shell loops, scripts, or chains (\`&&\`, \`;\`, \`|\` into another graph command) that run several graph commands at once. ## Read commands @@ -44,9 +63,11 @@ ${ - \`./flatbread proof get [--resolve head]\` shows one record in full, Blob payloads included. \`--resolve head\` follows replacements to the newest version. - \`./flatbread proof relations --relations derives_from,supersedes,superseded_by,invalidates,invalidated_by,resolved_by,rejected_by,mitigated_by,evidence,cites\` follows one hop of links. - \`./flatbread proof blocking-decisions \`. -${brief ? '' : state ? FIND_FIRST_STATE : find ? FIND_FIRST : NO_BRIEF} +${brief ? '' : state ? FIND_FIRST_STATE : find ? FIND_FIRST : NO_BRIEF}${ + strategy === 'findcode' ? FIND_CODE : '' + } Records can be superseded (replaced), invalidated (shown wrong), rejected, reopened, or retracted (noise, hidden from browse reads unless asked; \`get\` still shows them). Titles repeat on purpose. Efforts never link to each other; one may name another's record by title in prose. A summary can be out of date; check its claims against the records. - +${repo ? GIT_SECTION : ''} ## The question ${question} diff --git a/benchmarks/proof-recall/scripts/testbed.mjs b/benchmarks/proof-recall/scripts/testbed.mjs new file mode 100644 index 00000000..d448807b --- /dev/null +++ b/benchmarks/proof-recall/scripts/testbed.mjs @@ -0,0 +1,478 @@ +// Checks against the git test bed that build-repo.mjs makes, for the code +// and branch question sets. check.mjs opens it with --repo . +// +// | type | fields | returns | +// | -------------- | ------------------------------------------------------- | ---------------------------------------------- | +// | `atRef` | `ref`, `check`; for `ref: "*"`, `equals` and `report` | the inner check's result on that ref's graph | +// | `trailer` | `path`, `ref`, `commit`, `symbol`, filters | records named by `Proof:` trailers | +// | `codeAnchor` | `path`, `symbol`, `ref`, `commit`, `citations`, filters | records that cite a code anchor for the path | +// | `pr` | `branch`, `field` | a draft PR's number, author, login or isDraft | +// | `refDiff` | `ref`, `change`, `effort`, `kinds`, `where`, `count` | records a branch adds or changes | +// | `changedPaths` | `ref`, `under`, `count` | code files a branch changes | +// | `conflicts` | `refs` (two), `kind`, `kinds`, `count` | records in that conflict if both refs merged | +// +// `ref` defaults to main; `"*"` means every branch but main. A graph on a +// ref is written out with `git archive` into a temporary folder and read +// like any other. Branch records have keys in the key map like main's, and +// a key whose record is missing on a ref fails the check that names it. +import { execFileSync } from 'node:child_process'; +import { mkdirSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { isCompaction, isLiveStanding } from '@flatbread/proof'; +import { headRecord, matches, readGraph, standing } from './graph.mjs'; +import { specAnchors, specCommits } from '../spec/repo.mjs'; + +const list = (value) => + Array.isArray(value) ? value : value === undefined ? [] : [value]; + +/** True when `path` is `prefix`, or a file under the folder `prefix`. */ +function under(path, prefix) { + const dir = prefix.replace(/\/+$/, ''); + return path === dir || path.startsWith(`${dir}/`); +} + +/** + * Filters shared by `trailer` and `codeAnchor`, applied in this order: + * `follow` (a link field to step along, such as `derives_from`), `resolve: + * "head"`, then `kinds`, `where`, `live`, `titleIncludes`, `bodyIncludes`. + * Returns sorted keys, or their count with `count`. + */ +function finish(graph, recs, check) { + let out = recs; + if (check.follow) + out = out.flatMap((rec) => + list(rec.frontmatter[check.follow]).map((id) => { + const next = graph.records.get(id); + if (!next) throw new Error(`${rec.id} links to missing ${id}`); + return next; + }) + ); + if (check.resolve === 'head') out = out.map((rec) => headRecord(graph, rec)); + if (check.kinds) out = out.filter((rec) => check.kinds.includes(rec.kind)); + if (check.where) out = out.filter((rec) => matches(rec, check.where)); + if (check.live !== undefined) + out = out.filter( + (rec) => isLiveStanding(rec.kind, standing(rec)) === check.live + ); + const contains = (text, phrases) => + list(phrases).every((p) => text.toLowerCase().includes(p.toLowerCase())); + if (check.titleIncludes) + out = out.filter((rec) => + contains(String(rec.frontmatter.title ?? ''), check.titleIncludes) + ); + if (check.bodyIncludes) + out = out.filter((rec) => contains(rec.body, check.bodyIncludes)); + const keys = [ + ...new Set( + out.map((rec) => { + const key = graph.keyById.get(rec.id); + if (!key) throw new Error(`no key for ${rec.id}`); + return key; + }) + ), + ].sort(); + return check.count ? keys.length : keys; +} + +export function openTestBed(dir) { + const keymap = JSON.parse(readFileSync(`${dir}.keymap.json`, 'utf8')); + const prs = JSON.parse( + readFileSync(join(dir, '.flatbread', 'proof', 'prs.json'), 'utf8') + ); + const env = { + ...process.env, + GIT_CONFIG_NOSYSTEM: '1', + GIT_CONFIG_GLOBAL: '/dev/null', + }; + const git = (...argv) => + execFileSync('git', argv, { + cwd: dir, + env, + encoding: 'utf8', + maxBuffer: 1 << 28, + stdio: ['ignore', 'pipe', 'pipe'], + }).trim(); + const succeeds = (...argv) => { + try { + git(...argv); + return true; + } catch { + return false; + } + }; + const temp = mkdtempSync(join(tmpdir(), 'proof-recall-refs-')); + process.on('exit', () => rmSync(temp, { recursive: true, force: true })); + const graphs = new Map(); + const bed = { + dir, + keymap, + prs, + git, + commitOf: (ref) => git('rev-parse', '--verify', `${ref}^{commit}`), + mergeBase: (a, b) => git('merge-base', a, b), + branches: () => + git('for-each-ref', '--format=%(refname:short)', 'refs/heads') + .split('\n') + .filter((name) => name && name !== 'main') + .sort(), + pr(branch) { + const pr = prs.find((p) => p.headRefName === branch); + if (!pr) throw new Error(`no PR for ${branch}`); + return pr; + }, + /** The graph as it is on `ref`, read once per commit. */ + graphAt(ref) { + const commit = bed.commitOf(ref); + if (!graphs.has(commit)) { + const out = join(temp, commit); + mkdirSync(out); + const tar = execFileSync( + 'git', + ['archive', '--format=tar', commit, '.flatbread-proof'], + { cwd: dir, env, maxBuffer: 1 << 28 } + ); + execFileSync('tar', ['-x', '-C', out], { input: tar }); + const graph = readGraph(join(out, '.flatbread-proof'), keymap); + graph.repo = bed; + graph.ref = ref; + graph.commit = commit; + graphs.set(commit, graph); + } + return graphs.get(commit); + }, + runCheck: (graph, effort, check, runCheck) => + CHECKS[check.type](bed, graph, effort, check, runCheck), + }; + // Questions name commits by what they did; the spec names them by key. + const bySubject = new Map( + git('log', '--all', '--format=%H%x09%s') + .split('\n') + .map((line) => [line.slice(41), line.slice(0, 40)]) + ); + bed.commitSha = new Map( + specCommits().map((c) => [c.key, bySubject.get(c.subject)]) + ); + for (const [key, sha] of bed.commitSha) + if (!sha) throw new Error(`spec commit ${key} is not in ${dir}`); + bed.succeeds = succeeds; + return bed; +} + +/** Commits on `ref` that touch `path`, newest first. */ +function commitsTouching(bed, ref, path, symbol) { + const argv = ['log', '--format=%H', ref]; + if (symbol) argv.push(`-S${symbol}`); + const out = bed.git(...argv, '--', path); + return out ? out.split('\n') : []; +} + +const CHECKS = { + atRef(bed, graph, effort, check, runCheck) { + if (!check.check) throw new Error('atRef needs `check`'); + if (check.ref !== '*') + return runCheck(bed.graphAt(check.ref ?? 'main'), effort, check.check); + // Every branch where the inner check gives `equals`, as branch names, + // PR numbers, or PR authors. A check that fails on a branch, such as one + // naming a record the branch lacks, does not match there. + if (!('equals' in check)) throw new Error('atRef on "*" needs `equals`'); + const hits = bed.branches().filter((branch) => { + try { + const got = runCheck(bed.graphAt(branch), effort, check.check); + return JSON.stringify(got) === JSON.stringify(check.equals); + } catch { + return false; + } + }); + const report = (branch) => + check.report === 'pr' + ? bed.pr(branch).number + : check.report === 'author' + ? bed.pr(branch).author.name + : branch; + const out = hits.map(report); + return out.length === 1 ? out[0] : out; + }, + + // Records named by `Proof: ` trailers on commits that touch `path` (a + // file or a folder). `commit`: "all" (the default), "latest", or + // "introducing", the oldest; with `symbol`, only commits that add or + // remove that text count. + trailer(bed, _graph, _effort, check) { + const ref = check.ref ?? 'main'; + const graph = bed.graphAt(ref); + let shas = commitsTouching(bed, ref, check.path, check.symbol); + if (!shas.length) + throw new Error(`no commit on ${ref} touches ${check.path}`); + if (check.commit === 'latest') shas = shas.slice(0, 1); + else if (check.commit === 'introducing') shas = shas.slice(-1); + else if (check.commit && check.commit !== 'all') + throw new Error(`bad commit ${check.commit}`); + const ids = shas.flatMap((sha) => + bed + .git( + 'log', + '-1', + '--format=%(trailers:key=Proof,valueonly,separator=%x2C)', + sha + ) + .split(',') + .map((id) => id.trim()) + .filter(Boolean) + ); + const recs = ids.map((id) => { + const rec = graph.records.get(id); + if (!rec) throw new Error(`a trailer names ${id}, which ${ref} lacks`); + return rec; + }); + return finish(graph, recs, check); + }, + + // Records that cite a `role: code` Citation for `path` (a file or a + // folder), and `symbol` if given. Every such Citation must be one that + // spec/repo.mjs declares, and git must agree with it: the commit exists on + // the ref, touches the file, and the file holds the symbol at that commit. + // `commit: "introducing"` keeps anchors on the commit that first added the + // symbol to the file. `citations: true` returns the Citations themselves. + codeAnchor(bed, _graph, _effort, check) { + const ref = check.ref ?? 'main'; + const graph = bed.graphAt(ref); + const declared = new Map(specAnchors().map((a) => [a.key, a])); + const anchors = []; + for (const rec of graph.records.values()) { + if (rec.kind !== 'citation' || rec.frontmatter.role !== 'code') continue; + if (rec.frontmatter.retracted === true) continue; + const key = graph.keyById.get(rec.id); + const match = /^(\S+)#(\S+)@([0-9a-f]{7,40})$/.exec(rec.body.trim()); + if (!match) + throw new Error(`code anchor ${key} is not #@`); + const [, path, symbol, abbrev] = match; + if (!under(path, check.path)) continue; + if (check.symbol && symbol !== check.symbol) continue; + const spec = declared.get(key); + if ( + !spec || + spec.path !== path || + spec.symbol !== symbol || + !bed.commitSha.get(spec.commit)?.startsWith(abbrev) + ) + throw new Error(`code anchor ${key} is not one spec/repo.mjs declares`); + const sha = bed.commitOf(abbrev); + if (!bed.succeeds('merge-base', '--is-ancestor', sha, ref)) + throw new Error(`code anchor ${key} names ${abbrev}, not on ${ref}`); + const touched = bed + .git('show', '--name-only', '--format=', sha) + .split('\n'); + if (!touched.includes(path)) + throw new Error(`code anchor ${key}: ${abbrev} does not touch ${path}`); + if (!bed.git('show', `${sha}:${path}`).includes(symbol)) + throw new Error( + `code anchor ${key}: ${path} has no ${symbol} at ${abbrev}` + ); + anchors.push({ rec, path, symbol, sha }); + } + let kept = anchors; + if (check.commit === 'introducing') + kept = anchors.filter( + (a) => commitsTouching(bed, ref, a.path, a.symbol).at(-1) === a.sha + ); + else if (check.commit && check.commit !== 'all') + throw new Error(`bad commit ${check.commit}`); + const ids = new Set(kept.map((a) => a.rec.id)); + const recs = check.citations + ? kept.map((a) => a.rec) + : [...graph.records.values()].filter((rec) => + list(rec.frontmatter.cites).some((id) => ids.has(id)) + ); + return finish(graph, recs, check); + }, + + pr(bed, _graph, _effort, check) { + const pr = bed.pr(check.branch); + if (check.field === 'number') return pr.number; + if (check.field === 'author') return pr.author.name; + if (check.field === 'login') return pr.author.login; + if (check.field === 'isDraft') return pr.isDraft; + throw new Error(`bad pr field ${check.field}`); + }, + + // Records on a branch measured against where it left main. `change`: + // "added" (new on the branch), "restanded" (its standing differs), + // "changed" (any byte of frontmatter or body differs), or "retracted". + refDiff(bed, _graph, _effort, check) { + const refs = check.ref === '*' ? bed.branches() : [check.ref]; + const effortId = check.effort && bed.keymap[`${check.effort}.effort`]; + if (check.effort && !effortId) + throw new Error(`unknown Effort ${check.effort}`); + const keys = new Set(); + for (const ref of refs) { + const graph = bed.graphAt(ref); + const base = bed.graphAt(bed.mergeBase('main', ref)); + for (const rec of graph.records.values()) { + if (check.kinds && !check.kinds.includes(rec.kind)) continue; + if (check.where && !matches(rec, check.where)) continue; + if (check.effort && rec.frontmatter.effort !== effortId) continue; + const old = base.records.get(rec.id); + const hit = + check.change === 'added' + ? !old + : check.change === 'restanded' + ? old && standing(old) !== standing(rec) + : check.change === 'retracted' + ? old && + old.frontmatter.retracted !== true && + rec.frontmatter.retracted === true + : check.change === 'changed' + ? old && + JSON.stringify([old.frontmatter, old.body]) !== + JSON.stringify([rec.frontmatter, rec.body]) + : (() => { + throw new Error(`bad change ${check.change}`); + })(); + if (hit) keys.add(graph.keyById.get(rec.id)); + } + } + const out = [...keys].sort(); + return check.count ? out.length : out; + }, + + // Code files (anything outside .flatbread-proof/) that a branch changes + // since it left main, under the folder `under`. + changedPaths(bed, _graph, _effort, check) { + const refs = check.ref === '*' ? bed.branches() : [check.ref]; + const paths = new Set(); + for (const ref of refs) { + const out = bed.git( + 'diff', + '--name-only', + bed.mergeBase('main', ref), + ref, + '--', + check.under ?? '.' + ); + for (const path of out ? out.split('\n') : []) + if (!under(path, '.flatbread-proof')) paths.add(path); + } + const out = [...paths].sort(); + return check.count ? out.length : out; + }, + + conflicts(bed, _graph, _effort, check) { + const [a, b] = check.refs ?? []; + if (!a || !b) throw new Error('conflicts needs two refs'); + const out = conflicts(bed, a, b, check.kind).filter( + (rec) => !check.kinds || check.kinds.includes(rec.kind) + ); + const keys = [ + ...new Set(out.map((rec) => bed.graphAt(a).keyById.get(rec.id))), + ].sort(); + return check.count ? keys.length : keys; + }, +}; + +/** + * What merging refs `a` and `b` would do to the graph, measured against + * their merge base. A record's merged standing is the side that changed it; + * when both changed it to different standings, it holds both. Reverse links + * such as `superseded_by` are rebuilt from forward fields on either side. + * + * `kind`: + * - "double-accept": question Issues with two or more accepted Decisions + * deriving from them; + * - "double-supersede": records superseded by two or more different records; + * - "state-differs": records whose standing each side changed, differently; + * - "retracted-relied": records retracted on one side, and not the other, + * that a record still live after the merge derives from (never when one + * ref contains the other); + * - "compaction-fork": the compaction heads of an Effort left with more than + * one. + */ +export function conflicts(bed, a, b, kind) { + const base = bed.graphAt(bed.mergeBase(a, b)); + const sides = [bed.graphAt(a), bed.graphAt(b)]; + const ids = new Set(sides.flatMap((g) => [...g.records.keys()])); + const merged = new Map(); + for (const id of ids) { + const [ra, rb] = sides.map((g) => g.records.get(id)); + const r0 = base.records.get(id); + const [sa, sb, s0] = [ra, rb, r0].map((rec) => rec && standing(rec)); + let standings; + if (!ra || !rb) standings = [sa ?? sb]; + else if (sa === s0) standings = [sb]; + else if (sb === s0 || sa === sb) standings = [sa]; + else standings = [sa, sb]; + merged.set(id, { rec: ra ?? rb, sides: [ra, rb], standings }); + } + const supersessors = new Map(); + for (const [id, { sides: both }] of merged) + for (const rec of both) + for (const target of list(rec?.frontmatter.supersedes)) { + if (!supersessors.has(target)) supersessors.set(target, new Set()); + supersessors.get(target).add(id); + } + const recs = [...merged.values()]; + if (kind === 'state-differs') + return recs.filter((m) => m.standings.length > 1).map((m) => m.rec); + if (kind === 'double-supersede') + return recs + .filter((m) => (supersessors.get(m.rec.id)?.size ?? 0) > 1) + .map((m) => m.rec); + if (kind === 'double-accept') + return recs + .filter( + (m) => m.rec.kind === 'issue' && m.rec.frontmatter.kind === 'question' + ) + .filter( + (q) => + recs.filter( + (d) => + d.rec.kind === 'decision' && + d.standings.includes('accepted') && + list(d.rec.frontmatter.derives_from).includes(q.rec.id) + ).length > 1 + ) + .map((m) => m.rec); + if (kind === 'retracted-relied') { + // When one ref already contains the other, nothing diverged. + if ( + bed.succeeds('merge-base', '--is-ancestor', a, b) || + bed.succeeds('merge-base', '--is-ancestor', b, a) + ) + return []; + // A relying record counts only if it is still live after the merge. + const liveAfterMerge = (r) => + r.standings.every((s) => isLiveStanding(r.rec.kind, s)); + const reliesOn = (r, id) => + r.sides.some((rec) => list(rec?.frontmatter.derives_from).includes(id)); + return recs + .filter((m) => + [0, 1].some((side) => { + const [mine, theirs] = [m.sides[side], m.sides[1 - side]]; + return ( + mine?.frontmatter.retracted === true && + theirs && + theirs.frontmatter.retracted !== true + ); + }) + ) + .filter((m) => + recs.some((r) => liveAfterMerge(r) && reliesOn(r, m.rec.id)) + ) + .map((m) => m.rec); + } + if (kind === 'compaction-fork') { + const heads = recs.filter( + (m) => + isCompaction(m.rec) && + !m.standings.includes('retracted') && + !supersessors.has(m.rec.id) + ); + const byEffort = new Map(); + for (const m of heads) { + const effort = m.rec.frontmatter.effort; + byEffort.set(effort, [...(byEffort.get(effort) ?? []), m.rec]); + } + return [...byEffort.values()].filter((group) => group.length > 1).flat(); + } + throw new Error(`bad conflict kind ${kind}`); +} diff --git a/benchmarks/proof-recall/spec/repo.mjs b/benchmarks/proof-recall/spec/repo.mjs new file mode 100644 index 00000000..630f71dd --- /dev/null +++ b/benchmarks/proof-recall/spec/repo.mjs @@ -0,0 +1,1703 @@ +// The git test bed for the code-context and branch questions: a small +// codebase for Saltglass Reach, the committed 1x graph, records that anchor +// Proof to the code, and four engineers' branches with draft PRs. +// scripts/build-repo.mjs replays it into a real repository. +// +// A commit has: +// key a local name; record bodies and anchors refer to its hash as +// {{sha:}}, and questions find it by `subject` +// at the author and commit date +// author a person in PEOPLE +// subject the first line of the message; unique in the repository +// body optional message text after the subject +// proof record keys for `Proof: ` trailers +// files { path: text } to write, or { path: null } to delete +// graph true to copy the committed benchmark graph in +// events Proof mutations, replayed through the real writer +// +// Events use the build.mjs spec format with two changes. Every key is +// qualified as `.`, since one commit can touch several Efforts, +// and a create event names its Effort in `effort`. One extra type exists: +// +// { type: 'Anchor', effort, key, title, path, symbol, commit, created_by } +// +// writes a Citation with `role: code` whose body is `#@`, +// the 12-character hash of `commit`. These are the spec's declared code +// anchors; check.mjs refuses a code anchor in the graph that is not one. +// +// Branches start at the tip of main. `pr` is the draft PR that +// .flatbread/proof/prs.json maps the branch to. + +export const PEOPLE = { + ines: { name: 'Ines Okafor', login: 'ines-okafor' }, + tomasz: { name: 'Tomasz Brandt', login: 'tomasz-brandt' }, + yusuf: { name: 'Yusuf Demir', login: 'yusuf-demir' }, + rhea: { name: 'Rhea Valdés', login: 'rhea-valdes' }, + kenji: { name: 'Kenji Arakawa', login: 'kenji-arakawa' }, + marta: { name: 'Marta Lindqvist', login: 'marta-lindqvist' }, + priya: { name: 'Priya Natarajan', login: 'priya-natarajan' }, + liam: { name: "Liam O'Connell", login: 'liam-oconnell' }, + sofia: { name: 'Sofia Marchetti', login: 'sofia-marchetti' }, + dara: { name: 'Dara Whitlock', login: 'dara-whitlock' }, +}; + +const at = (text) => `${text.replace(' ', 'T')}:00Z`; + +// ---------------------------------------------------------------- code ---- + +const README_1 = `# Saltglass Reach + +Engine, platform and tools source for Saltglass Reach, Tidewater Forge's +co-op exploration game. + +| Path | What lives there | +| ------------------------------------- | -------------------------------------- | +| \`engine/kestrel/\` | Kestrel: streaming, tides, saves, ECS | +| \`engine/lantern/\` | Lantern, Kestrel's renderer | +| \`platform/skiff/\`, \`platform/meridian/\` | Console ports | +| \`net/coop/\` | Co-op sync | +| \`tools/shadercache/\` | The build farm's shader compile cache | +`; + +const README_2 = `${README_1} +## Project memory + +Decisions, findings and open questions live in \`.flatbread-proof/\`. Read +them with \`flatbread proof\`, never by opening the files. A commit names the +records it implements with one \`Proof: \` trailer per record. +`; + +const FILES_M01 = { + 'README.md': README_1, + 'Cargo.toml': `[workspace] +members = ["engine/*", "platform/*", "net/*"] +resolver = "2" +`, + '.gitignore': `/target/ +/.flatbread/ +.flatbread-proof/.journal/ +`, + 'engine/kestrel/lib.rs': `//! Kestrel, the Saltglass Reach engine. +pub mod ecs; +pub mod gull; +pub mod jobs; +pub mod save; +pub mod streaming; +pub mod tides; +`, + 'engine/lantern/lib.rs': `//! Lantern, Kestrel's renderer. +pub mod cull; +pub mod gi; +pub mod shadows; +pub mod tern; +pub mod tonemap; +pub mod vb; +`, + 'engine/kestrel/ecs/names.rs': `//! Debug names for entities. Release builds compile them out. +use super::Entity; + +#[cfg(debug_assertions)] +pub fn debug_name(entity: Entity) -> &'static str { + NAMES.with(|names| names.get(entity)) +} + +#[cfg(not(debug_assertions))] +pub fn debug_name(_entity: Entity) -> &'static str { + "" +} +`, + 'engine/kestrel/ecs/storage.rs': `//! Component storage. +pub struct ArchetypeTable { + pub columns: Vec, + pub len: usize, +} + +/// Tags that come and go every few frames live outside the tables. +pub struct SparseTags { + pub dense: Vec, + pub sparse: Vec, +} +`, + 'engine/kestrel/jobs/pool.rs': `//! The engine job scheduler. +pub struct ThreadPool { + workers: Vec, +} + +impl ThreadPool { + pub fn spawn(&self, job: Job) -> JobHandle { + self.workers[self.pick()].push(job) + } +} +`, + 'engine/kestrel/gull/vm.rs': `//! Runs Gull scripts. +pub struct Interpreter { + pub env: Env, +} + +impl Interpreter { + pub fn call(&mut self, func: &FuncDef, args: &[Value]) -> Value { + self.eval_block(&func.body, args) + } +} +`, + 'engine/lantern/shadows/local_atlas.rs': `//! Shadows for local lights: one 8k atlas, cached until something moves. +pub const ATLAS_SIZE: u32 = 8192; + +pub struct LocalShadowAtlas { + slots: Vec, +} +`, + 'engine/lantern/gi/sdf_volumes.rs': `//! Signed-distance-field volumes for global illumination. +pub const CASCADES: usize = 4; + +pub struct SdfVolumes { + cascades: [Cascade; CASCADES], +} + +impl SdfVolumes { + pub fn update(&mut self, camera: &Camera) { + for cascade in &mut self.cascades { + cascade.recenter(camera.position); + } + } +} +`, + 'engine/lantern/cull/two_phase.rs': `//! GPU occlusion culling in two phases. +pub fn cull_phase_one(instances: &InstanceBuffer, hzb: &Hzb) -> VisibleList { + instances.test_against(hzb) +} + +pub fn cull_phase_two(rejected: &InstanceBuffer, hzb: &Hzb) -> VisibleList { + rejected.test_against(hzb) +} +`, + 'platform/skiff/frame_modes.rs': `//! Display modes on Skiff. +pub enum FrameMode { + Hz30, +} +`, +}; + +const KEY_V1 = `"""Cache keys for the farm's shader compile cache.""" +import hashlib + + +def cache_key(source_path, compiler_version): + """SHA-256 of the shader file text and the compiler version string.""" + digest = hashlib.sha256() + with open(source_path, "rb") as source: + digest.update(source.read()) + digest.update(compiler_version.encode()) + return digest.hexdigest() +`; + +const KEY_V2 = `"""Cache keys for the farm's shader compile cache.""" +import hashlib + +from .preprocess import preprocess + + +def cache_key(source_path, defines, target, compiler_version): + """SHA-256 of the preprocessed source, defines, compiler version and target.""" + digest = hashlib.sha256() + digest.update(preprocess(source_path, defines)) + for name, value in sorted(defines.items()): + digest.update(f"{name}={value}".encode()) + digest.update(compiler_version.encode()) + digest.update(target.encode()) + return digest.hexdigest() +`; + +const KEY_V3 = `"""Cache keys for the farm's shader compile cache.""" +import hashlib + +from .compiler import compiler_binary_hash +from .preprocess import normalize_flags, preprocess + + +def cache_key(source_path, defines, target, flags): + """SHA-256 of the key inputs, in a fixed order. + + The compiler is identified by the hash of its binary and optimizer + library, computed once per agent at startup, not by its version string. + """ + digest = hashlib.sha256() + digest.update(preprocess(source_path, defines)) + for name, value in sorted(defines.items()): + digest.update(f"{name}={value}".encode()) + digest.update(target.encode()) + digest.update(compiler_binary_hash()) + digest.update(" ".join(normalize_flags(flags)).encode()) + return digest.hexdigest() +`; + +const IO_QUEUE_1 = `//! One async read queue per platform for streamed assets. +pub enum Priority { + Proximity, + Background, +} + +pub struct IoQueue { + lanes: [VecDeque; 2], +} + +impl IoQueue { + pub fn push_read(&mut self, request: ReadRequest, priority: Priority) { + self.lanes[priority as usize].push_back(request); + } + + pub fn next(&mut self) -> Option { + self.lanes.iter_mut().find_map(|lane| lane.pop_front()) + } +} +`; + +const IO_QUEUE_2 = `//! One async read queue per platform for streamed assets. +pub enum Priority { + Tide, + Proximity, + Background, +} + +pub struct IoQueue { + lanes: [VecDeque; 3], +} + +impl IoQueue { + pub fn push_read(&mut self, request: ReadRequest, priority: Priority) { + self.lanes[priority as usize].push_back(request); + } + + /// Reads for cells the tide is about to expose go ahead of everything. + pub fn push_tide_read(&mut self, request: ReadRequest) { + self.lanes[Priority::Tide as usize].push_front(request); + } + + pub fn next(&mut self) -> Option { + self.lanes.iter_mut().find_map(|lane| lane.pop_front()) + } +} +`; + +const POOL_1 = `//! The streaming pool: which cells stay resident. +pub struct CellPool { + resident: Vec, + budget_bytes: u64, +} + +impl CellPool { + /// Evicts the cell touched longest ago. + pub fn evict_lru(&mut self) -> Option { + let oldest = self.resident.iter().min_by_key(|cell| cell.last_touched)?; + Some(oldest.id) + } +} +`; + +const POOL_2 = `//! The streaming pool: which cells stay resident. +pub struct CellPool { + resident: Vec, + budget_bytes: u64, +} + +const HALF_TURN_TICKS: u32 = 1_200; + +impl CellPool { + /// Lower scores go first. Cells the tide forecast needs soon are kept. + pub fn evict_score(&self, cell: &ResidentCell, forecast: &TideForecast) -> i64 { + if forecast.exposes_within(cell.id, HALF_TURN_TICKS) { + return i64::MAX; + } + -(cell.distance_to_nearest_player as i64) + } + + pub fn evict(&mut self, forecast: &TideForecast) -> Option { + let cell = self.resident.iter().min_by_key(|c| self.evict_score(c, forecast))?; + Some(cell.id) + } +} +`; + +const POOL_3 = `//! The streaming pool: which cells stay resident. +pub struct CellPool { + resident: Vec, + budget_bytes: u64, +} + +const TURN_TICKS: u32 = 2_400; + +impl CellPool { + /// Lower scores go first. Cells the tide exposes within one turn are kept. + pub fn evict_score(&self, cell: &ResidentCell, forecast: &TideForecast) -> i64 { + if forecast.exposes_within(cell.id, TURN_TICKS) { + return i64::MAX; + } + -(cell.distance_to_nearest_player as i64) + } + + pub fn evict(&mut self, forecast: &TideForecast) -> Option { + let cell = self.resident.iter().min_by_key(|c| self.evict_score(c, forecast))?; + Some(cell.id) + } +} +`; + +const TIDES_MOD_1 = `//! The tide simulation. Steps at 10 Hz; every peer computes the same bits. +pub mod fixed; +pub mod phase; +mod sine_table; +`; + +const TIDES_MOD_LOD = `//! The tide simulation. Steps at 10 Hz; every peer computes the same bits. +pub mod fixed; +pub mod lod; +pub mod phase; +mod sine_table; +`; + +const FIXED = `//! 32.32 fixed-point arithmetic for the tide simulation. +#[derive(Clone, Copy, PartialEq, Eq)] +pub struct Fixed32x32(pub i64); + +impl Fixed32x32 { + pub const ONE: Self = Self(1 << 32); + + pub fn mul(self, other: Self) -> Self { + Self(((self.0 as i128 * other.0 as i128) >> 32) as i64) + } + + pub fn sin(self) -> Self { + super::sine_table::lookup(self) + } +} + +/// One tide step over the preallocated height field. +pub fn step_tide(heights: &mut HeightField, phase: Fixed32x32) { + for cell in heights.cells_mut() { + cell.height = cell.amplitude.mul(phase.sin()); + } +} +`; + +const SINE_TABLE = `//! Generated at build time by tools/gen_sine_table.py. Do not edit. +pub const ENTRIES: usize = 4096; + +pub static TABLE: [i64; ENTRIES] = include!("sine_table.in"); + +pub fn lookup(angle: super::fixed::Fixed32x32) -> super::fixed::Fixed32x32 { + super::fixed::Fixed32x32(interpolate(&TABLE, angle.0)) +} +`; + +const PHASE_1 = `//! The tide phase, as a fraction of the 40-minute cycle. +use super::fixed::Fixed32x32; + +pub struct TidePhase { + pub main: Fixed32x32, +} + +/// Wraps the accumulator at each cycle so it never holds more than one. +pub fn wrap_phase(phase: Fixed32x32) -> Fixed32x32 { + Fixed32x32(phase.0 & (Fixed32x32::ONE.0 - 1)) +} +`; + +const PHASE_2 = `//! The tide phase, as a fraction of the 40-minute cycle. +use super::fixed::Fixed32x32; + +pub struct TidePhase { + pub main: Fixed32x32, + /// Second harmonic, seven cycles long: spring and neap tides. + pub harmonic: Fixed32x32, +} + +/// Wraps the accumulator at each cycle so it never holds more than one. +pub fn wrap_phase(phase: Fixed32x32) -> Fixed32x32 { + Fixed32x32(phase.0 & (Fixed32x32::ONE.0 - 1)) +} + +pub fn harmonic_phase(main_cycles: u64, phase: Fixed32x32) -> Fixed32x32 { + wrap_phase(Fixed32x32(((main_cycles % 7) as i64 * Fixed32x32::ONE.0 + phase.0) / 7)) +} +`; + +const VB_1 = `//! Material resolve from the visibility buffer. +/// Foliage and saltglass still draw in the forward pass. +pub const FORWARD_PASSES: &[MaterialClass] = &[MaterialClass::Foliage, MaterialClass::Saltglass]; + +pub fn resolve_materials(vb: &VisibilityBuffer, tables: &MaterialTables, out: &mut GBufferLite) { + for tile in vb.tiles() { + out.write(tile, tables.shade(vb.triangle(tile))); + } +} +`; + +const VB_2 = `//! Material resolve from the visibility buffer, on every platform. +/// Saltglass still draws in the forward pass. +pub const FORWARD_PASSES: &[MaterialClass] = &[MaterialClass::Saltglass]; + +/// Skiff resolves materials every other frame and reuses the last result. +pub const SKIFF_HALF_RATE: bool = true; + +pub fn resolve_materials(vb: &VisibilityBuffer, tables: &MaterialTables, out: &mut GBufferLite, frame: u64) { + if cfg!(target_platform = "skiff") && SKIFF_HALF_RATE && frame % 2 == 1 { + return; + } + for tile in vb.tiles() { + out.write(tile, tables.shade(vb.triangle(tile))); + } +} +`; + +const PSO_CACHE = `//! The on-device pipeline cache for Skiff. Players' machines compile each +//! pipeline once and keep it in the save partition. +pub struct PipelineCache { + file: CacheFile, + driver_version: u32, +} + +impl PipelineCache { + pub fn get_or_compile(&mut self, desc: &PipelineDesc) -> Pipeline { + let key = (desc.hash(), self.driver_version); + self.file.get(key).unwrap_or_else(|| self.compile_and_store(desc, key)) + } + + /// Compiles every pipeline on a district's list while a load screen is up. + pub fn warm_district(&mut self, district: &DistrictPipelines) { + for desc in district.pipelines() { + self.get_or_compile(desc); + } + } +} +`; + +const VSM_POOL_1 = `//! The virtual shadow map page pool for sun shadows. +pub const PAGE_TEXELS: u32 = 128; +pub const POOL_PAGES: usize = 4096; + +pub struct PagePool { + pages: Vec, + moving: Vec, +} + +impl PagePool { + /// Frees pages for this frame's requests. Static pages go oldest-sampled + /// first; pages in the moving-object set are never evicted. + pub fn evict_pages(&mut self, needed: usize) -> usize { + let mut freed = 0; + let mut candidates: Vec<_> = self.pages.iter().filter(|p| !p.moving).collect(); + candidates.sort_by_key(|p| p.last_sampled); + for page in candidates.into_iter().take(needed) { + self.release(page.id); + freed += 1; + } + freed + } +} +`; + +const VSM_POOL_GROW = `//! The virtual shadow map page pool for sun shadows. +pub const PAGE_TEXELS: u32 = 128; +pub const POOL_PAGES: usize = 4096; +/// Pool size near spring low water, paid for by the far GI cascade. +pub const SPRING_LOW_POOL_PAGES: usize = 6144; + +pub struct PagePool { + pages: Vec, + moving: Vec, + capacity: usize, +} + +impl PagePool { + /// Frees pages for this frame's requests. Static pages go oldest-sampled + /// first; pages in the moving-object set are never evicted. + pub fn evict_pages(&mut self, needed: usize) -> usize { + let mut freed = 0; + let mut candidates: Vec<_> = self.pages.iter().filter(|p| !p.moving).collect(); + candidates.sort_by_key(|p| p.last_sampled); + for page in candidates.into_iter().take(needed) { + self.release(page.id); + freed += 1; + } + freed + } + + /// Grows the pool before spring low water and shrinks it after mid flood. + pub fn grow_for_tide(&mut self, forecast: &TideForecast, gi: &mut SdfVolumes) { + if forecast.spring_low_within_one_step() && self.capacity == POOL_PAGES { + gi.lend_far_cascade(); + self.capacity = SPRING_LOW_POOL_PAGES; + } else if forecast.past_mid_flood() && self.capacity == SPRING_LOW_POOL_PAGES { + self.capacity = POOL_PAGES; + gi.reclaim_far_cascade(); + } + } +} +`; + +const VSM_POOL_COARSE = `//! The virtual shadow map page pool for sun shadows. +pub const PAGE_TEXELS: u32 = 128; +pub const POOL_PAGES: usize = 4096; +/// Exposed wet sand gets pages one mip level coarser. +pub const WET_SAND_PAGE_LEVEL_BIAS: u32 = 1; + +pub struct PagePool { + pages: Vec, + moving: Vec, +} + +impl PagePool { + /// Frees pages for this frame's requests. Fine pages on exposed wet sand + /// go first, then static pages oldest-sampled first; pages in the + /// moving-object set are never evicted. + pub fn evict_pages(&mut self, needed: usize) -> usize { + let mut freed = 0; + let mut candidates: Vec<_> = self.pages.iter().filter(|p| !p.moving).collect(); + candidates.sort_by_key(|p| (!p.on_wet_sand, p.last_sampled)); + for page in candidates.into_iter().take(needed) { + self.release(page.id); + freed += 1; + } + freed + } + + pub fn page_level(&self, tile: &ShadowTile) -> u32 { + tile.level + if tile.on_wet_sand { WET_SAND_PAGE_LEVEL_BIAS } else { 0 } + } +} +`; + +const SDF_VOLUMES_LEND = `//! Signed-distance-field volumes for global illumination. +pub const CASCADES: usize = 4; + +pub struct SdfVolumes { + cascades: [Cascade; CASCADES], + far_lent: bool, +} + +impl SdfVolumes { + pub fn update(&mut self, camera: &Camera) { + let live = if self.far_lent { CASCADES - 1 } else { CASCADES }; + for cascade in &mut self.cascades[..live] { + cascade.recenter(camera.position); + } + } + + /// Stops updating the far cascade and hands its 128 MB to the shadow pool. + pub fn lend_far_cascade(&mut self) { + self.far_lent = true; + } + + pub fn reclaim_far_cascade(&mut self) { + self.far_lent = false; + } +} +`; + +const PRESETS_1 = `//! Tern quality presets. +pub enum TernPreset { + Quality, + Balanced, + Performance, +} + +impl TernPreset { + pub fn input_scale(self) -> f32 { + match self { + TernPreset::Quality => 0.67, + TernPreset::Balanced => 0.58, + TernPreset::Performance => 0.5, + } + } +} +`; + +const PRESETS_2 = `//! Tern quality presets. +pub enum TernPreset { + Quality, + Balanced, + Performance, + /// For the Meridian 120 Hz performance mode. + UltraPerformance, +} + +impl TernPreset { + pub fn input_scale(self) -> f32 { + match self { + TernPreset::Quality => 0.67, + TernPreset::Balanced => 0.58, + TernPreset::Performance => 0.5, + TernPreset::UltraPerformance => 0.33, + } + } +} +`; + +const REMOTE = `"""Remote execution for shader cache misses.""" +MAX_JOBS_PER_AGENT = 6 +FETCH_TIMEOUT_S = 30 + + +def fan_out_misses(misses, agents): + """Sends each cache miss to the least busy farm agent.""" + for miss in misses: + agent = min(agents, key=lambda a: a.running) + if agent.running < MAX_JOBS_PER_AGENT: + agent.submit(miss) + + +def fetch(entry, store): + try: + return store.get(entry, timeout=FETCH_TIMEOUT_S) + except TimeoutError: + return store.get(entry, timeout=FETCH_TIMEOUT_S) +`; + +const UPGRADE_1 = `//! Save upgrades: one step per schema version, run in order on load. +pub const SAVE_SCHEMA_VERSION: u32 = 1; + +type Step = fn(SaveBlob) -> SaveBlob; + +/// Step i upgrades a save from version i to i + 1. Never edit a shipped step. +pub fn upgrade_steps() -> &'static [Step] { + &[stamp_version] +} + +fn stamp_version(save: SaveBlob) -> SaveBlob { + save.with_header_version(1) +} + +pub fn upgrade(mut save: SaveBlob) -> SaveBlob { + for step in &upgrade_steps()[save.version() as usize..] { + save = step(save); + } + save +} +`; + +const HOST_SYNC = `//! Host-authoritative co-op sync. The host runs the only tide simulation; +//! clients send inputs and predict their own player. +pub struct HostSync { + peers: Vec, +} + +impl HostSync { + pub fn send_snapshot(&mut self, world: &WorldState) { + for peer in &self.peers { + self.transport.send(*peer, world.snapshot_for(*peer)); + } + } +} +`; + +const CHECKSUM = `//! Desync detection: hash tide, physics and Gull state separately. +pub const HASH_INTERVAL_FRAMES: u64 = 30; + +pub fn hash_blocks(frame: u64, world: &WorldState) -> Option<[u32; 3]> { + if frame % HASH_INTERVAL_FRAMES != 0 { + return None; + } + Some([crc32(world.tide()), crc32(world.physics()), crc32(world.gull())]) +} +`; + +const HISTORY = `//! Tern's history buffer. +/// Where saltglass sets the reactive mask, trust the current frame. +pub fn apply_reactive_mask(history: &mut HistoryBuffer, mask: &ReactiveMask) { + for (texel, reactive) in history.texels_mut().zip(mask.values()) { + if *reactive > 0 { + texel.weight = 0.0; + } + } +} + +pub fn clamp_history(history: &mut HistoryBuffer, current: &Frame) { + for (texel, neighborhood) in history.texels_mut().zip(current.neighborhoods()) { + if texel.weight > 0.0 { + texel.color = neighborhood.clamp(texel.color); + } + } +} +`; + +const DYNRES = `//! Dynamic resolution on Skiff. Tern upscales to 720p. +pub const FLOOR_LINES: u32 = 504; +pub const CEILING_LINES: u32 = 720; +pub const HOLD_FRAMES: u32 = 60; + +pub fn next_lines(current: u32, gpu_ms: f32, budget_ms: f32, held: u32) -> u32 { + if gpu_ms > budget_ms { + return FLOOR_LINES; + } + if held < HOLD_FRAMES { + return current; + } + (current + 24).min(CEILING_LINES) +} +`; + +const STORM_1 = `//! Tide-storm particles. +pub const STORM_PARTICLES: u32 = 14_000; + +pub fn storm_particle_budget(_platform: Platform) -> u32 { + STORM_PARTICLES +} +`; + +const STORM_CAP = `//! Tide-storm particles. +pub const STORM_PARTICLES: u32 = 14_000; +pub const STORM_PARTICLES_SKIFF: u32 = 6_000; +/// Skiff skips tide collision for particles farther than this. +pub const TIDE_COLLISION_RANGE_M: f32 = 40.0; + +pub fn storm_particle_budget(platform: Platform) -> u32 { + match platform { + Platform::Skiff => STORM_PARTICLES_SKIFF, + _ => STORM_PARTICLES, + } +} +`; + +const BRINE_1 = `//! The Brine tonemapping curve. +/// Where the HDR shoulder starts, in nits. +pub const HDR_SHOULDER_NITS: f32 = 1000.0; + +pub fn brine_curve(x: f32, peak_nits: f32) -> f32 { + let toe = x * x / (x + 0.18); + shoulder(toe, HDR_SHOULDER_NITS.min(peak_nits), peak_nits) +} +`; + +// The code lags behind the Decision in force (render.dec-brine-3, 800 nits). +const BRINE_2 = `//! The Brine tonemapping curve. +/// Where the HDR shoulder starts, in nits. +pub const HDR_SHOULDER_NITS: f32 = 1000.0; + +pub fn brine_curve(x: f32, peak_nits: f32) -> f32 { + // Negative scene values come from the bloom pass; they would bend the toe. + let x = x.max(0.0); + let toe = x * x / (x + 0.18); + shoulder(toe, HDR_SHOULDER_NITS.min(peak_nits), peak_nits) +} +`; + +const LOD_QUARTER = `//! Tide level of detail: cells far from every player step less often. +use super::fixed::Fixed32x32; + +pub const FAR_CELL_M: u32 = 600; +pub const FAR_STEP_DIVISOR: u64 = 4; + +/// Far cells step on every fourth tick, four ticks of phase at a time. +/// Every peer picks the same cells, so tide state stays bit-identical. +pub fn step_far_cells(heights: &mut HeightField, tick: u64, phase: Fixed32x32) { + if tick % FAR_STEP_DIVISOR != 0 { + return; + } + for cell in heights.cells_beyond_mut(FAR_CELL_M) { + cell.height = cell.amplitude.mul(phase.sin()); + } +} +`; + +const LOD_FREEZE = `//! Tide level of detail: cells far from every player stop stepping. +use super::fixed::Fixed32x32; + +pub const FAR_CELL_M: u32 = 600; + +/// Far cells keep the tick they froze at. +pub fn freeze_far_cells(heights: &mut HeightField, tick: u64) { + for cell in heights.cells_beyond_mut(FAR_CELL_M) { + cell.frozen_at.get_or_insert(tick); + } +} + +/// Runs a cell's missed ticks in one batch when a player comes within range. +pub fn catch_up(cell: &mut TideCell, tick: u64, phase_at: impl Fn(u64) -> Fixed32x32) { + if let Some(from) = cell.frozen_at.take() { + cell.height = cell.amplitude.mul(phase_at(tick).sin()); + cell.missed_ticks = tick - from; + } +} +`; + +const FLATBREAD_CONFIG = `import { source as filesystem } from '@flatbread/source-filesystem'; +import { transformer as markdownTransformer } from '@flatbread/transformer-markdown'; +import { defineConfig } from '@flatbread/config'; +import { proofContent } from '@flatbread/proof'; + +export default defineConfig({ + source: filesystem(), + transformer: markdownTransformer(), + content: proofContent(), +}); +`; + +// ------------------------------------------------------------- history ---- + +const main = [ + { + key: 'm01', + at: at('2025-09-02 10:00'), + author: 'rhea', + subject: 'Lay out the engine, platform and tools source tree', + files: FILES_M01, + }, + { + key: 'm02', + at: at('2025-09-24 16:00'), + author: 'liam', + subject: 'Key the shader cache on source text and compiler version', + proof: ['tools.dec-key-v1'], + files: { + 'tools/shadercache/__init__.py': + '"""The build farm\'s shader compile cache."""\n', + 'tools/shadercache/key.py': KEY_V1, + 'tools/shadercache/README.md': `# Shader compile cache + +A content-addressed cache of compiled shaders, shared by every farm agent. +\`key.py\` builds the cache key; \`remote.py\` sends misses to other agents. +`, + }, + }, + { + key: 'm03', + at: at('2025-10-01 11:00'), + author: 'kenji', + subject: 'Read streamed assets through one request queue per platform', + proof: ['world.dec-io-queue'], + files: { 'engine/kestrel/streaming/io_queue.rs': IO_QUEUE_1 }, + }, + { + key: 'm04', + at: at('2025-10-17 15:00'), + author: 'liam', + subject: 'Hash preprocessed source, defines and target in the cache key', + body: 'Included headers were not part of the key, so a header change could serve a stale shader.', + proof: ['tools.dec-key-v2'], + files: { 'tools/shadercache/key.py': KEY_V2 }, + }, + { + key: 'm05', + at: at('2025-10-28 14:00'), + author: 'kenji', + subject: 'Evict streaming cells least recently used', + proof: ['world.dec-pool-lru'], + files: { 'engine/kestrel/streaming/pool.rs': POOL_1 }, + }, + { + key: 'm06', + at: at('2025-11-03 12:00'), + author: 'rhea', + subject: 'Run the tide simulation in 32.32 fixed point', + proof: ['world.dec-tide-fixed'], + files: { + 'engine/kestrel/tides/mod.rs': TIDES_MOD_1, + 'engine/kestrel/tides/fixed.rs': FIXED, + 'engine/kestrel/tides/sine_table.rs': SINE_TABLE, + }, + }, + { + key: 'm07', + at: at('2025-11-13 11:00'), + author: 'rhea', + subject: 'Wrap the tide phase at each 40-minute cycle', + proof: ['world.dec-tide-wrap'], + files: { 'engine/kestrel/tides/phase.rs': PHASE_1 }, + }, + { + key: 'm08', + at: at('2025-11-15 17:00'), + author: 'tomasz', + subject: 'Resolve opaque materials from the visibility buffer', + proof: ['render.dec-vb-3'], + files: { 'engine/lantern/vb/resolve.rs': VB_1 }, + }, + { + key: 'm09', + at: at('2025-12-13 15:00'), + author: 'priya', + subject: 'Keep compiled pipelines in an on-device cache on Skiff', + proof: ['ports.dec-pso-ondevice'], + files: { 'platform/skiff/pso_cache.rs': PSO_CACHE }, + }, + { + key: 'm10', + at: at('2026-01-13 16:00'), + author: 'kenji', + subject: 'Evict streaming cells by distance and tide forecast', + proof: ['world.dec-pool-forecast'], + files: { 'engine/kestrel/streaming/pool.rs': POOL_2 }, + }, + { + key: 'm11', + at: at('2026-01-16 10:00'), + author: 'liam', + subject: 'Hash the compiler binary instead of its version string', + body: 'Also normalize compile flags, so flag order no longer changes the key. The first build after this runs cold.', + files: { 'tools/shadercache/key.py': KEY_V3 }, + }, + { + key: 'm12', + at: at('2026-02-18 15:00'), + author: 'tomasz', + subject: 'Add the virtual shadow map page pool', + proof: ['render.dec-vsm'], + files: { 'engine/lantern/shadows/vsm_pool.rs': VSM_POOL_1 }, + }, + { + key: 'm13', + at: at('2026-02-26 11:00'), + author: 'ines', + subject: 'Ship three Tern quality presets', + proof: ['render.dec-presets-3'], + files: { 'engine/lantern/tern/presets.rs': PRESETS_1 }, + }, + { + key: 'm14', + at: at('2026-02-27 14:00'), + author: 'liam', + subject: 'Fan cache misses out to farm agents, six jobs per agent', + proof: ['tools.dec-remote', 'tools.dec-cap-jobs'], + files: { 'tools/shadercache/remote.py': REMOTE }, + }, + { + key: 'm15', + at: at('2026-03-13 10:00'), + author: 'marta', + subject: 'Stamp saves with a schema version and upgrade them on load', + proof: ['world.dec-save-version'], + files: { 'engine/kestrel/save/upgrade.rs': UPGRADE_1 }, + }, + { + key: 'm16', + at: at('2026-04-03 16:00'), + author: 'sofia', + subject: 'Sync co-op from the host and hash state blocks every 30 frames', + proof: ['netcode.dec-host', 'netcode.dec-checksum'], + files: { + 'net/coop/host_sync.rs': HOST_SYNC, + 'net/coop/checksum.rs': CHECKSUM, + }, + }, + { + key: 'm17', + at: at('2026-05-07 12:00'), + author: 'ines', + subject: 'Give the Brine curve a separate HDR shoulder', + proof: ['render.dec-brine-2'], + files: { 'engine/lantern/tonemap/brine.rs': BRINE_1 }, + }, + { + key: 'm18', + at: at('2026-05-12 13:00'), + author: 'kenji', + subject: 'Put reads for tide-exposed cells at the head of the queue', + proof: ['world.dec-io-tide-priority'], + files: { 'engine/kestrel/streaming/io_queue.rs': IO_QUEUE_2 }, + }, + { + key: 'm19', + at: at('2026-05-19 15:00'), + author: 'ines', + subject: 'Feed the saltglass reactive mask into Tern', + proof: ['render.dec-tern-reactive'], + files: { 'engine/lantern/tern/history.rs': HISTORY }, + }, + { + key: 'm20', + at: at('2026-06-11 10:00'), + author: 'ines', + subject: 'Add the Ultra Performance Tern preset', + proof: ['render.dec-presets-4b'], + files: { 'engine/lantern/tern/presets.rs': PRESETS_2 }, + }, + { + key: 'm21', + at: at('2026-07-07 11:00'), + author: 'rhea', + subject: 'Add the second tide harmonic', + proof: ['world.dec-tide-harmonic'], + files: { 'engine/kestrel/tides/phase.rs': PHASE_2 }, + }, + { + key: 'm22', + at: at('2026-07-09 16:00'), + author: 'tomasz', + subject: 'Resolve Skiff materials at half rate and bring foliage back', + files: { 'engine/lantern/vb/resolve.rs': VB_2 }, + }, + { + key: 'm23', + at: at('2026-07-23 10:00'), + author: 'priya', + subject: 'Lower the Skiff dynamic resolution floor to 504p', + proof: ['ports.dec-dynres-3'], + files: { 'platform/skiff/dynres.rs': DYNRES }, + }, + { + key: 'm24', + at: at('2026-08-12 15:00'), + author: 'priya', + subject: 'Add a per-platform tide-storm particle budget', + files: { + 'platform/skiff/storm.rs': STORM_1, + 'platform/skiff/frame_modes.rs': `//! Display modes on Skiff. +pub enum FrameMode { + Hz30, + /// 25 ms per frame. + Hz40, +} +`, + }, + }, + { + key: 'm25', + at: at('2026-09-09 12:00'), + author: 'ines', + subject: 'Clamp negative scene values before the Brine curve', + files: { 'engine/lantern/tonemap/brine.rs': BRINE_2 }, + }, + { + key: 'm26', + at: at('2026-09-26 10:00'), + author: 'dara', + subject: 'Move the Proof graph into the engine repository', + body: 'Project memory used to live in its own repository. Keeping it here lets commits and records name each other.', + graph: true, + files: { 'README.md': README_2, 'flatbread.config.js': FLATBREAD_CONFIG }, + }, + { + key: 'm27', + at: at('2026-09-28 10:00'), + author: 'rhea', + subject: 'Anchor Proof records to the code that implements them', + events: [ + { + type: 'Anchor', + effort: 'render', + key: 'cit-code-vsm-evict', + title: 'Code: VSM page pool eviction', + path: 'engine/lantern/shadows/vsm_pool.rs', + symbol: 'evict_pages', + commit: 'm12', + created_by: 'Tomasz Brandt', + }, + { + type: 'WriteFinding', + effort: 'render', + key: 'fnd-impl-vsm-evict', + kind: 'implementation', + title: + 'Sun shadow pages are evicted oldest-sampled first, never from the moving set', + created_by: 'Tomasz Brandt', + derives_from: ['render.dec-vsm'], + cites: ['render.cit-code-vsm-evict'], + body: + 'The page pool frees pages when a frame asks for more than the 4,096-page pool has free. Static pages go in the order they were last sampled, oldest first. ' + + 'Pages in the moving-object set are never evicted, since they are redrawn every frame anyway. When no static page is left to drop, the tile falls back to the coarse distance-field shadow. ' + + 'This is the path that runs dry on the flats at spring low tide.', + }, + { + type: 'Anchor', + effort: 'render', + key: 'cit-code-tern-reactive', + title: 'Code: Tern reactive mask', + path: 'engine/lantern/tern/history.rs', + symbol: 'apply_reactive_mask', + commit: 'm19', + created_by: 'Ines Okafor', + }, + { + type: 'WriteFinding', + effort: 'render', + key: 'fnd-impl-tern-reactive', + kind: 'implementation', + title: + 'Tern zeroes history weight wherever saltglass sets the reactive mask', + created_by: 'Ines Okafor', + derives_from: ['render.dec-tern-reactive'], + cites: ['render.cit-code-tern-reactive'], + body: + 'Saltglass writes the reactive mask each frame. Tern sets the history weight to zero under it, so those pixels take the current frame, and the history clamp skips them. ' + + 'Caustics do not write the mask. That is why this code does nothing for the caustic flicker blocker.', + }, + { + type: 'Anchor', + effort: 'world', + key: 'cit-code-pool-score', + title: 'Code: streaming pool eviction score', + path: 'engine/kestrel/streaming/pool.rs', + symbol: 'evict_score', + commit: 'm10', + created_by: 'Kenji Arakawa', + }, + { + type: 'WriteFinding', + effort: 'world', + key: 'fnd-impl-pool-score', + kind: 'implementation', + title: 'Pool eviction keeps cells the tide exposes within half a turn', + created_by: 'Kenji Arakawa', + derives_from: ['world.dec-pool-forecast'], + cites: ['world.cit-code-pool-score'], + body: + 'The eviction score is the negative distance from the nearest player, so the farthest cell goes first. Cells the tide forecast exposes soon are never evicted. ' + + 'The Decision says within one tide turn; the code protects only half a turn, because the forecast lookup was written before the turn length was fixed. Nobody has seen a problem from it yet.', + }, + { + type: 'WriteRisk', + effort: 'world', + key: 'rsk-pool-score-cost', + likelihood: 'medium', + severity: 'low', + title: + 'Eviction scoring asks the tide forecast about every resident cell each frame', + created_by: 'Kenji Arakawa', + derives_from: ['world.dec-pool-forecast'], + cites: ['world.cit-code-pool-score'], + body: + 'Scoring makes one forecast lookup per resident cell per frame, about 180 lookups. On Skiff that is 0.2 ms of the main thread. ' + + 'If the pool grows, or the forecast needs a second lookup per cell for the second harmonic, it could double.', + }, + { + type: 'Anchor', + effort: 'world', + key: 'cit-code-tide-push', + title: 'Code: tide reads in the request queue', + path: 'engine/kestrel/streaming/io_queue.rs', + symbol: 'push_tide_read', + commit: 'm18', + created_by: 'Kenji Arakawa', + }, + { + type: 'WriteFinding', + effort: 'world', + key: 'fnd-impl-tide-push', + kind: 'implementation', + title: + 'Tide-exposed cell reads go to the front of the platform read queue', + created_by: 'Kenji Arakawa', + derives_from: ['world.dec-io-tide-priority'], + cites: ['world.cit-code-tide-push'], + body: + 'Reads for cells the tide is about to expose are pushed onto the front of their own lane, and that lane is served before proximity and background reads. ' + + 'It is the only place the tide outranks the player; every other read keeps its fixed priority.', + }, + { + type: 'Anchor', + effort: 'world', + key: 'cit-code-save-upgrade', + title: 'Code: save upgrade steps', + path: 'engine/kestrel/save/upgrade.rs', + symbol: 'upgrade_steps', + commit: 'm15', + created_by: 'Marta Lindqvist', + }, + { + type: 'WriteFinding', + effort: 'world', + key: 'fnd-impl-save-upgrade', + kind: 'implementation', + title: + 'Save upgrades run every step from the save version on load, one function per version', + created_by: 'Marta Lindqvist', + derives_from: ['world.dec-save-version', 'world.con-save-compat'], + cites: ['world.cit-code-save-upgrade'], + body: + 'There is one step function per schema version, and loading runs every step from the save version up to the current one before deltas apply. ' + + 'Steps are plain functions of the old save. There is one step today, the version 1 stamp. The second harmonic phase has no step yet.', + }, + { + type: 'Anchor', + effort: 'world', + key: 'cit-code-tide-wrap', + title: 'Code: tide phase wrap', + path: 'engine/kestrel/tides/phase.rs', + symbol: 'wrap_phase', + commit: 'm07', + created_by: 'Rhea Valdés', + }, + { + type: 'WriteFinding', + effort: 'world', + key: 'fnd-impl-tide-wrap', + kind: 'implementation', + title: 'The tide phase is masked to one cycle on every step', + created_by: 'Rhea Valdés', + derives_from: ['world.dec-tide-wrap'], + cites: ['world.cit-code-tide-wrap'], + body: 'The phase is a 32.32 fraction of the cycle and is masked to its fraction bits on every step, so it can never hold more than one 40-minute cycle.', + }, + { + type: 'WriteConstraint', + effort: 'world', + key: 'con-tide-step-alloc', + kind: 'hard', + title: 'The tide step allocates no memory', + created_by: 'Rhea Valdés', + derives_from: ['world.dec-tide-fixed'], + body: + 'The fixed-point tide step in engine/kestrel/tides/fixed.rs must not allocate. It runs at 10 Hz on every peer, and one allocation in it showed up as 0.3 ms spikes on Skiff. ' + + 'Use the preallocated height field and the checked-in sine table.', + }, + { + type: 'WriteIssue', + effort: 'world', + key: 'iss-tide-lod-q', + kind: 'question', + title: + 'How should the tide simulation save time in cells far from every player?', + created_by: 'Rhea Valdés', + derives_from: ['world.rsk-tide-skiff-cost'], + body: + 'The fixed-point tide step costs 0.9 ms on Skiff, and most of it goes to cells no player is near. ' + + 'We want a cheaper step for cells more than 600 m from every player. It must keep peers bit-identical and leave far cells in the right state when a player arrives.', + }, + { + type: 'WriteDecision', + effort: 'world', + key: 'dec-tide-lod-quarter', + title: 'Step tide cells beyond 600 m at a quarter of the tide rate', + created_by: 'Kenji Arakawa', + derives_from: ['world.iss-tide-lod-q', 'world.con-tide-bitexact'], + body: + 'Cells more than 600 m from every player step on every fourth tide tick, taking four ticks of phase at once. Every peer picks the same cells from the same player positions, so tide state stays bit-identical. ' + + 'Kenji Arakawa estimates the Skiff tide step drops from 0.9 ms to 0.4 ms on the coast map.', + }, + { + type: 'Anchor', + effort: 'ports', + key: 'cit-code-pso-warm', + title: 'Code: pipeline cache warm-up', + path: 'platform/skiff/pso_cache.rs', + symbol: 'warm_district', + commit: 'm09', + created_by: 'Priya Natarajan', + }, + { + type: 'WriteFinding', + effort: 'ports', + key: 'fnd-impl-pso-warm', + kind: 'implementation', + title: + 'Load screens compile each district pipeline list into the on-device cache', + created_by: 'Priya Natarajan', + derives_from: ['ports.dec-pso-ondevice'], + cites: ['ports.cit-code-pso-warm'], + body: + 'While a load screen is up, the game walks the district list and compiles every pipeline that is not cached yet. ' + + 'Entries are keyed by pipeline hash and driver version, so a driver change misses every entry and the lists rebuild the cache.', + }, + { + type: 'WriteIssue', + effort: 'ports', + key: 'iss-dynres-hold', + kind: 'defect', + title: + 'Skiff stays at the lowest resolution for two seconds after one slow frame', + created_by: 'Yusuf Demir', + derives_from: ['ports.dec-dynres-3'], + body: + 'platform/skiff/dynres.rs drops straight to the 504p floor when one frame misses the GPU budget, then holds there for 60 frames before stepping back up. ' + + 'A single spike from a tide splash leaves the picture soft for two seconds. Yusuf Demir wants a gradual step down as well as up.', + }, + { + type: 'Anchor', + effort: 'tools', + key: 'cit-code-cache-key', + title: 'Code: shader cache key', + path: 'tools/shadercache/key.py', + symbol: 'cache_key', + commit: 'm11', + created_by: "Liam O'Connell", + }, + { + type: 'WriteFinding', + effort: 'tools', + key: 'fnd-impl-cache-key', + kind: 'implementation', + title: + 'The cache key hashes its inputs in the order the key Decision lists them', + created_by: "Liam O'Connell", + derives_from: ['tools.dec-key-v3'], + cites: ['tools.cit-code-cache-key'], + body: + 'The key hashes the preprocessed source, the sorted defines, the target, the compiler binary hash and the normalized flags, in that order. ' + + 'The compiler binary hash is computed once per agent at startup.', + }, + { + type: 'WriteIssue', + effort: 'tools', + key: 'iss-remote-retry', + kind: 'defect', + title: + 'A timed-out cache fetch is retried once and then compiled locally', + created_by: "Liam O'Connell", + derives_from: ['tools.dec-timeout'], + body: + 'tools/shadercache/remote.py retries a fetch that times out exactly once, with the same 30-second timeout, and then the agent compiles the shader itself. ' + + 'At the 02:00 network peak that turns one slow fetch into a minute of waiting followed by a local compile.', + }, + { + type: 'Anchor', + effort: 'netcode', + key: 'cit-code-checksum', + title: 'Code: desync checksums', + path: 'net/coop/checksum.rs', + symbol: 'hash_blocks', + commit: 'm16', + created_by: 'Sofia Marchetti', + }, + { + type: 'WriteFinding', + effort: 'netcode', + key: 'fnd-impl-checksum', + kind: 'implementation', + title: 'Desync checks hash three state blocks every 30th frame', + created_by: 'Sofia Marchetti', + derives_from: ['netcode.dec-checksum'], + cites: ['netcode.cit-code-checksum'], + body: 'Tide, physics and Gull state are hashed separately with CRC-32 on every 30th frame. The host keeps sending them after the switch to host-authoritative sync.', + }, + ], + }, + { + key: 'm28', + at: at('2026-09-29 09:00'), + author: 'kenji', + subject: 'Keep cells the tide exposes within one turn, not half a turn', + proof: ['world.dec-pool-forecast'], + files: { 'engine/kestrel/streaming/pool.rs': POOL_3 }, + }, + { + key: 'm29', + at: at('2026-09-29 10:00'), + author: 'kenji', + subject: 'Record the pool eviction fix in Proof', + events: [ + { + type: 'Anchor', + effort: 'world', + key: 'cit-code-pool-score-2', + title: 'Code: streaming pool eviction score, one-turn fix', + path: 'engine/kestrel/streaming/pool.rs', + symbol: 'evict_score', + commit: 'm28', + created_by: 'Kenji Arakawa', + }, + { + type: 'WriteFinding', + effort: 'world', + key: 'fnd-impl-pool-score-2', + kind: 'implementation', + title: + 'Pool eviction keeps cells the tide exposes within one full turn', + created_by: 'Kenji Arakawa', + supersedes: ['world.fnd-impl-pool-score'], + derives_from: ['world.dec-pool-forecast'], + cites: ['world.cit-code-pool-score-2'], + body: + 'The eviction score now protects cells the forecast exposes within one full tide turn, 2,400 ticks, as the Decision says. ' + + 'The half-turn window let the pool evict cells about 12 minutes before the tide uncovered them on the coast map.', + }, + ], + }, +]; + +// ------------------------------------------------------------ branches ---- + +const branches = [ + { + name: 'ines/vsm-pool', + pr: 41, + author: 'ines', + title: 'Grow the shadow page pool at spring low tide', + commits: [ + { + key: 'b1a', + at: at('2026-09-29 14:00'), + author: 'ines', + subject: 'Grow the VSM page pool to 6,144 pages near spring low water', + body: 'The far GI cascade lends its 128 MB while the sun is low over the flats.', + proof: ['render.dec-vsm-grow'], + files: { + 'engine/lantern/shadows/vsm_pool.rs': VSM_POOL_GROW, + 'engine/lantern/gi/sdf_volumes.rs': SDF_VOLUMES_LEND, + }, + events: [ + { + type: 'WriteDecision', + effort: 'render', + key: 'dec-vsm-grow', + title: + 'Use virtual shadow maps for the sun with a 6,144-page pool at spring low tide', + created_by: 'Ines Okafor', + supersedes: ['render.dec-vsm'], + derives_from: [ + 'render.fnd-vsm-overflow', + 'render.iss-vsm-tide', + 'render.con-vram', + ], + body: + 'Everything in the sun-shadow Decision stands, with one change. When the tide forecast puts spring low water less than one tide step away, the page pool grows from 4,096 to 6,144 pages. ' + + 'The extra 128 MB comes from the far SDF GI cascade, which stops updating while the sun is low over the flats; GI out there holds its last result for those minutes. ' + + 'The pool shrinks back once the tide passes mid flood. On the spring-tide trace the peak of 5,912 pages fits with 232 to spare.', + }, + { + type: 'AcceptDecision', + decisionId: 'render.dec-vsm-grow', + }, + { + type: 'ResolveIssue', + issueId: 'render.iss-vsm-tide', + resolution: 'resolved', + resolvedBy: ['render.dec-vsm-grow'], + }, + ], + }, + { + key: 'b1b', + at: at('2026-09-30 10:00'), + author: 'ines', + subject: 'Summarize Lantern after the page pool change', + events: [ + { + type: 'WriteCompaction', + effort: 'render', + key: 'cmp-5', + title: 'Lantern after the spring-tide page pool change', + created_by: 'Ines Okafor', + derives_from: [ + 'render.dec-vsm-grow', + 'render.dec-vb-6', + 'render.dec-tern', + 'render.dec-presets-4b', + 'render.dec-brine-3', + 'render.dec-tern-reactive', + 'render.con-120-2', + 'render.iss-caustic-flicker', + 'render.iss-hdr-cert', + 'render.rsk-hdr-cert', + ], + body: + 'Sun shadows: virtual shadow maps, with the page pool growing to 6,144 pages near spring low water and the far GI cascade lending its memory. The spring low tide blocker is closed. ' + + 'Geometry: visibility buffer on all platforms, Skiff resolving materials at half rate. Upscaling: Tern only, four presets including Ultra Performance for the 120 Hz mode, with the saltglass reactive mask. ' + + 'Tonemapping: Brine with the HDR shoulder at 800 nits. Open: caustic flicker under Tern, and Meridian HDR certification, whose slip risk is realized.', + }, + ], + }, + ], + }, + { + name: 'tomasz/vsm-evict', + pr: 44, + author: 'tomasz', + title: 'Coarser shadow pages on exposed wet sand', + commits: [ + { + key: 'b2a', + at: at('2026-09-29 16:00'), + author: 'tomasz', + subject: + 'Evict fine wet-sand shadow pages first and draw them a level coarser', + proof: ['render.dec-vsm-coarse'], + files: { 'engine/lantern/shadows/vsm_pool.rs': VSM_POOL_COARSE }, + events: [ + { + type: 'WriteFinding', + effort: 'render', + key: 'fnd-coarse-pages', + kind: 'measurement', + title: + 'Coarser wet-sand pages keep spring low tide at 3,980 shadow pages', + created_by: 'Tomasz Brandt', + derives_from: ['render.fnd-vsm-overflow'], + body: + 'With exposed wet sand drawn one page level coarser, the spring-tide trace on the flats peaks at 3,980 pages, inside the 4,096-page pool. ' + + 'Shadow edges on the wet sand soften from about 9 cm to 18 cm at 300 m. Yusuf Demir could not see the difference at gameplay distance.', + }, + { + type: 'WriteDecision', + effort: 'render', + key: 'dec-vsm-coarse', + title: + 'Use virtual shadow maps for the sun with coarser pages on exposed wet sand', + created_by: 'Tomasz Brandt', + supersedes: ['render.dec-vsm'], + derives_from: ['render.fnd-coarse-pages', 'render.iss-vsm-tide'], + body: + 'Everything in the sun-shadow Decision stands, except that exposed wet sand gets shadow pages one level coarser, and fine pages on wet sand are evicted first. ' + + 'The pool stays at 4,096 pages and 256 MB, so no other system gives up memory.', + }, + ], + }, + ], + }, + { + name: 'kenji/tide-lod', + pr: 46, + author: 'kenji', + title: 'Step far tide cells at a quarter rate', + commits: [ + { + key: 'b3a', + at: at('2026-09-30 11:00'), + author: 'kenji', + subject: 'Step tide cells beyond 600 m every fourth tick', + proof: ['world.dec-tide-lod-quarter'], + files: { + 'engine/kestrel/tides/mod.rs': TIDES_MOD_LOD, + 'engine/kestrel/tides/lod.rs': LOD_QUARTER, + }, + events: [ + { + type: 'AcceptDecision', + decisionId: 'world.dec-tide-lod-quarter', + }, + { + type: 'ResolveIssue', + issueId: 'world.iss-tide-lod-q', + resolution: 'resolved', + resolvedBy: ['world.dec-tide-lod-quarter'], + }, + ], + }, + { + key: 'b3b', + at: at('2026-09-30 15:00'), + author: 'kenji', + subject: 'Summarize the world core with tide LOD', + events: [ + { + type: 'WriteCompaction', + effort: 'world', + key: 'cmp-5', + title: 'World core state with tide level of detail', + created_by: 'Kenji Arakawa', + derives_from: [ + 'world.dec-tide-lod-quarter', + 'world.con-pool-3', + 'world.dec-tide-harmonic', + 'world.dec-gull-bytecode', + 'world.dec-cell-192', + 'world.dec-jobs-6', + 'world.dec-tide-fixed', + 'world.dec-save-version', + 'world.iss-blk-save-harmonic', + 'world.iss-blk-io-starve', + ], + body: + 'Tides: 32.32 fixed point with a second harmonic; cells beyond 600 m from every player now step at a quarter of the tide rate, which settles the far-cell question. ' + + 'Streaming: 192 m cells, a 1.6 GB pool. Jobs and Gull share one thread pool with continuation waits; Gull runs as bytecode. Saves carry a schema version. ' + + 'Open blockers: beta saves lack the harmonic phase, and tide turns starve inland reads.', + }, + ], + }, + ], + }, + { + name: 'priya/skiff-cpu', + pr: 48, + author: 'priya', + title: 'Free Skiff CPU time for tide storms', + commits: [ + { + key: 'b4a', + at: at('2026-09-30 12:00'), + author: 'priya', + subject: 'Cap tide-storm particles at 6,000 on Skiff', + proof: ['ports.dec-storm-cap'], + files: { 'platform/skiff/storm.rs': STORM_CAP }, + events: [ + { + type: 'Retract', + recordId: 'ports.fnd-storm-capture', + reason: + 'Captured with the particle debug overlay on, which alone costs 1.4 ms of CPU. The retail recapture replaces it.', + }, + { + type: 'WriteFinding', + effort: 'ports', + key: 'fnd-storm-recapture', + kind: 'measurement', + title: + 'Tide storms push Skiff 40 Hz frames to 27.0 ms on the CPU with the overlay off', + created_by: 'Priya Natarajan', + derives_from: ['ports.iss-blk-storm'], + body: + 'Priya Natarajan recaptured the storm sequence on retail firmware with the particle debug overlay off. The CPU frame peaks at 27.0 ms in the 40 Hz mode, 2 ms over the 25 ms budget. ' + + 'Storm particles and their tide collision cost 4.9 ms of it, so the 6,000-particle cap brings the peak to about 23.2 ms.', + }, + { + type: 'AcceptDecision', + decisionId: 'ports.dec-storm-cap', + }, + { + type: 'ResolveIssue', + issueId: 'ports.iss-blk-storm', + resolution: 'resolved', + resolvedBy: ['ports.dec-storm-cap'], + }, + ], + }, + { + key: 'b4b', + at: at('2026-09-30 16:00'), + author: 'priya', + subject: + 'Freeze far tide cells on every platform and catch them up on approach', + proof: ['world.dec-tide-lod-freeze'], + files: { + 'engine/kestrel/tides/mod.rs': TIDES_MOD_LOD, + 'engine/kestrel/tides/lod.rs': LOD_FREEZE, + }, + events: [ + { + type: 'WriteDecision', + effort: 'world', + key: 'dec-tide-lod-freeze', + title: + 'Freeze tide cells beyond 600 m and catch them up when a player comes near', + created_by: 'Priya Natarajan', + derives_from: ['world.iss-tide-lod-q', 'world.con-tide-bitexact'], + body: + 'Cells more than 600 m from every player stop stepping. When a player comes within 600 m, the cell runs its missed ticks in one batch, which costs at most 0.6 ms in that frame. ' + + 'Freezing saves the whole step for far cells: the Skiff tide step drops from 0.9 ms to 0.25 ms on the coast map, against 0.4 ms for stepping them at a quarter rate. ' + + 'Priya Natarajan needs that time for tide storms on Skiff.', + }, + { + type: 'AcceptDecision', + decisionId: 'world.dec-tide-lod-freeze', + }, + { + type: 'ResolveIssue', + issueId: 'world.iss-tide-lod-q', + resolution: 'resolved', + resolvedBy: ['world.dec-tide-lod-freeze'], + }, + ], + }, + ], + }, +]; + +export default { main, branches }; + +/** Every commit, main first, each with its branch name. */ +export function specCommits() { + return [ + ...main.map((c) => ({ ...c, branch: 'main' })), + ...branches.flatMap((b) => + b.commits.map((c) => ({ ...c, branch: b.name })) + ), + ]; +} + +/** Every declared code anchor, keyed `.`, with its branch. */ +export function specAnchors() { + return specCommits().flatMap((c) => + (c.events ?? []) + .filter((e) => e.type === 'Anchor') + .map((e) => ({ ...e, key: `${e.effort}.${e.key}`, branch: c.branch })) + ); +}