diff --git a/.flatbread-proof/findings/fnd-stale-code-anchors-cost-little-and-rename-aware--2kyseq5ryvsxc9jc.md b/.flatbread-proof/findings/fnd-stale-code-anchors-cost-little-and-rename-aware--2kyseq5ryvsxc9jc.md new file mode 100644 index 00000000..5632409b --- /dev/null +++ b/.flatbread-proof/findings/fnd-stale-code-anchors-cost-little-and-rename-aware--2kyseq5ryvsxc9jc.md @@ -0,0 +1,17 @@ +--- +id: fnd-stale-code-anchors-cost-little-and-rename-aware--2kyseq5ryvsxc9jc +effort: eff-effort-graph-memory-and-agent-wedge--szeqvmgqjqnhd002 +title: 'Stale code anchors cost little, and rename-aware reads cut the recovery' +kind: measurement +created_at: '2026-10-20T20:02:52.736Z' +derives_from: + - fnd-at-scale-code-and-branch-reads-still-match-git-o--e2ws7dke8rr7qbej +--- + +Phase 3 asked Haiku, twice, 74 questions on the 10x test bed after about ten refactor commits that Proof never saw: two folder moves, file and function renames, a split, a merge, and one behaviour change (a hash interval from 30 to 60 frames while the accepted Decision still says 30). Ten code anchors no longer hold. + +- With today's reads, `findcode` got 133 of 148 question-runs and read-only git 129 (11 to 7, p = 0.48), at 27 percent fewer tokens (24,983 against 34,303). On questions about behaviour no single path names it read a third less (29,705 against 45,047) at the same accuracy. On many-branch questions it saved nothing. +- On the refactor set it did slightly worse than git alone (34 against 35; 8 against 10 on holdout). `--path` on a moved file returned nothing, and cards showed old paths with no warning. +- Reads that follow git renames and flag stale anchors lifted answers on the cards from 9 to 12 of 21 on the offline holdout. Paid, they cut refactor-set reading by a fifth (24,125 against 30,248; a third on holdout) and tool calls from 14 to 10, but accuracy did not move (129 against 133 overall, 3 to 7, p = 0.34). + +A refactor Proof missed costs less than expected, because git still shows the moves; the stale-anchor notes make recovery cheaper, not more accurate. The remaining misses are graph-wide audits of stale anchors, which agents try to answer by searching for words like "stale" that no read indexes. diff --git a/benchmarks/proof-recall/README.md b/benchmarks/proof-recall/README.md index 13d4d355..513bde92 100644 --- a/benchmarks/proof-recall/README.md +++ b/benchmarks/proof-recall/README.md @@ -1614,3 +1614,52 @@ codebase grow, and the filler never touches a real file. `--path` and `--refs` earn their place by cost, about half the reading and half the tool calls, not by accuracy. A test that could show more would ask about code no single path names, or about many branches at once. + +### Phase 3: behaviour, many branches, and a refactor Proof missed + +The 10x test bed with `build-repo.mjs --refactor`: about ten commits on +main move folders, rename files and functions, split one module, merge +two, and change one behaviour (a hash interval from 30 to 60 frames, while +the accepted Decision still says 30). None of them updates Proof, so ten +code anchors no longer hold. The sets are the 27 code and branch questions +the refactor leaves true, plus `behaviour` (16), `multibranch` (12) and +`refactor` (19). Haiku, twice, `--timeout 1200`. Run files: +`runs/p3-r{1,2}-*.json` (today's reads) and `runs/p3s-r{1,2}-findcode.json` +(the reads that follow renames and flag stale anchors). No-brief reads no +Proof, so both `findcode` builds are compared with the same no-brief runs. + +| Set (question-runs) | `findcode`, today's reads | `findcode`, rename-aware | no-brief | Read tokens, today / rename-aware / no-brief | +| ---------------------- | ------------------------- | ------------------------ | -------- | -------------------------------------------- | +| all (148) | 133 | 129 | 129 | 24,983 / 23,948 / 34,303 | +| code and branch (54) | 52 | 52 | 51 | 15,512 / 15,641 / 25,629 | +| behaviour (32) | 24 | 21 | 21 | 29,705 / 27,799 / 45,047 | +| multibranch (24) | 23 | 23 | 22 | 31,660 / 37,222 / 35,842 | +| refactor (38) | 34 | 33 | 35 | 30,248 / 24,125 / 36,610 | +| refactor, holdout (10) | 8 | 8 | 10 | 27,623 / 18,627 / 37,395 | + +No gap here is larger than chance (today's reads against no-brief, 11 to +7, p = 0.48; rename-aware against today's, 3 to 7, p = 0.34). No session +timed out. Ten sessions wrote scratch files outside their folder; none +read the questions, the keymap or the refactor map. + +- **Behaviour no single path names.** Search reads a third fewer tokens + than git alone (29,705 against 45,047) and makes 15 tool calls instead + of 26, at the same accuracy. +- **Many branches.** No saving: `--refs all` cards on every question cost + as much as `git diff` per branch. +- **Stale anchors, today's reads.** `findcode` does slightly worse than git + alone on the refactor set. `--path` on a moved file returns nothing, and + cards show old paths with no warning, so an agent that trusts them + misses a function that moved out of a file that still exists. +- **Rename-aware reads.** On the offline gate they lift answers on the cards + from 9 to 12 of 21 on holdout, with no holdout question worse. Paid, they + cut reading on the refactor set by a fifth (a third on holdout) and tool + calls from 14 to 10, but do not change accuracy. The remaining refactor + misses are graph-wide audits ("which records cite an anchor that no + longer holds?"): agents search for words like "stale", which no read + indexes, and stop at what they find. + +So a refactor that Proof missed costs an agent that trusts Proof less than +expected, because git still shows the moves, and the new notes make the +recovery cheaper rather than more accurate. A read that lists every stale +anchor at once is the change these numbers point to next. diff --git a/benchmarks/proof-recall/runs/p3-r1-findcode.json b/benchmarks/proof-recall/runs/p3-r1-findcode.json new file mode 100644 index 00000000..c628e320 --- /dev/null +++ b/benchmarks/proof-recall/runs/p3-r1-findcode.json @@ -0,0 +1,838 @@ +{ + "run": { + "name": "p3-r1-findcode", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "c80612938d90f14747e3be3341c6434f3d78efcbd611bf5085facc8653eee5c2", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "6e94ef68632e522abca8eaf873b3238cf4ae45394a4810e66f1595c3fe2ee839" + }, + "answers": { + "behaviour-001": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg" + ], + "tokens": 20303, + "fixed_tokens": 2737, + "processed_tokens": 114711, + "tool_calls": 10, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "behaviour-002": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh" + ], + "tokens": 43713, + "fixed_tokens": 2738, + "processed_tokens": 254820, + "tool_calls": 15, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-003": { + "answer": [ + "dec-delay-autosaves-until-the-tide-turn-finishes--4agcgjjj70djdv6b", + "dec-save-player-changes-as-deltas-keyed-to-the-tide--1qxvc9985c8pzke8", + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 26291, + "fixed_tokens": 1110, + "processed_tokens": 174176, + "tool_calls": 12, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "behaviour-004": { + "answer": "unknown", + "tokens": 73901, + "fixed_tokens": 1106, + "processed_tokens": 1333154, + "tool_calls": 31, + "turns": 30, + "timed_out": false, + "left_folder": false + }, + "behaviour-005": { + "answer": [ + "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "dec-rejoin-the-co-op-session-after-meridian-resumes--gp2btfw62qm4khy9" + ], + "tokens": 23664, + "fixed_tokens": 1115, + "processed_tokens": 165296, + "tool_calls": 13, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "behaviour-006": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-tide-turns-leave-inland-terrain-untextured-for-s--ccv6nhwk68kj732c" + ], + "tokens": 19931, + "fixed_tokens": 1115, + "processed_tokens": 151067, + "tool_calls": 12, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-007": { + "answer": [ + "con-tide-state-must-be-bit-identical-on-every-peer--zrpnp8qjyxzkas7t", + "con-co-op-peers-must-agree-on-tide-state-bit-for-bit--4v0sdbc3t4ywzr3g" + ], + "tokens": 19172, + "fixed_tokens": 1109, + "processed_tokens": 95149, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "behaviour-008": { + "answer": [ + "con-meridian-memory-budget--99q6099e423wdtqn", + "con-skiff-memory-budget--nf7qbhsnmcn6v479", + "con-streaming-pool-budget--8czwb4fjr5p5vpy7", + "con-render-target-memory-on-meridian--q4szajxse9c1csdv" + ], + "tokens": 15423, + "fixed_tokens": 1116, + "processed_tokens": 92075, + "tool_calls": 12, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "behaviour-009": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-use-host-authoritative-sync-for-co-op--17hqrmtja7pn98d6", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 24260, + "fixed_tokens": 1108, + "processed_tokens": 194103, + "tool_calls": 11, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "behaviour-010": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k" + ], + "tokens": 83917, + "fixed_tokens": 1109, + "processed_tokens": 1620345, + "tool_calls": 56, + "turns": 32, + "timed_out": false, + "left_folder": false + }, + "behaviour-011": { + "answer": [ + "iss-a-timed-out-cache-fetch-is-retried-once-and-then--sn5thmgf02v8rfe4", + "iss-skiff-stays-at-the-lowest-resolution-for-two-sec--erke2h7xf7vepdv2", + "fnd-the-kite-physics-stats-keep-a-rolling-window-of--j6qv6qchghanqh1v" + ], + "tokens": 71152, + "fixed_tokens": 1112, + "processed_tokens": 1063181, + "tool_calls": 38, + "turns": 22, + "timed_out": false, + "left_folder": false + }, + "behaviour-012": { + "answer": false, + "tokens": 12026, + "fixed_tokens": 1110, + "processed_tokens": 80637, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "behaviour-013": { + "answer": false, + "tokens": 26841, + "fixed_tokens": 1115, + "processed_tokens": 292370, + "tool_calls": 15, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "behaviour-014": { + "answer": false, + "tokens": 23205, + "fixed_tokens": 1110, + "processed_tokens": 218173, + "tool_calls": 12, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-015": { + "answer": [ + "dec-exclude-caustic-luminance-from-tern-s-history-cl--xhxjn26ss343v2k4", + "dec-lock-caustic-photon-jitter-to-the-tide-phase--85x10k4h11t8nx7c" + ], + "tokens": 21404, + "fixed_tokens": 1110, + "processed_tokens": 187484, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "behaviour-016": { + "answer": [ + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-lower-the-skiff-dynamic-resolution-floor-to-504p--15esnt90gcn29enn", + "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1" + ], + "tokens": 19614, + "fixed_tokens": 1118, + "processed_tokens": 228221, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "branch-001": { + "answer": 41, + "tokens": 24476, + "fixed_tokens": 1103, + "processed_tokens": 119307, + "tool_calls": 8, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-003": { + "answer": "proposed", + "tokens": 10264, + "fixed_tokens": 1105, + "processed_tokens": 56785, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-004": { + "answer": "ines/vsm-pool", + "tokens": 14072, + "fixed_tokens": 1105, + "processed_tokens": 60837, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-005": { + "answer": "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "tokens": 15504, + "fixed_tokens": 1104, + "processed_tokens": 63610, + "tool_calls": 5, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-006": { + "answer": "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "tokens": 16015, + "fixed_tokens": 1106, + "processed_tokens": 97168, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-007": { + "answer": 48, + "tokens": 10030, + "fixed_tokens": 1105, + "processed_tokens": 42018, + "tool_calls": 3, + "turns": 4, + "timed_out": false, + "left_folder": false + }, + "branch-008": { + "answer": "fnd-tide-storms-push-skiff-40-hz-frames-to-28-4-ms-o--28gemvtznv4zztdc", + "tokens": 17214, + "fixed_tokens": 1101, + "processed_tokens": 120905, + "tool_calls": 8, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-009": { + "answer": "27.0 ms", + "tokens": 11809, + "fixed_tokens": 1103, + "processed_tokens": 90173, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-010": { + "answer": [ + "fnd-world-core-state-with-tide-level-of-detail--x947x5kj16hcngef", + "fnd-lantern-after-the-spring-tide-page-pool-change--9zbcsqh53mgv7h4p" + ], + "tokens": 16780, + "fixed_tokens": 1106, + "processed_tokens": 111245, + "tool_calls": 8, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-011": { + "answer": false, + "tokens": 20953, + "fixed_tokens": 1107, + "processed_tokens": 118413, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-012": { + "answer": "superseded", + "tokens": 19361, + "fixed_tokens": 1105, + "processed_tokens": 157073, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-013": { + "answer": "kenji/tide-lod", + "tokens": 16653, + "fixed_tokens": 1103, + "processed_tokens": 91169, + "tool_calls": 9, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-014": { + "answer": "Priya Natarajan", + "tokens": 13166, + "fixed_tokens": 1103, + "processed_tokens": 102744, + "tool_calls": 11, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-015": { + "answer": 3, + "tokens": 21304, + "fixed_tokens": 1108, + "processed_tokens": 163842, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "branch-016": { + "answer": "proposed", + "tokens": 16457, + "fixed_tokens": 1103, + "processed_tokens": 92027, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-018": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96", + "tokens": 16096, + "fixed_tokens": 1106, + "processed_tokens": 129596, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-020": { + "answer": "Two open branches would replace it: origin/ines/vsm-pool, with the accepted `dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96` (6,144-page pool at spring low tide), and origin/tomasz/vsm-evict, with the proposed `dec-use-virtual-shadow-maps-for-the-sun-with-coarser--jtcbmrm4c48ma8z8` (coarser wet-sand pages). Both supersede `dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4`.", + "tokens": 18966, + "fixed_tokens": 1108, + "processed_tokens": 124985, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-021": { + "answer": "dec-cap-tide-storm-particles-at-6-000-on-skiff--resze52afjxyxys1", + "tokens": 23706, + "fixed_tokens": 1111, + "processed_tokens": 138251, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-022": { + "answer": "priya/skiff-cpu", + "tokens": 45618, + "fixed_tokens": 1105, + "processed_tokens": 311061, + "tool_calls": 30, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "code-003": { + "answer": [ + "rsk-eviction-scoring-asks-the-tide-forecast-about-ev--zfwghmgds54h8y46" + ], + "tokens": 13631, + "fixed_tokens": 1097, + "processed_tokens": 77251, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-004": { + "answer": "fnd-pool-eviction-keeps-cells-the-tide-exposes-withi--tzseexbv0m0qw2jk", + "tokens": 14983, + "fixed_tokens": 1102, + "processed_tokens": 110207, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "code-005": { + "answer": "false: dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "tokens": 16147, + "fixed_tokens": 1107, + "processed_tokens": 114395, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "code-006": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 8370, + "fixed_tokens": 1097, + "processed_tokens": 61172, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-011": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 12524, + "fixed_tokens": 1099, + "processed_tokens": 72998, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "code-012": { + "answer": "dec-ship-four-tern-quality-presets-adding-ultra-perf--3jg1wyq3bshv839c", + "tokens": 15042, + "fixed_tokens": 1097, + "processed_tokens": 75906, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-018": { + "answer": "unknown", + "tokens": 13224, + "fixed_tokens": 1100, + "processed_tokens": 68061, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-019": { + "answer": "800 nits", + "tokens": 13873, + "fixed_tokens": 1100, + "processed_tokens": 61505, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "multibranch-001": { + "answer": false, + "tokens": 43099, + "fixed_tokens": 1119, + "processed_tokens": 329638, + "tool_calls": 13, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "multibranch-002": { + "answer": [ + "dec-tooltip-renderer-leave-it-as-is-0467--y44qjef2x7j0xv7e", + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj", + "dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3", + "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20" + ], + "tokens": 27531, + "fixed_tokens": 1111, + "processed_tokens": 129831, + "tool_calls": 8, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "multibranch-003": { + "answer": "origin/sofia/party-invites-refresh-timer and origin/marta/party-invites-refresh-on-open conflict in Proof: dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3 is accepted on sofia's branch and rejected on marta's (rejected_by dec-refresh-the-party-invites-list-once-each-time-th--zfak0g1cx14968sd), and they share no code files.", + "tokens": 14246, + "fixed_tokens": 1114, + "processed_tokens": 79674, + "tool_calls": 13, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-004": { + "answer": "Draft PRs #41 and #44. Both touch engine/lantern/shadows/vsm_pool.rs, which the accepted main Decision dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4 governs (via the finding fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx).", + "tokens": 34644, + "fixed_tokens": 1111, + "processed_tokens": 288545, + "tool_calls": 12, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "multibranch-005": { + "answer": 4, + "tokens": 29111, + "fixed_tokens": 1109, + "processed_tokens": 224168, + "tool_calls": 15, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "multibranch-006": { + "answer": 4, + "tokens": 17680, + "fixed_tokens": 1116, + "processed_tokens": 108830, + "tool_calls": 8, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "multibranch-007": { + "answer": "engine/kestrel/world/ocean/", + "tokens": 18335, + "fixed_tokens": 1110, + "processed_tokens": 125586, + "tool_calls": 9, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "multibranch-008": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-skiff-cert-blocker-40-hz-mode-misses-frames-in-t--8r81ajkewab5gymr" + ], + "tokens": 46294, + "fixed_tokens": 1109, + "processed_tokens": 415170, + "tool_calls": 21, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "multibranch-009": { + "answer": [ + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj", + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p" + ], + "tokens": 41340, + "fixed_tokens": 1115, + "processed_tokens": 385640, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "multibranch-010": { + "answer": 8, + "tokens": 109588, + "fixed_tokens": 1117, + "processed_tokens": 2159253, + "tool_calls": 37, + "turns": 36, + "timed_out": false, + "left_folder": false + }, + "multibranch-011": { + "answer": 50, + "tokens": 8889, + "fixed_tokens": 1115, + "processed_tokens": 48468, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "multibranch-012": { + "answer": [ + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr", + "cit-code-enqueue-in-the-ambient-chatter-queue--ksz0m3yhk8a0j0qq" + ], + "tokens": 36973, + "fixed_tokens": 1109, + "processed_tokens": 330072, + "tool_calls": 30, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "refactor-001": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "tokens": 20799, + "fixed_tokens": 1105, + "processed_tokens": 137553, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "refactor-002": { + "answer": "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "tokens": 17832, + "fixed_tokens": 1106, + "processed_tokens": 123284, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "refactor-003": { + "answer": "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "tokens": 15970, + "fixed_tokens": 1106, + "processed_tokens": 120156, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-004": { + "answer": "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "tokens": 23965, + "fixed_tokens": 1111, + "processed_tokens": 189754, + "tool_calls": 11, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "refactor-005": { + "answer": "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "tokens": 15766, + "fixed_tokens": 1108, + "processed_tokens": 124790, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-006": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw" + ], + "tokens": 13691, + "fixed_tokens": 1111, + "processed_tokens": 166586, + "tool_calls": 15, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "refactor-007": { + "answer": "con-the-tide-step-allocates-no-memory--bgce2y8wndwrwe0r", + "tokens": 22845, + "fixed_tokens": 1109, + "processed_tokens": 195564, + "tool_calls": 11, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "refactor-008": { + "answer": "iss-skiff-stays-at-the-lowest-resolution-for-two-sec--erke2h7xf7vepdv2", + "tokens": 19750, + "fixed_tokens": 1109, + "processed_tokens": 157614, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "refactor-009": { + "answer": [ + "fnd-the-cache-key-hashes-its-inputs-in-the-order-the--w74ey1s5g91aranp", + "fnd-load-screens-compile-each-district-pipeline-list--8gbea0h3zt5r14gs", + "fnd-save-upgrades-run-every-step-from-the-save-versi--agp84ez57kd03vpw", + "fnd-the-tide-phase-is-masked-to-one-cycle-on-every-s--p3by8f1vs0fj71ph", + "fnd-the-kite-physics-stats-keep-a-rolling-window-of--j6qv6qchghanqh1v", + "fnd-tern-zeroes-history-weight-wherever-saltglass-se--hhgcrcq2n03rwr53", + "fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx", + "fnd-desync-checks-hash-three-state-blocks-every-30th--0vdm4w5cnjvqegjp" + ], + "tokens": 119318, + "fixed_tokens": 1103, + "processed_tokens": 1277859, + "tool_calls": 31, + "turns": 20, + "timed_out": false, + "left_folder": false + }, + "refactor-010": { + "answer": 6, + "tokens": 108694, + "fixed_tokens": 1107, + "processed_tokens": 2430771, + "tool_calls": 43, + "turns": 31, + "timed_out": false, + "left_folder": false + }, + "refactor-011": { + "answer": "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "tokens": 59464, + "fixed_tokens": 1106, + "processed_tokens": 768416, + "tool_calls": 22, + "turns": 20, + "timed_out": false, + "left_folder": false + }, + "refactor-012": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-kite-physics-leave-it-as-is-0741--9xtc7xz57n0znv4v", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 87751, + "fixed_tokens": 1106, + "processed_tokens": 1775936, + "tool_calls": 53, + "turns": 33, + "timed_out": false, + "left_folder": false + }, + "refactor-013": { + "answer": false, + "tokens": 17287, + "fixed_tokens": 1114, + "processed_tokens": 78500, + "tool_calls": 7, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-014": { + "answer": "The Decision dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq says every 30 frames, but the code on main hashes them every 60 frames (HASH_INTERVAL_FRAMES = 60 in net/coop/desync.rs).", + "tokens": 15075, + "fixed_tokens": 1105, + "processed_tokens": 86183, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-015": { + "answer": "On main the function is `reclaim_pages` (renamed from `evict_pages` in df77028), in `engine/lantern/shadows/page_pool.rs` (renamed from `vsm_pool.rs` in eb579ca). The order note is fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx, which cites cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr.", + "tokens": 11245, + "fixed_tokens": 1112, + "processed_tokens": 148034, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "refactor-016": { + "answer": "engine/lantern/tern/reactive.rs", + "tokens": 9706, + "fixed_tokens": 1114, + "processed_tokens": 100949, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-017": { + "answer": "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "tokens": 20320, + "fixed_tokens": 1107, + "processed_tokens": 89949, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-018": { + "answer": "false dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "tokens": 18578, + "fixed_tokens": 1106, + "processed_tokens": 101612, + "tool_calls": 8, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-019": { + "answer": "unknown", + "tokens": 16907, + "fixed_tokens": 1105, + "processed_tokens": 90434, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + } + } +} diff --git a/benchmarks/proof-recall/runs/p3-r1-nobrief.json b/benchmarks/proof-recall/runs/p3-r1-nobrief.json new file mode 100644 index 00000000..ddb05a17 --- /dev/null +++ b/benchmarks/proof-recall/runs/p3-r1-nobrief.json @@ -0,0 +1,837 @@ +{ + "run": { + "name": "p3-r1-nobrief", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "c80612938d90f14747e3be3341c6434f3d78efcbd611bf5085facc8653eee5c2", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "6e94ef68632e522abca8eaf873b3238cf4ae45394a4810e66f1595c3fe2ee839" + }, + "answers": { + "behaviour-001": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg" + ], + "tokens": 56794, + "fixed_tokens": 2744, + "processed_tokens": 1155086, + "tool_calls": 31, + "turns": 32, + "timed_out": false, + "left_folder": false + }, + "behaviour-002": { + "answer": [ + "dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg" + ], + "tokens": 33738, + "fixed_tokens": 2744, + "processed_tokens": 304379, + "tool_calls": 14, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-003": { + "answer": [ + "dec-delay-autosaves-until-the-tide-turn-finishes--4agcgjjj70djdv6b", + "dec-save-player-changes-as-deltas-keyed-to-the-tide--1qxvc9985c8pzke8", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 46613, + "fixed_tokens": 1114, + "processed_tokens": 1254597, + "tool_calls": 39, + "turns": 40, + "timed_out": false, + "left_folder": false + }, + "behaviour-004": { + "answer": [ + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 72826, + "fixed_tokens": 1119, + "processed_tokens": 1068986, + "tool_calls": 25, + "turns": 23, + "timed_out": false, + "left_folder": false + }, + "behaviour-005": { + "answer": [ + "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "dec-use-a-6-frame-rollback-window-with-1-frame-of-in--s53944zhkw103frk", + "dec-use-host-authoritative-sync-for-co-op--17hqrmtja7pn98d6" + ], + "tokens": 56202, + "fixed_tokens": 1115, + "processed_tokens": 537372, + "tool_calls": 24, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "behaviour-006": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-tide-turns-leave-inland-terrain-untextured-for-s--ccv6nhwk68kj732c" + ], + "tokens": 24305, + "fixed_tokens": 1111, + "processed_tokens": 239640, + "tool_calls": 17, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "behaviour-007": { + "answer": [ + "con-tide-state-must-be-bit-identical-on-every-peer--zrpnp8qjyxzkas7t", + "con-co-op-peers-must-agree-on-tide-state-bit-for-bit--4v0sdbc3t4ywzr3g" + ], + "tokens": 29037, + "fixed_tokens": 1110, + "processed_tokens": 262782, + "tool_calls": 16, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-008": { + "answer": [ + "con-skiff-memory-budget--nf7qbhsnmcn6v479", + "con-meridian-memory-budget--99q6099e423wdtqn", + "con-streaming-pool-budget--8czwb4fjr5p5vpy7", + "con-render-target-memory-on-meridian--q4szajxse9c1csdv" + ], + "tokens": 23439, + "fixed_tokens": 1115, + "processed_tokens": 176290, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-009": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw", + "dec-use-host-authoritative-sync-for-co-op--17hqrmtja7pn98d6", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 31330, + "fixed_tokens": 1117, + "processed_tokens": 539653, + "tool_calls": 22, + "turns": 22, + "timed_out": false, + "left_folder": false + }, + "behaviour-010": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 56242, + "fixed_tokens": 1113, + "processed_tokens": 723695, + "tool_calls": 34, + "turns": 21, + "timed_out": false, + "left_folder": false + }, + "behaviour-011": { + "answer": ["con-the-tide-step-allocates-no-memory--bgce2y8wndwrwe0r"], + "tokens": 125391, + "fixed_tokens": 1114, + "processed_tokens": 7740667, + "tool_calls": 89, + "turns": 89, + "timed_out": false, + "left_folder": true + }, + "behaviour-012": { + "answer": false, + "tokens": 24592, + "fixed_tokens": 1120, + "processed_tokens": 583482, + "tool_calls": 32, + "turns": 33, + "timed_out": false, + "left_folder": false + }, + "behaviour-013": { + "answer": false, + "tokens": 50984, + "fixed_tokens": 1111, + "processed_tokens": 891602, + "tool_calls": 26, + "turns": 25, + "timed_out": false, + "left_folder": false + }, + "behaviour-014": { + "answer": false, + "tokens": 43697, + "fixed_tokens": 1116, + "processed_tokens": 393183, + "tool_calls": 13, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-015": { + "answer": [ + "dec-exclude-caustic-luminance-from-tern-s-history-cl--xhxjn26ss343v2k4", + "dec-lock-caustic-photon-jitter-to-the-tide-phase--85x10k4h11t8nx7c" + ], + "tokens": 42577, + "fixed_tokens": 1112, + "processed_tokens": 510926, + "tool_calls": 26, + "turns": 20, + "timed_out": false, + "left_folder": false + }, + "behaviour-016": { + "answer": [ + "dec-lower-the-skiff-dynamic-resolution-floor-to-504p--15esnt90gcn29enn", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1" + ], + "tokens": 35655, + "fixed_tokens": 1112, + "processed_tokens": 312510, + "tool_calls": 17, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "branch-001": { + "answer": 41, + "tokens": 12455, + "fixed_tokens": 1112, + "processed_tokens": 145414, + "tool_calls": 13, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "branch-003": { + "answer": "proposed", + "tokens": 8363, + "fixed_tokens": 1106, + "processed_tokens": 76917, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-004": { + "answer": "ines/vsm-pool", + "tokens": 22893, + "fixed_tokens": 1106, + "processed_tokens": 281115, + "tool_calls": 19, + "turns": 18, + "timed_out": false, + "left_folder": false + }, + "branch-005": { + "answer": "unknown", + "tokens": 35911, + "fixed_tokens": 1105, + "processed_tokens": 329521, + "tool_calls": 14, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "branch-006": { + "answer": "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "tokens": 15032, + "fixed_tokens": 1111, + "processed_tokens": 140230, + "tool_calls": 15, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "branch-007": { + "answer": 48, + "tokens": 32971, + "fixed_tokens": 1103, + "processed_tokens": 377610, + "tool_calls": 15, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "branch-008": { + "answer": "fnd-tide-storms-push-skiff-40-hz-frames-to-28-4-ms-o--28gemvtznv4zztdc", + "tokens": 26873, + "fixed_tokens": 1104, + "processed_tokens": 312819, + "tool_calls": 15, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "branch-009": { + "answer": "27.0 ms", + "tokens": 10357, + "fixed_tokens": 1108, + "processed_tokens": 107950, + "tool_calls": 12, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-010": { + "answer": [ + "fnd-lantern-after-the-spring-tide-page-pool-change--9zbcsqh53mgv7h4p", + "fnd-world-core-state-with-tide-level-of-detail--x947x5kj16hcngef" + ], + "tokens": 26766, + "fixed_tokens": 1106, + "processed_tokens": 142427, + "tool_calls": 17, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-011": { + "answer": false, + "tokens": 32680, + "fixed_tokens": 1106, + "processed_tokens": 347939, + "tool_calls": 15, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "branch-012": { + "answer": "superseded", + "tokens": 23334, + "fixed_tokens": 1110, + "processed_tokens": 310445, + "tool_calls": 18, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "branch-013": { + "answer": "kenji/tide-lod", + "tokens": 26724, + "fixed_tokens": 1104, + "processed_tokens": 399970, + "tool_calls": 22, + "turns": 23, + "timed_out": false, + "left_folder": false + }, + "branch-014": { + "answer": "Priya Natarajan", + "tokens": 8595, + "fixed_tokens": 1108, + "processed_tokens": 81397, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-015": { + "answer": 3, + "tokens": 13403, + "fixed_tokens": 1108, + "processed_tokens": 134416, + "tool_calls": 11, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "branch-016": { + "answer": "proposed", + "tokens": 21671, + "fixed_tokens": 1107, + "processed_tokens": 270961, + "tool_calls": 20, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "branch-018": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96", + "tokens": 28541, + "fixed_tokens": 1108, + "processed_tokens": 307690, + "tool_calls": 15, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "branch-020": { + "answer": "ines/vsm-pool (its accepted Decision dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96 supersedes dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4) and tomasz/vsm-evict (its proposed Decision dec-use-virtual-shadow-maps-for-the-sun-with-coarser--jtcbmrm4c48ma8z8 also lists dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4 as superseded, though it is only proposed).", + "tokens": 37455, + "fixed_tokens": 1104, + "processed_tokens": 350067, + "tool_calls": 15, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "branch-021": { + "answer": "dec-cap-tide-storm-particles-at-6-000-on-skiff--resze52afjxyxys1", + "tokens": 14405, + "fixed_tokens": 1107, + "processed_tokens": 131499, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "branch-022": { + "answer": "priya/skiff-cpu", + "tokens": 23483, + "fixed_tokens": 1108, + "processed_tokens": 147500, + "tool_calls": 13, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "code-003": { + "answer": [ + "rsk-eviction-scoring-asks-the-tide-forecast-about-ev--zfwghmgds54h8y46" + ], + "tokens": 58950, + "fixed_tokens": 1108, + "processed_tokens": 764574, + "tool_calls": 25, + "turns": 26, + "timed_out": false, + "left_folder": false + }, + "code-004": { + "answer": "fnd-pool-eviction-keeps-cells-the-tide-exposes-withi--tzseexbv0m0qw2jk", + "tokens": 44988, + "fixed_tokens": 1102, + "processed_tokens": 931029, + "tool_calls": 31, + "turns": 32, + "timed_out": false, + "left_folder": false + }, + "code-005": { + "answer": false, + "tokens": 20832, + "fixed_tokens": 1109, + "processed_tokens": 296495, + "tool_calls": 18, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "code-006": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 25518, + "fixed_tokens": 1100, + "processed_tokens": 254218, + "tool_calls": 13, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "code-011": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 7622, + "fixed_tokens": 1102, + "processed_tokens": 62021, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "code-012": { + "answer": "dec-ship-four-tern-quality-presets-adding-ultra-perf--3jg1wyq3bshv839c", + "tokens": 40420, + "fixed_tokens": 1109, + "processed_tokens": 918678, + "tool_calls": 36, + "turns": 35, + "timed_out": false, + "left_folder": false + }, + "code-018": { + "answer": "unknown", + "tokens": 48876, + "fixed_tokens": 1104, + "processed_tokens": 1019313, + "tool_calls": 34, + "turns": 32, + "timed_out": false, + "left_folder": false + }, + "code-019": { + "answer": "800 nits", + "tokens": 35194, + "fixed_tokens": 1109, + "processed_tokens": 391441, + "tool_calls": 15, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "multibranch-001": { + "answer": false, + "tokens": 41497, + "fixed_tokens": 1109, + "processed_tokens": 403799, + "tool_calls": 22, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "multibranch-002": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-tooltip-renderer-leave-it-as-is-0467--y44qjef2x7j0xv7e", + "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj" + ], + "tokens": 50041, + "fixed_tokens": 1110, + "processed_tokens": 309731, + "tool_calls": 24, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "multibranch-003": { + "answer": "sofia/party-invites-refresh-timer and marta/party-invites-refresh-on-open. Marta's decision dec-refresh-the-party-invites-list-once-each-time-th--zfak0g1cx14968sd rejects Sofia's dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3 (rejected_by), and the two branches share no code files (net/party/stats.rs vs net/party/loader.rs).", + "tokens": 23031, + "fixed_tokens": 1118, + "processed_tokens": 241664, + "tool_calls": 15, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "multibranch-004": { + "answer": "Draft PRs #41 (ines/vsm-pool) and #44 (tomasz/vsm-evict) touch the sun shadow page pool code (vsm_pool.rs, renamed page_pool.rs on main) that dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4 governs.", + "tokens": 28700, + "fixed_tokens": 1120, + "processed_tokens": 322926, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "multibranch-005": { + "answer": 4, + "tokens": 23919, + "fixed_tokens": 1115, + "processed_tokens": 217430, + "tool_calls": 32, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "multibranch-006": { + "answer": 4, + "tokens": 10571, + "fixed_tokens": 1114, + "processed_tokens": 64156, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-007": { + "answer": "engine/kestrel/world/ocean/lod.rs", + "tokens": 26965, + "fixed_tokens": 1119, + "processed_tokens": 205884, + "tool_calls": 23, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "multibranch-008": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-skiff-cert-blocker-40-hz-mode-misses-frames-in-t--8r81ajkewab5gymr" + ], + "tokens": 44724, + "fixed_tokens": 1115, + "processed_tokens": 748065, + "tool_calls": 28, + "turns": 24, + "timed_out": false, + "left_folder": false + }, + "multibranch-009": { + "answer": [ + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj" + ], + "tokens": 54242, + "fixed_tokens": 1113, + "processed_tokens": 727982, + "tool_calls": 25, + "turns": 23, + "timed_out": false, + "left_folder": false + }, + "multibranch-010": { + "answer": 8, + "tokens": 52357, + "fixed_tokens": 1120, + "processed_tokens": 992317, + "tool_calls": 29, + "turns": 29, + "timed_out": false, + "left_folder": false + }, + "multibranch-011": { + "answer": 50, + "tokens": 8959, + "fixed_tokens": 1111, + "processed_tokens": 48587, + "tool_calls": 5, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "multibranch-012": { + "answer": [ + "cit-code-enqueue-in-the-ambient-chatter-queue--ksz0m3yhk8a0j0qq" + ], + "tokens": 69094, + "fixed_tokens": 1114, + "processed_tokens": 961549, + "tool_calls": 27, + "turns": 26, + "timed_out": false, + "left_folder": false + }, + "refactor-001": { + "answer": "unknown", + "tokens": 40462, + "fixed_tokens": 1112, + "processed_tokens": 954274, + "tool_calls": 36, + "turns": 37, + "timed_out": false, + "left_folder": false + }, + "refactor-002": { + "answer": "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "tokens": 31926, + "fixed_tokens": 1109, + "processed_tokens": 403954, + "tool_calls": 17, + "turns": 18, + "timed_out": false, + "left_folder": false + }, + "refactor-003": { + "answer": "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "tokens": 27830, + "fixed_tokens": 1108, + "processed_tokens": 321601, + "tool_calls": 14, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "refactor-004": { + "answer": "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "tokens": 39414, + "fixed_tokens": 1116, + "processed_tokens": 864090, + "tool_calls": 28, + "turns": 29, + "timed_out": false, + "left_folder": false + }, + "refactor-005": { + "answer": "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "tokens": 29818, + "fixed_tokens": 1111, + "processed_tokens": 426178, + "tool_calls": 20, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "refactor-006": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw" + ], + "tokens": 11237, + "fixed_tokens": 1110, + "processed_tokens": 133006, + "tool_calls": 14, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "refactor-007": { + "answer": "con-the-tide-step-allocates-no-memory--bgce2y8wndwrwe0r", + "tokens": 20856, + "fixed_tokens": 1117, + "processed_tokens": 193402, + "tool_calls": 13, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "refactor-008": { + "answer": "unknown", + "tokens": 62254, + "fixed_tokens": 1108, + "processed_tokens": 1639493, + "tool_calls": 41, + "turns": 41, + "timed_out": false, + "left_folder": false + }, + "refactor-009": { + "answer": [ + "fnd-desync-checks-hash-three-state-blocks-every-30th--0vdm4w5cnjvqegjp", + "fnd-load-screens-compile-each-district-pipeline-list--8gbea0h3zt5r14gs", + "fnd-save-upgrades-run-every-step-from-the-save-versi--agp84ez57kd03vpw", + "fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx", + "fnd-tern-zeroes-history-weight-wherever-saltglass-se--hhgcrcq2n03rwr53", + "fnd-the-cache-key-hashes-its-inputs-in-the-order-the--w74ey1s5g91aranp", + "fnd-the-kite-physics-stats-keep-a-rolling-window-of--j6qv6qchghanqh1v", + "fnd-the-tide-phase-is-masked-to-one-cycle-on-every-s--p3by8f1vs0fj71ph" + ], + "tokens": 88932, + "fixed_tokens": 1109, + "processed_tokens": 1141768, + "tool_calls": 38, + "turns": 24, + "timed_out": false, + "left_folder": true + }, + "refactor-010": { + "answer": 7, + "tokens": 64268, + "fixed_tokens": 1110, + "processed_tokens": 896896, + "tool_calls": 24, + "turns": 25, + "timed_out": false, + "left_folder": true + }, + "refactor-011": { + "answer": "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "tokens": 50487, + "fixed_tokens": 1109, + "processed_tokens": 695537, + "tool_calls": 21, + "turns": 22, + "timed_out": false, + "left_folder": false + }, + "refactor-012": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-kite-physics-leave-it-as-is-0741--9xtc7xz57n0znv4v", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30" + ], + "tokens": 91547, + "fixed_tokens": 1115, + "processed_tokens": 1977474, + "tool_calls": 45, + "turns": 39, + "timed_out": false, + "left_folder": false + }, + "refactor-013": { + "answer": false, + "tokens": 23986, + "fixed_tokens": 1109, + "processed_tokens": 283328, + "tool_calls": 14, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "refactor-014": { + "answer": "The Decision dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq says every 30 frames; main hashes the state blocks every 60 frames (HASH_INTERVAL_FRAMES = 60 in net/coop/desync.rs, commit f6ef083, which has no Proof trailer and so is not tied to a record).", + "tokens": 28393, + "fixed_tokens": 1110, + "processed_tokens": 389109, + "tool_calls": 17, + "turns": 18, + "timed_out": false, + "left_folder": false + }, + "refactor-015": { + "answer": "On main it is `reclaim_pages` (formerly `evict_pages`) in `engine/lantern/shadows/page_pool.rs` (formerly `vsm_pool.rs`). The Proof note's citation is `cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr`, which points to `evict_pages` in `vsm_pool.rs` at ccd04c5.", + "tokens": 28637, + "fixed_tokens": 1112, + "processed_tokens": 296415, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "refactor-016": { + "answer": "engine/lantern/tern/reactive.rs", + "tokens": 29325, + "fixed_tokens": 1114, + "processed_tokens": 446582, + "tool_calls": 18, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "refactor-017": { + "answer": "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "tokens": 26661, + "fixed_tokens": 1113, + "processed_tokens": 260009, + "tool_calls": 14, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "refactor-018": { + "answer": "false: dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "tokens": 26759, + "fixed_tokens": 1111, + "processed_tokens": 300028, + "tool_calls": 14, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "refactor-019": { + "answer": "unknown", + "tokens": 42373, + "fixed_tokens": 1107, + "processed_tokens": 630854, + "tool_calls": 27, + "turns": 21, + "timed_out": false, + "left_folder": false + } + } +} diff --git a/benchmarks/proof-recall/runs/p3-r2-findcode.json b/benchmarks/proof-recall/runs/p3-r2-findcode.json new file mode 100644 index 00000000..0d225719 --- /dev/null +++ b/benchmarks/proof-recall/runs/p3-r2-findcode.json @@ -0,0 +1,834 @@ +{ + "run": { + "name": "p3-r2-findcode", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "c80612938d90f14747e3be3341c6434f3d78efcbd611bf5085facc8653eee5c2", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "6e94ef68632e522abca8eaf873b3238cf4ae45394a4810e66f1595c3fe2ee839" + }, + "answers": { + "behaviour-001": { + "answer": "unknown", + "tokens": 19568, + "fixed_tokens": 1106, + "processed_tokens": 138724, + "tool_calls": 10, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-002": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh" + ], + "tokens": 24010, + "fixed_tokens": 1114, + "processed_tokens": 207028, + "tool_calls": 12, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-003": { + "answer": [ + "dec-delay-autosaves-until-the-tide-turn-finishes--4agcgjjj70djdv6b", + "dec-save-player-changes-as-deltas-keyed-to-the-tide--1qxvc9985c8pzke8", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 24856, + "fixed_tokens": 1113, + "processed_tokens": 200418, + "tool_calls": 13, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-004": { + "answer": [ + "dec-brine-curve-with-an-hdr-shoulder-at-800-nits--fpznx0cnj4avsxk1" + ], + "tokens": 66286, + "fixed_tokens": 1114, + "processed_tokens": 1061210, + "tool_calls": 26, + "turns": 25, + "timed_out": false, + "left_folder": false + }, + "behaviour-005": { + "answer": [ + "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "dec-rejoin-the-co-op-session-after-meridian-resumes--gp2btfw62qm4khy9" + ], + "tokens": 31169, + "fixed_tokens": 1110, + "processed_tokens": 227397, + "tool_calls": 11, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-006": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-tide-turns-leave-inland-terrain-untextured-for-s--ccv6nhwk68kj732c" + ], + "tokens": 18836, + "fixed_tokens": 1119, + "processed_tokens": 182860, + "tool_calls": 13, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "behaviour-007": { + "answer": [ + "con-tide-state-must-be-bit-identical-on-every-peer--zrpnp8qjyxzkas7t", + "con-co-op-peers-must-agree-on-tide-state-bit-for-bit--4v0sdbc3t4ywzr3g" + ], + "tokens": 15859, + "fixed_tokens": 1109, + "processed_tokens": 109592, + "tool_calls": 8, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "behaviour-008": { + "answer": [ + "con-meridian-memory-budget--99q6099e423wdtqn", + "con-skiff-memory-budget--nf7qbhsnmcn6v479", + "con-streaming-pool-budget--8czwb4fjr5p5vpy7", + "con-render-target-memory-on-meridian--q4szajxse9c1csdv" + ], + "tokens": 14302, + "fixed_tokens": 1116, + "processed_tokens": 77262, + "tool_calls": 9, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "behaviour-009": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw", + "dec-use-host-authoritative-sync-for-co-op--17hqrmtja7pn98d6", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 28519, + "fixed_tokens": 1111, + "processed_tokens": 254127, + "tool_calls": 20, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-010": { + "answer": [ + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 54213, + "fixed_tokens": 1110, + "processed_tokens": 688684, + "tool_calls": 22, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "behaviour-011": { + "answer": "fnd-the-kite-physics-stats-keep-a-rolling-window-of--j6qv6qchghanqh1v", + "tokens": 38914, + "fixed_tokens": 1111, + "processed_tokens": 342374, + "tool_calls": 12, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-012": { + "answer": false, + "tokens": 11911, + "fixed_tokens": 1109, + "processed_tokens": 107390, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-013": { + "answer": false, + "tokens": 26795, + "fixed_tokens": 1108, + "processed_tokens": 247323, + "tool_calls": 15, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-014": { + "answer": false, + "tokens": 33171, + "fixed_tokens": 1116, + "processed_tokens": 317563, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "behaviour-015": { + "answer": [ + "dec-exclude-caustic-luminance-from-tern-s-history-cl--xhxjn26ss343v2k4", + "dec-lock-caustic-photon-jitter-to-the-tide-phase--85x10k4h11t8nx7c" + ], + "tokens": 25247, + "fixed_tokens": 1107, + "processed_tokens": 247419, + "tool_calls": 15, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "behaviour-016": { + "answer": [ + "dec-lower-the-skiff-dynamic-resolution-floor-to-504p--15esnt90gcn29enn", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1" + ], + "tokens": 30906, + "fixed_tokens": 1116, + "processed_tokens": 322903, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": true + }, + "branch-001": { + "answer": "#41", + "tokens": 16174, + "fixed_tokens": 1101, + "processed_tokens": 121478, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-003": { + "answer": "proposed", + "tokens": 15704, + "fixed_tokens": 1107, + "processed_tokens": 85252, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-004": { + "answer": "ines/vsm-pool", + "tokens": 18119, + "fixed_tokens": 1106, + "processed_tokens": 111134, + "tool_calls": 9, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-005": { + "answer": "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "tokens": 16240, + "fixed_tokens": 1103, + "processed_tokens": 79021, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "branch-006": { + "answer": "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "tokens": 13848, + "fixed_tokens": 1103, + "processed_tokens": 60752, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-007": { + "answer": 48, + "tokens": 15121, + "fixed_tokens": 1107, + "processed_tokens": 63085, + "tool_calls": 6, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-008": { + "answer": "fnd-tide-storms-push-skiff-40-hz-frames-to-28-4-ms-o--28gemvtznv4zztdc", + "tokens": 17275, + "fixed_tokens": 1104, + "processed_tokens": 133806, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-009": { + "answer": "27.0 ms", + "tokens": 19194, + "fixed_tokens": 1099, + "processed_tokens": 142506, + "tool_calls": 9, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-010": { + "answer": [ + "fnd-world-core-state-with-tide-level-of-detail--x947x5kj16hcngef", + "fnd-lantern-after-the-spring-tide-page-pool-change--9zbcsqh53mgv7h4p" + ], + "tokens": 14133, + "fixed_tokens": 1101, + "processed_tokens": 111868, + "tool_calls": 8, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-011": { + "answer": false, + "tokens": 25497, + "fixed_tokens": 1105, + "processed_tokens": 200128, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-012": { + "answer": "superseded", + "tokens": 11769, + "fixed_tokens": 1102, + "processed_tokens": 93937, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-013": { + "answer": "kenji/tide-lod", + "tokens": 16246, + "fixed_tokens": 1106, + "processed_tokens": 91307, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-014": { + "answer": "Priya Natarajan", + "tokens": 11482, + "fixed_tokens": 1105, + "processed_tokens": 59438, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-015": { + "answer": 3, + "tokens": 24964, + "fixed_tokens": 1109, + "processed_tokens": 172421, + "tool_calls": 10, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-016": { + "answer": "proposed", + "tokens": 12756, + "fixed_tokens": 1112, + "processed_tokens": 109754, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": true + }, + "branch-018": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96", + "tokens": 20512, + "fixed_tokens": 1108, + "processed_tokens": 185553, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-020": { + "answer": "Two open branches would replace it: origin/ines/vsm-pool (draft #41), whose dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96 supersedes dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4, and origin/tomasz/vsm-evict (draft #44), whose dec-use-virtual-shadow-maps-for-the-sun-with-coarser--jtcbmrm4c48ma8z8 also supersedes it.", + "tokens": 17041, + "fixed_tokens": 1104, + "processed_tokens": 118047, + "tool_calls": 9, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-021": { + "answer": "dec-cap-tide-storm-particles-at-6-000-on-skiff--resze52afjxyxys1", + "tokens": 19299, + "fixed_tokens": 1105, + "processed_tokens": 154987, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "branch-022": { + "answer": "priya/skiff-cpu", + "tokens": 27561, + "fixed_tokens": 1106, + "processed_tokens": 178824, + "tool_calls": 16, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "code-003": { + "answer": [ + "rsk-eviction-scoring-asks-the-tide-forecast-about-ev--zfwghmgds54h8y46" + ], + "tokens": 12938, + "fixed_tokens": 1099, + "processed_tokens": 108981, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "code-004": { + "answer": "fnd-pool-eviction-keeps-cells-the-tide-exposes-withi--tzseexbv0m0qw2jk", + "tokens": 15683, + "fixed_tokens": 1099, + "processed_tokens": 80203, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-005": { + "answer": "false dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "tokens": 22655, + "fixed_tokens": 1102, + "processed_tokens": 99626, + "tool_calls": 7, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-006": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 8502, + "fixed_tokens": 1104, + "processed_tokens": 61643, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-011": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 16149, + "fixed_tokens": 1101, + "processed_tokens": 107176, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "code-012": { + "answer": "dec-ship-four-tern-quality-presets-adding-ultra-perf--3jg1wyq3bshv839c", + "tokens": 11008, + "fixed_tokens": 1105, + "processed_tokens": 52808, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "code-018": { + "answer": "unknown", + "tokens": 11946, + "fixed_tokens": 1100, + "processed_tokens": 65577, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-019": { + "answer": "800 nits", + "tokens": 9171, + "fixed_tokens": 1098, + "processed_tokens": 64205, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-001": { + "answer": "true, dec-schedule-streaming-reads-by-deadline-instead-of--q2x5aknabgcssdj2", + "tokens": 23748, + "fixed_tokens": 1108, + "processed_tokens": 177300, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "multibranch-002": { + "answer": [ + "dec-tooltip-renderer-leave-it-as-is-0467--y44qjef2x7j0xv7e", + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj", + "dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3", + "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20" + ], + "tokens": 36931, + "fixed_tokens": 1114, + "processed_tokens": 197040, + "tool_calls": 11, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "multibranch-003": { + "answer": "sofia/party-invites-refresh-timer and marta/party-invites-refresh-on-open. Each proposes an accepted Decision on the same party-invites refresh question (dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3 and dec-refresh-the-party-invites-list-once-each-time-th--zfak0g1cx14968sd), and they change no code file in common (net/party/stats.rs vs net/party/loader.rs).", + "tokens": 13068, + "fixed_tokens": 1106, + "processed_tokens": 69775, + "tool_calls": 9, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-004": { + "answer": "#41 (origin/ines/vsm-pool) and #44 (origin/tomasz/vsm-evict). Both change engine/lantern/shadows/vsm_pool.rs, which the main sun-shadow Decision dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4 governs (via fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx). No open branch touches the code behind the local-lights atlas Decision dec-use-a-cached-shadow-atlas-for-local-lights--8p6f2v774he38t5j.", + "tokens": 26244, + "fixed_tokens": 1118, + "processed_tokens": 195741, + "tool_calls": 13, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "multibranch-005": { + "answer": 4, + "tokens": 34671, + "fixed_tokens": 1113, + "processed_tokens": 130530, + "tool_calls": 24, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "multibranch-006": { + "answer": 4, + "tokens": 17923, + "fixed_tokens": 1106, + "processed_tokens": 95369, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "multibranch-007": { + "answer": "engine/kestrel/world/ocean/lod.rs", + "tokens": 18551, + "fixed_tokens": 1119, + "processed_tokens": 139230, + "tool_calls": 10, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "multibranch-008": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-skiff-cert-blocker-40-hz-mode-misses-frames-in-t--8r81ajkewab5gymr" + ], + "tokens": 22966, + "fixed_tokens": 1115, + "processed_tokens": 137391, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "multibranch-009": { + "answer": [ + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj" + ], + "tokens": 21887, + "fixed_tokens": 1116, + "processed_tokens": 106177, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "multibranch-010": { + "answer": 8, + "tokens": 99794, + "fixed_tokens": 1112, + "processed_tokens": 3206866, + "tool_calls": 52, + "turns": 50, + "timed_out": false, + "left_folder": true + }, + "multibranch-011": { + "answer": 50, + "tokens": 15444, + "fixed_tokens": 1110, + "processed_tokens": 75432, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-012": { + "answer": [ + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr", + "cit-code-enqueue-in-the-ambient-chatter-queue--ksz0m3yhk8a0j0qq" + ], + "tokens": 27583, + "fixed_tokens": 1110, + "processed_tokens": 168656, + "tool_calls": 23, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-001": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "tokens": 29831, + "fixed_tokens": 1105, + "processed_tokens": 245300, + "tool_calls": 12, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "refactor-002": { + "answer": "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "tokens": 18741, + "fixed_tokens": 1107, + "processed_tokens": 125765, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-003": { + "answer": "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "tokens": 11785, + "fixed_tokens": 1109, + "processed_tokens": 85231, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-004": { + "answer": "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "tokens": 16868, + "fixed_tokens": 1109, + "processed_tokens": 150737, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "refactor-005": { + "answer": "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "tokens": 15461, + "fixed_tokens": 1091, + "processed_tokens": 118246, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "refactor-006": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw" + ], + "tokens": 12062, + "fixed_tokens": 614, + "processed_tokens": 97303, + "tool_calls": 10, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "refactor-007": { + "answer": "con-the-tide-step-allocates-no-memory--bgce2y8wndwrwe0r", + "tokens": 13420, + "fixed_tokens": 1106, + "processed_tokens": 130520, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "refactor-008": { + "answer": "iss-skiff-stays-at-the-lowest-resolution-for-two-sec--erke2h7xf7vepdv2", + "tokens": 16367, + "fixed_tokens": 1107, + "processed_tokens": 138308, + "tool_calls": 10, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "refactor-009": { + "answer": [ + "cit-code-desync-checksums--r7skab53zbz8xgkd", + "cit-code-save-upgrade-steps--6cyqfczkyzn9b5p6", + "cit-code-tide-phase-wrap--11vxtafqbq7ar7ez", + "cit-code-engine-kestrel-kites-stats-rs--c5d4ftr0yhnev7g4", + "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr", + "cit-code-pipeline-cache-warm-up--w6b39byrdfpnb1kn", + "cit-code-shader-cache-key--wdfsmaj5fmvewbv2" + ], + "tokens": 108825, + "fixed_tokens": 1104, + "processed_tokens": 2849952, + "tool_calls": 50, + "turns": 39, + "timed_out": false, + "left_folder": false + }, + "refactor-010": { + "answer": 7, + "tokens": 72385, + "fixed_tokens": 1108, + "processed_tokens": 1778292, + "tool_calls": 38, + "turns": 36, + "timed_out": false, + "left_folder": false + }, + "refactor-011": { + "answer": "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "tokens": 84386, + "fixed_tokens": 1114, + "processed_tokens": 852013, + "tool_calls": 27, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "refactor-012": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-kite-physics-leave-it-as-is-0741--9xtc7xz57n0znv4v" + ], + "tokens": 74142, + "fixed_tokens": 1109, + "processed_tokens": 820316, + "tool_calls": 21, + "turns": 22, + "timed_out": false, + "left_folder": false + }, + "refactor-013": { + "answer": false, + "tokens": 9138, + "fixed_tokens": 1106, + "processed_tokens": 87005, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "refactor-014": { + "answer": "The Decision dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq (accepted) says every 30 frames; the code on main hashes the state blocks every 60 frames (HASH_INTERVAL_FRAMES = 60 in net/coop/desync.rs), so the two disagree.", + "tokens": 14859, + "fixed_tokens": 1106, + "processed_tokens": 86437, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-015": { + "answer": "On main the function is `reclaim_pages` (formerly `evict_pages`) in `engine/lantern/shadows/page_pool.rs` (formerly `vsm_pool.rs`). The Proof note is `fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx`, which cites the old `evict_pages` in `vsm_pool.rs`.", + "tokens": 10607, + "fixed_tokens": 1106, + "processed_tokens": 119124, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "refactor-016": { + "answer": "engine/lantern/tern/reactive.rs", + "tokens": 7209, + "fixed_tokens": 1104, + "processed_tokens": 47342, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "refactor-017": { + "answer": "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "tokens": 13905, + "fixed_tokens": 1111, + "processed_tokens": 94027, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-018": { + "answer": false, + "tokens": 12370, + "fixed_tokens": 1111, + "processed_tokens": 89256, + "tool_calls": 8, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-019": { + "answer": "unknown", + "tokens": 13675, + "fixed_tokens": 1107, + "processed_tokens": 69113, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + } + } +} diff --git a/benchmarks/proof-recall/runs/p3-r2-nobrief.json b/benchmarks/proof-recall/runs/p3-r2-nobrief.json new file mode 100644 index 00000000..6dc881e6 --- /dev/null +++ b/benchmarks/proof-recall/runs/p3-r2-nobrief.json @@ -0,0 +1,836 @@ +{ + "run": { + "name": "p3-r2-nobrief", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "c80612938d90f14747e3be3341c6434f3d78efcbd611bf5085facc8653eee5c2", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "6e94ef68632e522abca8eaf873b3238cf4ae45394a4810e66f1595c3fe2ee839" + }, + "answers": { + "behaviour-001": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg" + ], + "tokens": 61691, + "fixed_tokens": 1120, + "processed_tokens": 2055706, + "tool_calls": 48, + "turns": 49, + "timed_out": false, + "left_folder": false + }, + "behaviour-002": { + "answer": [ + "dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg" + ], + "tokens": 33147, + "fixed_tokens": 1111, + "processed_tokens": 345942, + "tool_calls": 15, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "behaviour-003": { + "answer": [ + "dec-delay-autosaves-until-the-tide-turn-finishes--4agcgjjj70djdv6b", + "dec-save-player-changes-as-deltas-keyed-to-the-tide--1qxvc9985c8pzke8", + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 57953, + "fixed_tokens": 1120, + "processed_tokens": 1496216, + "tool_calls": 38, + "turns": 39, + "timed_out": false, + "left_folder": false + }, + "behaviour-004": { + "answer": "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq", + "tokens": 65739, + "fixed_tokens": 1112, + "processed_tokens": 1508556, + "tool_calls": 42, + "turns": 36, + "timed_out": false, + "left_folder": false + }, + "behaviour-005": { + "answer": [ + "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "dec-use-a-6-frame-rollback-window-with-1-frame-of-in--s53944zhkw103frk" + ], + "tokens": 47727, + "fixed_tokens": 1110, + "processed_tokens": 943237, + "tool_calls": 28, + "turns": 28, + "timed_out": false, + "left_folder": false + }, + "behaviour-006": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-tide-turns-leave-inland-terrain-untextured-for-s--ccv6nhwk68kj732c" + ], + "tokens": 24783, + "fixed_tokens": 1112, + "processed_tokens": 245643, + "tool_calls": 19, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "behaviour-007": { + "answer": [ + "con-tide-state-must-be-bit-identical-on-every-peer--zrpnp8qjyxzkas7t", + "con-co-op-peers-must-agree-on-tide-state-bit-for-bit--4v0sdbc3t4ywzr3g" + ], + "tokens": 38981, + "fixed_tokens": 1113, + "processed_tokens": 499294, + "tool_calls": 22, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "behaviour-008": { + "answer": [ + "con-skiff-memory-budget--nf7qbhsnmcn6v479", + "con-meridian-memory-budget--99q6099e423wdtqn", + "con-streaming-pool-budget--8czwb4fjr5p5vpy7", + "con-render-target-memory-on-meridian--q4szajxse9c1csdv" + ], + "tokens": 21165, + "fixed_tokens": 1115, + "processed_tokens": 121809, + "tool_calls": 9, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "behaviour-009": { + "answer": [ + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw", + "dec-use-host-authoritative-sync-for-co-op--17hqrmtja7pn98d6", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 20882, + "fixed_tokens": 1111, + "processed_tokens": 288015, + "tool_calls": 20, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "behaviour-010": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 53400, + "fixed_tokens": 1118, + "processed_tokens": 612478, + "tool_calls": 24, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "behaviour-011": { + "answer": "unknown", + "tokens": 121597, + "fixed_tokens": 1115, + "processed_tokens": 2566318, + "tool_calls": 49, + "turns": 38, + "timed_out": false, + "left_folder": false + }, + "behaviour-012": { + "answer": false, + "tokens": 27087, + "fixed_tokens": 1119, + "processed_tokens": 380893, + "tool_calls": 18, + "turns": 18, + "timed_out": false, + "left_folder": false + }, + "behaviour-013": { + "answer": false, + "tokens": 44854, + "fixed_tokens": 1111, + "processed_tokens": 546444, + "tool_calls": 21, + "turns": 18, + "timed_out": false, + "left_folder": false + }, + "behaviour-014": { + "answer": false, + "tokens": 25206, + "fixed_tokens": 1108, + "processed_tokens": 265774, + "tool_calls": 14, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "behaviour-015": { + "answer": [ + "dec-exclude-caustic-luminance-from-tern-s-history-cl--xhxjn26ss343v2k4", + "dec-lock-caustic-photon-jitter-to-the-tide-phase--85x10k4h11t8nx7c" + ], + "tokens": 38599, + "fixed_tokens": 1111, + "processed_tokens": 400782, + "tool_calls": 20, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "behaviour-016": { + "answer": [ + "dec-lower-the-skiff-dynamic-resolution-floor-to-504p--15esnt90gcn29enn", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1" + ], + "tokens": 44168, + "fixed_tokens": 1112, + "processed_tokens": 333665, + "tool_calls": 20, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "branch-001": { + "answer": 41, + "tokens": 14820, + "fixed_tokens": 1107, + "processed_tokens": 161488, + "tool_calls": 13, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "branch-003": { + "answer": "proposed", + "tokens": 14168, + "fixed_tokens": 1105, + "processed_tokens": 58869, + "tool_calls": 6, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-004": { + "answer": "ines/vsm-pool", + "tokens": 31919, + "fixed_tokens": 1105, + "processed_tokens": 404202, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "branch-005": { + "answer": "unknown", + "tokens": 43931, + "fixed_tokens": 1105, + "processed_tokens": 333784, + "tool_calls": 13, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "branch-006": { + "answer": "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "tokens": 15654, + "fixed_tokens": 1107, + "processed_tokens": 179968, + "tool_calls": 13, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "branch-007": { + "answer": 48, + "tokens": 32013, + "fixed_tokens": 1104, + "processed_tokens": 383264, + "tool_calls": 16, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "branch-008": { + "answer": "fnd-tide-storms-push-skiff-40-hz-frames-to-28-4-ms-o--28gemvtznv4zztdc", + "tokens": 28750, + "fixed_tokens": 1104, + "processed_tokens": 328595, + "tool_calls": 17, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "branch-009": { + "answer": "27.0 ms", + "tokens": 22155, + "fixed_tokens": 1111, + "processed_tokens": 123846, + "tool_calls": 13, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-010": { + "answer": [ + "fnd-world-core-state-with-tide-level-of-detail--x947x5kj16hcngef", + "fnd-lantern-after-the-spring-tide-page-pool-change--9zbcsqh53mgv7h4p" + ], + "tokens": 27306, + "fixed_tokens": 1104, + "processed_tokens": 195707, + "tool_calls": 12, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "branch-011": { + "answer": false, + "tokens": 42504, + "fixed_tokens": 1108, + "processed_tokens": 325650, + "tool_calls": 12, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "branch-012": { + "answer": "superseded", + "tokens": 25466, + "fixed_tokens": 1106, + "processed_tokens": 214889, + "tool_calls": 12, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "branch-013": { + "answer": "kenji/tide-lod", + "tokens": 30974, + "fixed_tokens": 1108, + "processed_tokens": 465216, + "tool_calls": 20, + "turns": 21, + "timed_out": false, + "left_folder": false + }, + "branch-014": { + "answer": "Priya Natarajan", + "tokens": 9406, + "fixed_tokens": 1108, + "processed_tokens": 106374, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-015": { + "answer": 3, + "tokens": 11760, + "fixed_tokens": 1105, + "processed_tokens": 98376, + "tool_calls": 9, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-016": { + "answer": "proposed", + "tokens": 23642, + "fixed_tokens": 1107, + "processed_tokens": 277628, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "branch-018": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96", + "tokens": 15585, + "fixed_tokens": 1105, + "processed_tokens": 133056, + "tool_calls": 11, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-020": { + "answer": "ines/vsm-pool and tomasz/vsm-evict. Both branches carry a Decision that supersedes main's sun-shadow Decision dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4: ines/vsm-pool has dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96 (accepted), and tomasz/vsm-evict has dec-use-virtual-shadow-maps-for-the-sun-with-coarser--jtcbmrm4c48ma8z8 (proposed).", + "tokens": 14303, + "fixed_tokens": 1105, + "processed_tokens": 158633, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "branch-021": { + "answer": "dec-cap-tide-storm-particles-at-6-000-on-skiff--resze52afjxyxys1", + "tokens": 16198, + "fixed_tokens": 1108, + "processed_tokens": 152668, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "branch-022": { + "answer": "priya/skiff-cpu", + "tokens": 48898, + "fixed_tokens": 1106, + "processed_tokens": 314369, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "code-003": { + "answer": [ + "fnd-pool-eviction-keeps-cells-the-tide-exposes-withi--tzseexbv0m0qw2jk", + "rsk-eviction-scoring-asks-the-tide-forecast-about-ev--zfwghmgds54h8y46" + ], + "tokens": 54041, + "fixed_tokens": 1114, + "processed_tokens": 1009361, + "tool_calls": 29, + "turns": 26, + "timed_out": false, + "left_folder": false + }, + "code-004": { + "answer": "fnd-pool-eviction-keeps-cells-the-tide-exposes-withi--tzseexbv0m0qw2jk", + "tokens": 44254, + "fixed_tokens": 1106, + "processed_tokens": 724219, + "tool_calls": 23, + "turns": 24, + "timed_out": false, + "left_folder": false + }, + "code-005": { + "answer": false, + "tokens": 28747, + "fixed_tokens": 1106, + "processed_tokens": 304552, + "tool_calls": 14, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "code-006": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 15763, + "fixed_tokens": 1110, + "processed_tokens": 203890, + "tool_calls": 14, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "code-011": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 10176, + "fixed_tokens": 1107, + "processed_tokens": 113091, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "code-012": { + "answer": "dec-ship-four-tern-quality-presets-adding-ultra-perf--3jg1wyq3bshv839c", + "tokens": 33675, + "fixed_tokens": 1103, + "processed_tokens": 222307, + "tool_calls": 10, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "code-018": { + "answer": "unknown", + "tokens": 39312, + "fixed_tokens": 1108, + "processed_tokens": 465303, + "tool_calls": 18, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "code-019": { + "answer": "800 nits", + "tokens": 44006, + "fixed_tokens": 1108, + "processed_tokens": 728159, + "tool_calls": 22, + "turns": 23, + "timed_out": false, + "left_folder": false + }, + "multibranch-001": { + "answer": false, + "tokens": 46660, + "fixed_tokens": 1116, + "processed_tokens": 520914, + "tool_calls": 20, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "multibranch-002": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-tooltip-renderer-leave-it-as-is-0467--y44qjef2x7j0xv7e", + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj", + "dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3" + ], + "tokens": 56607, + "fixed_tokens": 1110, + "processed_tokens": 462633, + "tool_calls": 25, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "multibranch-003": { + "answer": "marta/party-invites-refresh-on-open and sofia/party-invites-refresh-timer. Marta's decision dec-refresh-the-party-invites-list-once-each-time-th--zfak0g1cx14968sd rejects Sofia's dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3, and the two branches touch different code files (net/party/loader.rs and net/party/stats.rs).", + "tokens": 42665, + "fixed_tokens": 1117, + "processed_tokens": 280729, + "tool_calls": 12, + "turns": 12, + "timed_out": false, + "left_folder": true + }, + "multibranch-004": { + "answer": "PRs #41 and #44. Both branches (ines/vsm-pool, tomasz/vsm-evict) change engine/lantern/shadows/vsm_pool.rs, the sun shadow page pool that dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4 governs on main (renamed to page_pool.rs there).", + "tokens": 38974, + "fixed_tokens": 1116, + "processed_tokens": 439987, + "tool_calls": 20, + "turns": 21, + "timed_out": false, + "left_folder": false + }, + "multibranch-005": { + "answer": 4, + "tokens": 36836, + "fixed_tokens": 1112, + "processed_tokens": 273938, + "tool_calls": 21, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "multibranch-006": { + "answer": 4, + "tokens": 9455, + "fixed_tokens": 1114, + "processed_tokens": 61451, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-007": { + "answer": "engine/kestrel/world/ocean/", + "tokens": 17330, + "fixed_tokens": 1115, + "processed_tokens": 235823, + "tool_calls": 16, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "multibranch-008": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-skiff-cert-blocker-40-hz-mode-misses-frames-in-t--8r81ajkewab5gymr" + ], + "tokens": 37970, + "fixed_tokens": 1112, + "processed_tokens": 420482, + "tool_calls": 25, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "multibranch-009": { + "answer": [ + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj" + ], + "tokens": 43804, + "fixed_tokens": 1115, + "processed_tokens": 417777, + "tool_calls": 16, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "multibranch-010": { + "answer": 8, + "tokens": 63955, + "fixed_tokens": 1119, + "processed_tokens": 1232539, + "tool_calls": 31, + "turns": 32, + "timed_out": false, + "left_folder": false + }, + "multibranch-011": { + "answer": "#50", + "tokens": 7770, + "fixed_tokens": 1111, + "processed_tokens": 58151, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-012": { + "answer": [ + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr", + "cit-code-enqueue-in-the-ambient-chatter-queue--ksz0m3yhk8a0j0qq" + ], + "tokens": 50827, + "fixed_tokens": 1112, + "processed_tokens": 528022, + "tool_calls": 26, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "refactor-001": { + "answer": "unknown", + "tokens": 28994, + "fixed_tokens": 1109, + "processed_tokens": 656433, + "tool_calls": 31, + "turns": 32, + "timed_out": false, + "left_folder": false + }, + "refactor-002": { + "answer": "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "tokens": 26182, + "fixed_tokens": 621, + "processed_tokens": 353123, + "tool_calls": 15, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "refactor-003": { + "answer": "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "tokens": 16514, + "fixed_tokens": 1108, + "processed_tokens": 237004, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "refactor-004": { + "answer": "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "tokens": 14745, + "fixed_tokens": 614, + "processed_tokens": 214875, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "refactor-005": { + "answer": "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "tokens": 27928, + "fixed_tokens": 615, + "processed_tokens": 282203, + "tool_calls": 14, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "refactor-006": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw" + ], + "tokens": 9850, + "fixed_tokens": 610, + "processed_tokens": 86588, + "tool_calls": 12, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-007": { + "answer": "con-the-tide-step-allocates-no-memory--bgce2y8wndwrwe0r", + "tokens": 16798, + "fixed_tokens": 1117, + "processed_tokens": 154963, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "refactor-008": { + "answer": "iss-skiff-stays-at-the-lowest-resolution-for-two-sec--erke2h7xf7vepdv2", + "tokens": 43234, + "fixed_tokens": 1108, + "processed_tokens": 1096396, + "tool_calls": 33, + "turns": 34, + "timed_out": false, + "left_folder": false + }, + "refactor-009": { + "answer": [ + "fnd-desync-checks-hash-three-state-blocks-every-30th--0vdm4w5cnjvqegjp", + "fnd-load-screens-compile-each-district-pipeline-list--8gbea0h3zt5r14gs", + "fnd-save-upgrades-run-every-step-from-the-save-versi--agp84ez57kd03vpw", + "fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx", + "fnd-tern-zeroes-history-weight-wherever-saltglass-se--hhgcrcq2n03rwr53", + "fnd-the-cache-key-hashes-its-inputs-in-the-order-the--w74ey1s5g91aranp", + "fnd-the-kite-physics-stats-keep-a-rolling-window-of--j6qv6qchghanqh1v", + "fnd-the-tide-phase-is-masked-to-one-cycle-on-every-s--p3by8f1vs0fj71ph" + ], + "tokens": 62519, + "fixed_tokens": 1116, + "processed_tokens": 832425, + "tool_calls": 21, + "turns": 22, + "timed_out": false, + "left_folder": true + }, + "refactor-010": { + "answer": 7, + "tokens": 67834, + "fixed_tokens": 1110, + "processed_tokens": 1524597, + "tool_calls": 33, + "turns": 34, + "timed_out": false, + "left_folder": true + }, + "refactor-011": { + "answer": "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "tokens": 43041, + "fixed_tokens": 1112, + "processed_tokens": 492437, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "refactor-012": { + "answer": [ + "dec-kite-physics-leave-it-as-is-0741--9xtc7xz57n0znv4v", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq", + "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k" + ], + "tokens": 84726, + "fixed_tokens": 1113, + "processed_tokens": 1572047, + "tool_calls": 31, + "turns": 30, + "timed_out": false, + "left_folder": false + }, + "refactor-013": { + "answer": "false dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq", + "tokens": 26341, + "fixed_tokens": 1116, + "processed_tokens": 294878, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "refactor-014": { + "answer": "The Decision dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq says every 30 frames, but main hashes the state blocks every 60 frames (HASH_INTERVAL_FRAMES = 60 in net/coop/desync.rs).", + "tokens": 23960, + "fixed_tokens": 1109, + "processed_tokens": 256823, + "tool_calls": 13, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "refactor-015": { + "answer": "On main it is `reclaim_pages`, a method on `PagePool` in `engine/lantern/shadows/page_pool.rs` (renamed from `evict_pages` in `vsm_pool.rs`, per the cited note `fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx`).", + "tokens": 31724, + "fixed_tokens": 1109, + "processed_tokens": 373616, + "tool_calls": 15, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "refactor-016": { + "answer": "engine/lantern/tern/reactive.rs", + "tokens": 45866, + "fixed_tokens": 1110, + "processed_tokens": 651228, + "tool_calls": 19, + "turns": 20, + "timed_out": false, + "left_folder": false + }, + "refactor-017": { + "answer": "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "tokens": 27524, + "fixed_tokens": 1107, + "processed_tokens": 382563, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "refactor-018": { + "answer": "false, dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "tokens": 22661, + "fixed_tokens": 1116, + "processed_tokens": 219923, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "refactor-019": { + "answer": "unknown", + "tokens": 45823, + "fixed_tokens": 1109, + "processed_tokens": 930382, + "tool_calls": 29, + "turns": 29, + "timed_out": false, + "left_folder": false + } + } +} diff --git a/benchmarks/proof-recall/runs/p3s-r1-findcode.json b/benchmarks/proof-recall/runs/p3s-r1-findcode.json new file mode 100644 index 00000000..370126ad --- /dev/null +++ b/benchmarks/proof-recall/runs/p3s-r1-findcode.json @@ -0,0 +1,840 @@ +{ + "run": { + "name": "p3s-r1-findcode", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "0865e0ff999a895825e73296d226162d9dcc55e31be448247745ff113f6fb17a", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "27059ece934b7127047915a0cbb02a51ed8764ad65bc8fc149adf30d3564fb7c" + }, + "answers": { + "behaviour-001": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg" + ], + "tokens": 29952, + "fixed_tokens": 1114, + "processed_tokens": 167782, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-002": { + "answer": [ + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh" + ], + "tokens": 21830, + "fixed_tokens": 1114, + "processed_tokens": 210805, + "tool_calls": 12, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-003": { + "answer": [ + "dec-delay-autosaves-until-the-tide-turn-finishes--4agcgjjj70djdv6b", + "dec-save-player-changes-as-deltas-keyed-to-the-tide--1qxvc9985c8pzke8", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 22113, + "fixed_tokens": 1114, + "processed_tokens": 132399, + "tool_calls": 13, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "behaviour-004": { + "answer": [ + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 71649, + "fixed_tokens": 1119, + "processed_tokens": 1234658, + "tool_calls": 27, + "turns": 27, + "timed_out": false, + "left_folder": false + }, + "behaviour-005": { + "answer": [ + "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "dec-rejoin-the-co-op-session-after-meridian-resumes--gp2btfw62qm4khy9" + ], + "tokens": 22667, + "fixed_tokens": 1119, + "processed_tokens": 166319, + "tool_calls": 11, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "behaviour-006": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-tide-turns-leave-inland-terrain-untextured-for-s--ccv6nhwk68kj732c" + ], + "tokens": 19861, + "fixed_tokens": 1123, + "processed_tokens": 168437, + "tool_calls": 12, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "behaviour-007": { + "answer": [ + "con-tide-state-must-be-bit-identical-on-every-peer--zrpnp8qjyxzkas7t", + "con-co-op-peers-must-agree-on-tide-state-bit-for-bit--4v0sdbc3t4ywzr3g" + ], + "tokens": 13541, + "fixed_tokens": 1117, + "processed_tokens": 99455, + "tool_calls": 9, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "behaviour-008": { + "answer": [ + "con-meridian-memory-budget--99q6099e423wdtqn", + "con-skiff-memory-budget--nf7qbhsnmcn6v479", + "con-streaming-pool-budget--8czwb4fjr5p5vpy7", + "con-render-target-memory-on-meridian--q4szajxse9c1csdv" + ], + "tokens": 15941, + "fixed_tokens": 1119, + "processed_tokens": 117752, + "tool_calls": 11, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "behaviour-009": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw", + "dec-use-host-authoritative-sync-for-co-op--17hqrmtja7pn98d6", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 27216, + "fixed_tokens": 1117, + "processed_tokens": 193357, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-010": { + "answer": [ + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k" + ], + "tokens": 27258, + "fixed_tokens": 1115, + "processed_tokens": 243368, + "tool_calls": 15, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-011": { + "answer": [ + "fnd-the-kite-physics-stats-keep-a-rolling-window-of--j6qv6qchghanqh1v", + "iss-skiff-stays-at-the-lowest-resolution-for-two-sec--erke2h7xf7vepdv2", + "iss-a-timed-out-cache-fetch-is-retried-once-and-then--sn5thmgf02v8rfe4" + ], + "tokens": 60288, + "fixed_tokens": 1111, + "processed_tokens": 534984, + "tool_calls": 15, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "behaviour-012": { + "answer": false, + "tokens": 14333, + "fixed_tokens": 1119, + "processed_tokens": 85825, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "behaviour-013": { + "answer": false, + "tokens": 23459, + "fixed_tokens": 1112, + "processed_tokens": 201852, + "tool_calls": 12, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-014": { + "answer": false, + "tokens": 29686, + "fixed_tokens": 1114, + "processed_tokens": 210351, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-015": { + "answer": [ + "dec-exclude-caustic-luminance-from-tern-s-history-cl--xhxjn26ss343v2k4", + "dec-lock-caustic-photon-jitter-to-the-tide-phase--85x10k4h11t8nx7c" + ], + "tokens": 20644, + "fixed_tokens": 1113, + "processed_tokens": 161287, + "tool_calls": 10, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "behaviour-016": { + "answer": [ + "dec-lower-the-skiff-dynamic-resolution-floor-to-504p--15esnt90gcn29enn", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1" + ], + "tokens": 33523, + "fixed_tokens": 1119, + "processed_tokens": 379509, + "tool_calls": 18, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "branch-001": { + "answer": 41, + "tokens": 16501, + "fixed_tokens": 1114, + "processed_tokens": 101531, + "tool_calls": 9, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-003": { + "answer": "proposed", + "tokens": 12858, + "fixed_tokens": 1106, + "processed_tokens": 63024, + "tool_calls": 5, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-004": { + "answer": "ines/vsm-pool", + "tokens": 14681, + "fixed_tokens": 1108, + "processed_tokens": 61990, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-005": { + "answer": "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "tokens": 20259, + "fixed_tokens": 1107, + "processed_tokens": 75953, + "tool_calls": 5, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-006": { + "answer": "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "tokens": 13674, + "fixed_tokens": 1108, + "processed_tokens": 60429, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-007": { + "answer": 48, + "tokens": 13658, + "fixed_tokens": 1109, + "processed_tokens": 60210, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-008": { + "answer": "fnd-tide-storms-push-skiff-40-hz-frames-to-28-4-ms-o--28gemvtznv4zztdc", + "tokens": 11829, + "fixed_tokens": 1108, + "processed_tokens": 99244, + "tool_calls": 8, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-009": { + "answer": "27.0 ms", + "tokens": 17390, + "fixed_tokens": 1107, + "processed_tokens": 119490, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-010": { + "answer": [ + "fnd-world-core-state-with-tide-level-of-detail--x947x5kj16hcngef", + "fnd-lantern-after-the-spring-tide-page-pool-change--9zbcsqh53mgv7h4p" + ], + "tokens": 23362, + "fixed_tokens": 1107, + "processed_tokens": 113916, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-011": { + "answer": false, + "tokens": 24552, + "fixed_tokens": 1112, + "processed_tokens": 152251, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-012": { + "answer": "superseded", + "tokens": 18417, + "fixed_tokens": 1105, + "processed_tokens": 134680, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-013": { + "answer": "kenji/tide-lod", + "tokens": 16355, + "fixed_tokens": 1105, + "processed_tokens": 90897, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-014": { + "answer": "priya-natarajan", + "tokens": 10147, + "fixed_tokens": 1115, + "processed_tokens": 42381, + "tool_calls": 3, + "turns": 4, + "timed_out": false, + "left_folder": false + }, + "branch-015": { + "answer": 3, + "tokens": 19438, + "fixed_tokens": 1107, + "processed_tokens": 123866, + "tool_calls": 12, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-016": { + "answer": "proposed", + "tokens": 16039, + "fixed_tokens": 1107, + "processed_tokens": 109291, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-018": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96", + "tokens": 30794, + "fixed_tokens": 1106, + "processed_tokens": 281080, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "branch-020": { + "answer": "Two open branches would replace main's accepted sun-shadow Decision dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4: tomasz/vsm-evict (draft #44), via dec-use-virtual-shadow-maps-for-the-sun-with-coarser--jtcbmrm4c48ma8z8, and ines/vsm-pool (draft #41), via dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96.", + "tokens": 15176, + "fixed_tokens": 1105, + "processed_tokens": 113944, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-021": { + "answer": "dec-cap-tide-storm-particles-at-6-000-on-skiff--resze52afjxyxys1", + "tokens": 21905, + "fixed_tokens": 1106, + "processed_tokens": 109981, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-022": { + "answer": "priya/skiff-cpu", + "tokens": 27276, + "fixed_tokens": 1105, + "processed_tokens": 172718, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "code-003": { + "answer": [ + "rsk-eviction-scoring-asks-the-tide-forecast-about-ev--zfwghmgds54h8y46" + ], + "tokens": 26002, + "fixed_tokens": 1110, + "processed_tokens": 256888, + "tool_calls": 15, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "code-004": { + "answer": "fnd-pool-eviction-keeps-cells-the-tide-exposes-withi--tzseexbv0m0qw2jk", + "tokens": 10487, + "fixed_tokens": 1105, + "processed_tokens": 93420, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "code-005": { + "answer": "false dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "tokens": 16479, + "fixed_tokens": 1102, + "processed_tokens": 115847, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "code-006": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 8733, + "fixed_tokens": 1103, + "processed_tokens": 62280, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-011": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 12355, + "fixed_tokens": 1102, + "processed_tokens": 72893, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "code-012": { + "answer": "dec-ship-four-tern-quality-presets-adding-ultra-perf--3jg1wyq3bshv839c", + "tokens": 12411, + "fixed_tokens": 1101, + "processed_tokens": 86385, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "code-018": { + "answer": "unknown", + "tokens": 10923, + "fixed_tokens": 1100, + "processed_tokens": 63554, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-019": { + "answer": "800 nits", + "tokens": 9122, + "fixed_tokens": 1107, + "processed_tokens": 64738, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-001": { + "answer": false, + "tokens": 34382, + "fixed_tokens": 1119, + "processed_tokens": 325427, + "tool_calls": 12, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "multibranch-002": { + "answer": [ + "dec-freeze-tide-cells-beyond-600-m-and-catch-them-up--sxbt81hg4vnxzp7t", + "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3", + "dec-refresh-the-party-invites-list-once-each-time-th--zfak0g1cx14968sd", + "dec-tooltip-renderer-leave-it-as-is-0467--y44qjef2x7j0xv7e", + "dec-tooltip-renderer-build-the-events-on-a-worker-th--fpp5wz0rvafxefwn", + "dec-tooltip-renderer-lower-the-tooltip-renderer-list--6gn4dq3h7tna1j1c", + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96", + "dec-use-virtual-shadow-maps-for-the-sun-with-coarser--jtcbmrm4c48ma8z8" + ], + "tokens": 30462, + "fixed_tokens": 1115, + "processed_tokens": 206213, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "multibranch-003": { + "answer": "sofia/party-invites-refresh-timer and marta/party-invites-refresh-on-open. Their Proof conflicts on issue iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj: dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3 is accepted on the first branch and rejected on the second, and the second also adds its own accepted Decision dec-refresh-the-party-invites-list-once-each-time-th--zfak0g1cx14968sd. They touch different code files (net/party/stats.rs vs net/party/loader.rs).", + "tokens": 13155, + "fixed_tokens": 1118, + "processed_tokens": 70144, + "tool_calls": 9, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-004": { + "answer": "PRs #41 and #44. Both touch engine/lantern/shadows/vsm_pool.rs (renamed to page_pool.rs on main), which is governed by the main sun shadow Decision dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4 via finding fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx.", + "tokens": 36041, + "fixed_tokens": 1119, + "processed_tokens": 204619, + "tool_calls": 12, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "multibranch-005": { + "answer": 4, + "tokens": 27855, + "fixed_tokens": 1118, + "processed_tokens": 281446, + "tool_calls": 22, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "multibranch-006": { + "answer": 4, + "tokens": 23633, + "fixed_tokens": 1117, + "processed_tokens": 109893, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "multibranch-007": { + "answer": "engine/kestrel/world/ocean/", + "tokens": 18643, + "fixed_tokens": 1113, + "processed_tokens": 110810, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "multibranch-008": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-skiff-cert-blocker-40-hz-mode-misses-frames-in-t--8r81ajkewab5gymr" + ], + "tokens": 42763, + "fixed_tokens": 1115, + "processed_tokens": 365474, + "tool_calls": 12, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "multibranch-009": { + "answer": [ + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj", + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p" + ], + "tokens": 21635, + "fixed_tokens": 1109, + "processed_tokens": 106453, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "multibranch-010": { + "answer": 8, + "tokens": 103285, + "fixed_tokens": 1120, + "processed_tokens": 1548674, + "tool_calls": 36, + "turns": 29, + "timed_out": false, + "left_folder": false + }, + "multibranch-011": { + "answer": "50", + "tokens": 9161, + "fixed_tokens": 1115, + "processed_tokens": 63236, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-012": { + "answer": [ + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr", + "cit-code-enqueue-in-the-ambient-chatter-queue--ksz0m3yhk8a0j0qq" + ], + "tokens": 71446, + "fixed_tokens": 1114, + "processed_tokens": 304243, + "tool_calls": 19, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "refactor-001": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "tokens": 13828, + "fixed_tokens": 1112, + "processed_tokens": 109094, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-002": { + "answer": "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "tokens": 17487, + "fixed_tokens": 1110, + "processed_tokens": 110405, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-003": { + "answer": "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "tokens": 13487, + "fixed_tokens": 1110, + "processed_tokens": 92386, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-004": { + "answer": "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "tokens": 18905, + "fixed_tokens": 1108, + "processed_tokens": 99090, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-005": { + "answer": "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "tokens": 14504, + "fixed_tokens": 1109, + "processed_tokens": 101527, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "refactor-006": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw" + ], + "tokens": 8622, + "fixed_tokens": 1109, + "processed_tokens": 62919, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-007": { + "answer": "con-the-tide-step-allocates-no-memory--bgce2y8wndwrwe0r", + "tokens": 13853, + "fixed_tokens": 1112, + "processed_tokens": 81265, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-008": { + "answer": "iss-skiff-stays-at-the-lowest-resolution-for-two-sec--erke2h7xf7vepdv2", + "tokens": 13505, + "fixed_tokens": 1111, + "processed_tokens": 93139, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-009": { + "answer": [ + "cit-code-desync-checksums--r7skab53zbz8xgkd", + "cit-code-shader-cache-key--wdfsmaj5fmvewbv2", + "cit-code-pipeline-cache-warm-up--w6b39byrdfpnb1kn", + "cit-code-save-upgrade-steps--6cyqfczkyzn9b5p6", + "cit-code-tide-phase-wrap--11vxtafqbq7ar7ez", + "cit-code-engine-kestrel-kites-stats-rs--c5d4ftr0yhnev7g4", + "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr" + ], + "tokens": 79869, + "fixed_tokens": 1111, + "processed_tokens": 1366809, + "tool_calls": 31, + "turns": 28, + "timed_out": false, + "left_folder": false + }, + "refactor-010": { + "answer": 7, + "tokens": 65706, + "fixed_tokens": 1113, + "processed_tokens": 1040944, + "tool_calls": 28, + "turns": 28, + "timed_out": false, + "left_folder": false + }, + "refactor-011": { + "answer": "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "tokens": 73748, + "fixed_tokens": 1114, + "processed_tokens": 509502, + "tool_calls": 25, + "turns": 15, + "timed_out": false, + "left_folder": false + }, + "refactor-012": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-kite-physics-leave-it-as-is-0741--9xtc7xz57n0znv4v" + ], + "tokens": 32685, + "fixed_tokens": 1118, + "processed_tokens": 301109, + "tool_calls": 15, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "refactor-013": { + "answer": false, + "tokens": 8649, + "fixed_tokens": 1116, + "processed_tokens": 61766, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-014": { + "answer": "The Decision dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq says the state blocks are hashed every 30 frames, but main hashes them every 60 frames (HASH_INTERVAL_FRAMES = 60 in net/coop/desync.rs, set by commit f6ef083).", + "tokens": 15135, + "fixed_tokens": 1108, + "processed_tokens": 100953, + "tool_calls": 9, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "refactor-015": { + "answer": "On main it is `reclaim_pages` in `engine/lantern/shadows/page_pool.rs` (formerly `evict_pages` in `vsm_pool.rs`; renamed in df77028 and eb579ca). The note is finding fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx.", + "tokens": 9488, + "fixed_tokens": 1114, + "processed_tokens": 76099, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-016": { + "answer": "engine/lantern/tern/reactive.rs", + "tokens": 7967, + "fixed_tokens": 1116, + "processed_tokens": 61074, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-017": { + "answer": "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "tokens": 13427, + "fixed_tokens": 1114, + "processed_tokens": 59391, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "refactor-018": { + "answer": { + "answer": false, + "record": "dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh" + }, + "tokens": 14786, + "fixed_tokens": 1113, + "processed_tokens": 80290, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-019": { + "answer": "unknown", + "tokens": 24203, + "fixed_tokens": 1107, + "processed_tokens": 121917, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + } + } +} diff --git a/benchmarks/proof-recall/runs/p3s-r2-findcode.json b/benchmarks/proof-recall/runs/p3s-r2-findcode.json new file mode 100644 index 00000000..b55d64d6 --- /dev/null +++ b/benchmarks/proof-recall/runs/p3s-r2-findcode.json @@ -0,0 +1,827 @@ +{ + "run": { + "name": "p3s-r2-findcode", + "protocol": "one fresh session per question", + "models": ["claude-haiku-5-5"], + "find_hash": "0865e0ff999a895825e73296d226162d9dcc55e31be448247745ff113f6fb17a", + "state_hash": "be19faeb18e0059bde6482cc2b7e2f1df8368d691e4a903ce1b63cc9aa5a177b", + "read_hash": "27059ece934b7127047915a0cbb02a51ed8764ad65bc8fc149adf30d3564fb7c" + }, + "answers": { + "behaviour-001": { + "answer": "unknown", + "tokens": 21533, + "fixed_tokens": 1113, + "processed_tokens": 160168, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "behaviour-002": { + "answer": [ + "dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg" + ], + "tokens": 26716, + "fixed_tokens": 1115, + "processed_tokens": 274192, + "tool_calls": 16, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-003": { + "answer": [ + "dec-delay-autosaves-until-the-tide-turn-finishes--4agcgjjj70djdv6b", + "dec-save-player-changes-as-deltas-keyed-to-the-tide--1qxvc9985c8pzke8", + "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w" + ], + "tokens": 22207, + "fixed_tokens": 1115, + "processed_tokens": 126590, + "tool_calls": 13, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "behaviour-004": { + "answer": [ + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 60385, + "fixed_tokens": 1121, + "processed_tokens": 826139, + "tool_calls": 25, + "turns": 26, + "timed_out": false, + "left_folder": false + }, + "behaviour-005": { + "answer": [ + "dec-raise-the-shader-cache-fetch-timeout-from-5-seco--4sgdf42ajyct6gt6", + "dec-rejoin-the-co-op-session-after-meridian-resumes--gp2btfw62qm4khy9" + ], + "tokens": 16819, + "fixed_tokens": 1123, + "processed_tokens": 127444, + "tool_calls": 9, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-006": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-tide-turns-leave-inland-terrain-untextured-for-s--ccv6nhwk68kj732c" + ], + "tokens": 17613, + "fixed_tokens": 1113, + "processed_tokens": 129874, + "tool_calls": 13, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-007": { + "answer": [ + "con-tide-state-must-be-bit-identical-on-every-peer--zrpnp8qjyxzkas7t", + "con-co-op-peers-must-agree-on-tide-state-bit-for-bit--4v0sdbc3t4ywzr3g" + ], + "tokens": 15602, + "fixed_tokens": 1118, + "processed_tokens": 88986, + "tool_calls": 8, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "behaviour-008": { + "answer": [ + "con-meridian-memory-budget--99q6099e423wdtqn", + "con-skiff-memory-budget--nf7qbhsnmcn6v479", + "con-streaming-pool-budget--8czwb4fjr5p5vpy7", + "con-render-target-memory-on-meridian--q4szajxse9c1csdv" + ], + "tokens": 21635, + "fixed_tokens": 1119, + "processed_tokens": 135017, + "tool_calls": 12, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "behaviour-009": { + "answer": [ + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e", + "dec-use-host-authoritative-sync-for-co-op--17hqrmtja7pn98d6", + "dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq" + ], + "tokens": 21790, + "fixed_tokens": 1111, + "processed_tokens": 212385, + "tool_calls": 11, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "behaviour-010": { + "answer": [ + "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k" + ], + "tokens": 47847, + "fixed_tokens": 1112, + "processed_tokens": 655753, + "tool_calls": 29, + "turns": 21, + "timed_out": false, + "left_folder": false + }, + "behaviour-011": { + "answer": [ + "fnd-the-kite-physics-stats-keep-a-rolling-window-of--j6qv6qchghanqh1v" + ], + "tokens": 54897, + "fixed_tokens": 1118, + "processed_tokens": 510920, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "behaviour-012": { + "answer": false, + "tokens": 23124, + "fixed_tokens": 1115, + "processed_tokens": 179897, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "behaviour-013": { + "answer": false, + "tokens": 25611, + "fixed_tokens": 1118, + "processed_tokens": 254641, + "tool_calls": 15, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "behaviour-014": { + "answer": false, + "tokens": 35823, + "fixed_tokens": 1112, + "processed_tokens": 297644, + "tool_calls": 12, + "turns": 12, + "timed_out": false, + "left_folder": false + }, + "behaviour-015": { + "answer": [ + "dec-exclude-caustic-luminance-from-tern-s-history-cl--xhxjn26ss343v2k4", + "dec-lock-caustic-photon-jitter-to-the-tide-phase--85x10k4h11t8nx7c" + ], + "tokens": 26384, + "fixed_tokens": 1120, + "processed_tokens": 281847, + "tool_calls": 15, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "behaviour-016": { + "answer": [ + "dec-lower-the-skiff-dynamic-resolution-floor-to-504p--15esnt90gcn29enn", + "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1" + ], + "tokens": 33332, + "fixed_tokens": 1112, + "processed_tokens": 277732, + "tool_calls": 16, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "branch-001": { + "answer": "41", + "tokens": 16007, + "fixed_tokens": 1105, + "processed_tokens": 121930, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-003": { + "answer": "proposed", + "tokens": 11857, + "fixed_tokens": 1116, + "processed_tokens": 70211, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "branch-004": { + "answer": "ines/vsm-pool", + "tokens": 14706, + "fixed_tokens": 1111, + "processed_tokens": 62046, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-005": { + "answer": "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "tokens": 10169, + "fixed_tokens": 1109, + "processed_tokens": 42187, + "tool_calls": 3, + "turns": 4, + "timed_out": false, + "left_folder": false + }, + "branch-006": { + "answer": "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20", + "tokens": 15162, + "fixed_tokens": 1104, + "processed_tokens": 75274, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "branch-007": { + "answer": 48, + "tokens": 13178, + "fixed_tokens": 1114, + "processed_tokens": 59217, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "branch-008": { + "answer": "fnd-tide-storms-push-skiff-40-hz-frames-to-28-4-ms-o--28gemvtznv4zztdc", + "tokens": 19456, + "fixed_tokens": 1105, + "processed_tokens": 138331, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-009": { + "answer": "27.0 ms", + "tokens": 17652, + "fixed_tokens": 1110, + "processed_tokens": 86395, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "branch-010": { + "answer": [ + "fnd-world-core-state-with-tide-level-of-detail--x947x5kj16hcngef", + "fnd-lantern-after-the-spring-tide-page-pool-change--9zbcsqh53mgv7h4p" + ], + "tokens": 10464, + "fixed_tokens": 1109, + "processed_tokens": 42226, + "tool_calls": 3, + "turns": 4, + "timed_out": false, + "left_folder": false + }, + "branch-011": { + "answer": false, + "tokens": 25832, + "fixed_tokens": 1108, + "processed_tokens": 193821, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-012": { + "answer": "superseded", + "tokens": 17456, + "fixed_tokens": 1112, + "processed_tokens": 131054, + "tool_calls": 8, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "branch-013": { + "answer": "kenji/tide-lod", + "tokens": 16347, + "fixed_tokens": 1108, + "processed_tokens": 91837, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-014": { + "answer": "Priya Natarajan", + "tokens": 16081, + "fixed_tokens": 1114, + "processed_tokens": 103635, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-015": { + "answer": 3, + "tokens": 18352, + "fixed_tokens": 1101, + "processed_tokens": 97833, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-016": { + "answer": "proposed", + "tokens": 21542, + "fixed_tokens": 1110, + "processed_tokens": 127564, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "branch-018": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96", + "tokens": 21472, + "fixed_tokens": 1110, + "processed_tokens": 192367, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "branch-020": { + "answer": "ines/vsm-pool and tomasz/vsm-evict. ines/vsm-pool carries dec-use-virtual-shadow-maps-for-the-sun-with-a-6-144--gnxz9wk2q9xsxc96, which supersedes main's accepted dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4; tomasz/vsm-evict carries dec-use-virtual-shadow-maps-for-the-sun-with-coarser--jtcbmrm4c48ma8z8, which replaces that same Decision by way of the ines record.", + "tokens": 14343, + "fixed_tokens": 1108, + "processed_tokens": 111696, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-021": { + "answer": "dec-cap-tide-storm-particles-at-6-000-on-skiff--resze52afjxyxys1", + "tokens": 16763, + "fixed_tokens": 1107, + "processed_tokens": 117175, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "branch-022": { + "answer": "priya/skiff-cpu", + "tokens": 39084, + "fixed_tokens": 1113, + "processed_tokens": 242521, + "tool_calls": 11, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "code-003": { + "answer": [ + "rsk-eviction-scoring-asks-the-tide-forecast-about-ev--zfwghmgds54h8y46" + ], + "tokens": 17673, + "fixed_tokens": 1101, + "processed_tokens": 243922, + "tool_calls": 16, + "turns": 17, + "timed_out": false, + "left_folder": false + }, + "code-004": { + "answer": "fnd-pool-eviction-keeps-cells-the-tide-exposes-withi--tzseexbv0m0qw2jk", + "tokens": 22481, + "fixed_tokens": 1107, + "processed_tokens": 162839, + "tool_calls": 9, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "code-005": { + "answer": "false", + "tokens": 14920, + "fixed_tokens": 1109, + "processed_tokens": 86252, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "code-006": { + "answer": "fnd-tide-priority-reads-starve-player-proximity-load--by0d7b0bk39xz3rk", + "tokens": 8864, + "fixed_tokens": 1105, + "processed_tokens": 62755, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-011": { + "answer": "dec-visibility-buffer-on-all-platforms-skiff-resolve--k1ackk3c3275r5y1", + "tokens": 12958, + "fixed_tokens": 1108, + "processed_tokens": 82403, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "code-012": { + "answer": "dec-ship-four-tern-quality-presets-adding-ultra-perf--3jg1wyq3bshv839c", + "tokens": 10750, + "fixed_tokens": 1100, + "processed_tokens": 52236, + "tool_calls": 4, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "code-018": { + "answer": "unknown", + "tokens": 13237, + "fixed_tokens": 1104, + "processed_tokens": 68193, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "code-019": { + "answer": "800 nits", + "tokens": 16779, + "fixed_tokens": 1107, + "processed_tokens": 83103, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-001": { + "answer": false, + "tokens": 39784, + "fixed_tokens": 1123, + "processed_tokens": 380068, + "tool_calls": 14, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "multibranch-002": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-tooltip-renderer-leave-it-as-is-0467--y44qjef2x7j0xv7e", + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p", + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj", + "dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3", + "dec-step-tide-cells-beyond-600-m-at-a-quarter-of-the--1zccsxxw2014sh20" + ], + "tokens": 56723, + "fixed_tokens": 1113, + "processed_tokens": 370093, + "tool_calls": 15, + "turns": 11, + "timed_out": false, + "left_folder": false + }, + "multibranch-003": { + "answer": "sofia/party-invites-refresh-timer and marta/party-invites-refresh-on-open conflict: dec-refresh-the-party-invites-list-every-30-seconds--4pts3pt9681vvnv3 is accepted on Sofia's branch but rejected on Marta's, which accepts dec-refresh-the-party-invites-list-once-each-time-th--zfak0g1cx14968sd. Their code files do not overlap (net/party/stats.rs vs net/party/loader.rs).", + "tokens": 15856, + "fixed_tokens": 1110, + "processed_tokens": 123382, + "tool_calls": 10, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "multibranch-004": { + "answer": "Draft PRs #41 (ines/vsm-pool) and #44 (tomasz/vsm-evict) touch engine/lantern/shadows/vsm_pool.rs, the page pool governed by dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4. No other open branch touches code governed by the shadow Decisions on main.", + "tokens": 23532, + "fixed_tokens": 1116, + "processed_tokens": 216424, + "tool_calls": 13, + "turns": 13, + "timed_out": false, + "left_folder": false + }, + "multibranch-005": { + "answer": 4, + "tokens": 34423, + "fixed_tokens": 1114, + "processed_tokens": 209453, + "tool_calls": 14, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "multibranch-006": { + "answer": 4, + "tokens": 15542, + "fixed_tokens": 1111, + "processed_tokens": 101197, + "tool_calls": 9, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "multibranch-007": { + "answer": "engine/kestrel/world/ocean/", + "tokens": 23706, + "fixed_tokens": 1115, + "processed_tokens": 161528, + "tool_calls": 12, + "turns": 10, + "timed_out": false, + "left_folder": false + }, + "multibranch-008": { + "answer": [ + "iss-shadow-pages-run-out-at-spring-low-tide--f4fgy6b44jbkrsmh", + "iss-skiff-cert-blocker-40-hz-mode-misses-frames-in-t--8r81ajkewab5gymr" + ], + "tokens": 46608, + "fixed_tokens": 1117, + "processed_tokens": 393081, + "tool_calls": 23, + "turns": 14, + "timed_out": false, + "left_folder": false + }, + "multibranch-009": { + "answer": [ + "iss-when-should-the-party-invites-refresh-its-list--hgcxk1kbcfga0ahj", + "iss-how-should-the-tide-simulation-save-time-in-cell--72k2mwf4nb3g978p" + ], + "tokens": 51929, + "fixed_tokens": 1120, + "processed_tokens": 553052, + "tool_calls": 16, + "turns": 16, + "timed_out": false, + "left_folder": false + }, + "multibranch-010": { + "answer": 8, + "tokens": 90890, + "fixed_tokens": 1113, + "processed_tokens": 798844, + "tool_calls": 22, + "turns": 18, + "timed_out": false, + "left_folder": true + }, + "multibranch-011": { + "answer": "#50", + "tokens": 8946, + "fixed_tokens": 1114, + "processed_tokens": 60864, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "multibranch-012": { + "answer": [ + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr", + "cit-code-enqueue-in-the-ambient-chatter-queue--ksz0m3yhk8a0j0qq" + ], + "tokens": 79710, + "fixed_tokens": 1120, + "processed_tokens": 845397, + "tool_calls": 19, + "turns": 19, + "timed_out": false, + "left_folder": false + }, + "refactor-001": { + "answer": "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "tokens": 18992, + "fixed_tokens": 1119, + "processed_tokens": 96295, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-002": { + "answer": "dec-key-the-shader-cache-on-preprocessed-source-defi--3a05ff0r954vxh30", + "tokens": 12976, + "fixed_tokens": 1108, + "processed_tokens": 92073, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-003": { + "answer": "dec-stamp-each-save-with-a-schema-version-and-upgrad--nbg1h8g1bhn2ks8w", + "tokens": 19745, + "fixed_tokens": 1117, + "processed_tokens": 114883, + "tool_calls": 7, + "turns": 8, + "timed_out": false, + "left_folder": false + }, + "refactor-004": { + "answer": "dec-build-an-on-device-pipeline-cache-for-skiff--726r9mtk76qndeh7", + "tokens": 15004, + "fixed_tokens": 1108, + "processed_tokens": 79384, + "tool_calls": 6, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-005": { + "answer": "dec-feed-a-saltglass-reactive-mask-into-tern--xezjs7v3vq3r8e6k", + "tokens": 10830, + "fixed_tokens": 1111, + "processed_tokens": 84828, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-006": { + "answer": [ + "dec-add-a-second-tide-harmonic-for-spring-and-neap-t--1c3s7dtyxvj70gvw", + "dec-wrap-the-tide-phase-accumulator-at-each-40-minut--hx88m4hbg27n0xwy", + "dec-run-the-tide-simulation-in-32-32-fixed-point--c5rjfcnmhne4h05e" + ], + "tokens": 11975, + "fixed_tokens": 1111, + "processed_tokens": 75367, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-007": { + "answer": "con-the-tide-step-allocates-no-memory--bgce2y8wndwrwe0r", + "tokens": 10145, + "fixed_tokens": 1113, + "processed_tokens": 68648, + "tool_calls": 5, + "turns": 6, + "timed_out": false, + "left_folder": false + }, + "refactor-008": { + "answer": "iss-skiff-stays-at-the-lowest-resolution-for-two-sec--erke2h7xf7vepdv2", + "tokens": 13662, + "fixed_tokens": 1112, + "processed_tokens": 84648, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-009": { + "answer": [ + "cit-code-desync-checksums--r7skab53zbz8xgkd", + "cit-code-pipeline-cache-warm-up--w6b39byrdfpnb1kn", + "cit-code-shader-cache-key--wdfsmaj5fmvewbv2", + "cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr", + "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "cit-code-save-upgrade-steps--6cyqfczkyzn9b5p6", + "cit-code-tide-phase-wrap--11vxtafqbq7ar7ez", + "cit-code-engine-kestrel-kites-stats-rs--c5d4ftr0yhnev7g4" + ], + "tokens": 113115, + "fixed_tokens": 1113, + "processed_tokens": 1160488, + "tool_calls": 29, + "turns": 18, + "timed_out": false, + "left_folder": false + }, + "refactor-010": { + "answer": 7, + "tokens": 92342, + "fixed_tokens": 1113, + "processed_tokens": 1540280, + "tool_calls": 32, + "turns": 33, + "timed_out": false, + "left_folder": false + }, + "refactor-011": { + "answer": "cit-code-tern-reactive-mask--fchydw86bbgex9ax", + "tokens": 61399, + "fixed_tokens": 1109, + "processed_tokens": 698315, + "tool_calls": 23, + "turns": 21, + "timed_out": false, + "left_folder": false + }, + "refactor-012": { + "answer": [ + "dec-use-virtual-shadow-maps-for-the-sun--x8nqkn9q7eaf1ap4", + "dec-kite-physics-leave-it-as-is-0741--9xtc7xz57n0znv4v" + ], + "tokens": 31323, + "fixed_tokens": 1117, + "processed_tokens": 178339, + "tool_calls": 13, + "turns": 9, + "timed_out": false, + "left_folder": false + }, + "refactor-013": { + "answer": false, + "tokens": 7839, + "fixed_tokens": 1115, + "processed_tokens": 48555, + "tool_calls": 5, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "refactor-014": { + "answer": "The Decision dec-hash-tide-physics-and-gull-state-every-30-frames--yqf2m6hqbjhwvkzq says every 30 frames, but main hashes the state blocks every 60 frames (HASH_INTERVAL_FRAMES = 60 in net/coop/desync.rs, from commit f6ef083), so the code no longer matches the Decision.", + "tokens": 10126, + "fixed_tokens": 1108, + "processed_tokens": 79093, + "tool_calls": 8, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-015": { + "answer": "On main the function is `reclaim_pages` in `engine/lantern/shadows/page_pool.rs` (renamed from `evict_pages` in `vsm_pool.rs`, per commits df77028 and eb579ca). The Proof note is `fnd-sun-shadow-pages-are-evicted-oldest-sampled-firs--ndrr54bv4nvr1mbx`, whose code anchor `cit-code-vsm-page-pool-eviction--5txp56hwp4vrmvzr` is stale.", + "tokens": 9248, + "fixed_tokens": 1117, + "processed_tokens": 77338, + "tool_calls": 7, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-016": { + "answer": "engine/lantern/tern/reactive.rs", + "tokens": 7462, + "fixed_tokens": 1112, + "processed_tokens": 48155, + "tool_calls": 5, + "turns": 5, + "timed_out": false, + "left_folder": false + }, + "refactor-017": { + "answer": "dec-evict-cells-by-distance-and-tide-forecast--fyp1jp366pjjzxcg", + "tokens": 14714, + "fixed_tokens": 1112, + "processed_tokens": 97514, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-018": { + "answer": "false dec-give-reads-for-tide-exposed-cells-top-priority--8ftkvqhawbdf5djh", + "tokens": 16820, + "fixed_tokens": 1113, + "processed_tokens": 91538, + "tool_calls": 9, + "turns": 7, + "timed_out": false, + "left_folder": false + }, + "refactor-019": { + "answer": "unknown", + "tokens": 21440, + "fixed_tokens": 1110, + "processed_tokens": 102697, + "tool_calls": 6, + "turns": 7, + "timed_out": false, + "left_folder": false + } + } +}