Repository navigation
feat(proof): report compaction drift on get, filter records by standing - #291
Draft
tonyketcham wants to merge 2 commits into
Conversation
…ding Two read changes, chosen from the wrong answers on the benchmark's tune questions. The two largest causes were these: 1. Agents could not tell whether a compaction was stale. `brief` worked it out, but `get` and `state` did not. Now `get` on a compaction says whether it is the head, whether it is stale and why, and the exact number of records that changed since it was written, listing up to 25. `state` gives the head compaction its own section, with the same verdict and rule. The rule lives once, in `compactionReport` and `COMPACTION_STALE_RULE` in `packages/proof/src/compaction.ts`, and `brief` uses it too, with byte-identical output. 2. Agents tried `records --state invalidated`, which silently returned nothing, because "invalidated" is decided by links, not stored. `records` now takes `--standing <values>` (superseded, invalidated, rejected and the stored states) and `--rests-on <values>` (live records that derive from, cite, or give as evidence a dead record; the same rule `state` uses). An unknown `--state` or `--status` value is refused with the valid values and a pointer to the new flags. On the 10x graph the find and state gates are unchanged on every question, so this changes no search result. `scripts/read-recall.mjs` checks the new reads offline, and run files now record `read_hash`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Haiku ran `find` twice on this build on the 53 compaction, invalidation and dead-evidence questions. Against the earlier build's runs on the same questions: - tune: 60.3 against 50.3 of 62 (10 to 0, p = 0.002); - holdout: 42 against 42 of 44 (1 to 1), at 15% fewer tokens. The accuracy gain stays on the tune questions the change came from. The README and a measurement Finding say so plainly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
6 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
This is layer 11 of a stack. It sits on #290.
I chose these changes by reading the transcripts of every wrong answer on the benchmark's tune questions: 48 failures, from Haiku with
find, Haiku withfindstate, and Sonnet withfind. I read no holdout or private-graph transcripts. The two largest causes:--state invalidated), and got "0 records; complete"A compaction is the Finding that sums up an Effort. Standing is a record's state, such as superseded or invalidated.
1. Compaction drift on
getandstate.geton a compaction now says three things: whether it is the head, whether it is stale and why, and the exact number of records that changed since it was written. It lists up to 25 of them, and the overflow line says that count is exact.stategives the head compaction its own section, with the same verdict.compactionReportandCOMPACTION_STALE_RULEinpackages/proof/src/compaction.ts.briefnow uses it too, and its output is unchanged byte for byte; a test checks this.2. Filtering
recordsby standing.--standing <values>filters by the standing that cards print, for example superseded, invalidated or rejected.--rests-on <values>lists live records that derive from, cite, or give as evidence a dead record. It uses the same rule asstate's dead-evidence section, through one shared helper.--stateor--statusvalue is now refused. The error lists the valid values and points to the new flags. Before, it silently returned nothing.What this does not change.
find.tsand every strategy's task text are untouched. On the 10x graph, the find and state gates match before and after on every question. Replayed over 3,807 real reads, mean reply size rose by 0.5%. A newscripts/read-recall.mjschecks the new reads offline, and run files recordread_hash.Plain result. The new read check scores 92% on tune compaction questions but only 14% of 7 on holdout. One reviewer could not rule out that the README text was written after seeing holdout rows, so that holdout score counts as spent. A paid run of the
findarm with this build will decide whether it helps agents. Its result will come in the PR above this one.Please don't delete this checklist! Before submitting the PR, please make sure you do the following:
proof.test.ts; the state section instate.test.ts; and served-equals-local inserve.test.ts. 259 AVA tests pass, as do typecheck, lint andcheck.mjs.Does this introduce any non-backwards compatible changes?
--stateor--statusvalue is now refused rather than silently matching nothing.Does this include any user config changes?
Generated by Claude Code