Skip to content

feat(proof): report compaction drift on get, filter records by standing - #291

Draft
tonyketcham wants to merge 2 commits into
claude/agent-memory-context-potency-6sniud-resultsfrom
claude/agent-memory-context-potency-6sniud-iter2
Draft

tonyketcham wants to merge 2 commits into
claude/agent-memory-context-potency-6sniud-resultsfrom
claude/agent-memory-context-potency-6sniud-iter2

Conversation

@tonyketcham

Copy link
Copy Markdown
Collaborator

Summary of changes

This is layer 11 of a stack. It sits on #290.

I chose these changes by reading the transcripts of every wrong answer on the benchmark's tune questions: 48 failures, from Haiku with find, Haiku with findstate, and Sonnet with find. I read no holdout or private-graph transcripts. The two largest causes:

Cause Failures Seen on
Could not tell whether a compaction was stale, or what changed since it was written 16 Haiku and Sonnet
Tried to list records by a standing that links decide (--state invalidated), and got "0 records; complete" 8 Haiku and Sonnet

A compaction is the Finding that sums up an Effort. Standing is a record's state, such as superseded or invalidated.

1. Compaction drift on get and state.

  • get on a compaction now says three things: whether it is the head, whether it is stale and why, and the exact number of records that changed since it was written. It lists up to 25 of them, and the overflow line says that count is exact.
  • state gives the head compaction its own section, with the same verdict.
  • The rule lives once, in compactionReport and COMPACTION_STALE_RULE in packages/proof/src/compaction.ts. brief now uses it too, and its output is unchanged byte for byte; a test checks this.

2. Filtering records by standing.

  • --standing <values> filters by the standing that cards print, for example superseded, invalidated or rejected.
  • --rests-on <values> lists live records that derive from, cite, or give as evidence a dead record. It uses the same rule as state's dead-evidence section, through one shared helper.
  • An unknown --state or --status value is now refused. The error lists the valid values and points to the new flags. Before, it silently returned nothing.
  • Query hashes and cursors are unchanged when the new flags are not used.

What this does not change. find.ts and every strategy's task text are untouched. On the 10x graph, the find and state gates match before and after on every question. Replayed over 3,807 real reads, mean reply size rose by 0.5%. A new scripts/read-recall.mjs checks the new reads offline, and run files record read_hash.

Plain result. The new read check scores 92% on tune compaction questions but only 14% of 7 on holdout. One reviewer could not rule out that the README text was written after seeing holdout rows, so that holdout score counts as spent. A paid run of the find arm with this build will decide whether it helps agents. Its result will come in the PR above this one.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • I added doc comments to any new public exports, and inline comments to any hard-to-understand areas
  • My changes generate no new console errors locally
  • If applicable, try to include a test that fails without this PR but passes with it. New tests cover stale, not stale, and in-place standing changes in proof.test.ts; the state section in state.test.ts; and served-equals-local in serve.test.ts. 259 AVA tests pass, as do typecheck, lint and check.mjs.

Does this introduce any non-backwards compatible changes?

  • Yes: an unknown --state or --status value is now refused rather than silently matching nothing.
  • No

Does this include any user config changes?

  • Yes
    • If so, I have updated the relevant areas of documentation
  • No

Generated by Claude Code

…ding

Two read changes, chosen from the wrong answers on the benchmark's tune
questions. The two largest causes were these:

1. Agents could not tell whether a compaction was stale. `brief` worked
   it out, but `get` and `state` did not. Now `get` on a compaction
   says whether it is the head, whether it is stale and why, and the
   exact number of records that changed since it was written, listing
   up to 25. `state` gives the head compaction its own section, with
   the same verdict and rule. The rule lives once, in
   `compactionReport` and `COMPACTION_STALE_RULE` in
   `packages/proof/src/compaction.ts`, and `brief` uses it too, with
   byte-identical output.
2. Agents tried `records --state invalidated`, which silently returned
   nothing, because "invalidated" is decided by links, not stored.
   `records` now takes `--standing <values>` (superseded, invalidated,
   rejected and the stored states) and `--rests-on <values>` (live
   records that derive from, cite, or give as evidence a dead record;
   the same rule `state` uses). An unknown `--state` or `--status`
   value is refused with the valid values and a pointer to the new
   flags.

On the 10x graph the find and state gates are unchanged on every
question, so this changes no search result. `scripts/read-recall.mjs`
checks the new reads offline, and run files now record `read_hash`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Haiku ran `find` twice on this build on the 53 compaction,
invalidation and dead-evidence questions. Against the earlier build's
runs on the same questions:

- tune: 60.3 against 50.3 of 62 (10 to 0, p = 0.002);
- holdout: 42 against 42 of 44 (1 to 1), at 15% fewer tokens.

The accuracy gain stays on the tune questions the change came from.
The README and a measurement Finding say so plainly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants