Skip to content

feat(bench): keep private question sets outside the repository - #288

Draft
tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-serve-busyfrom
claude/agent-memory-context-potency-6sniud-real-harness
Draft

tonyketcham wants to merge 1 commit into
claude/agent-memory-context-potency-6sniud-serve-busyfrom
claude/agent-memory-context-potency-6sniud-real-harness

Conversation

@tonyketcham

Copy link
Copy Markdown
Collaborator

Summary of changes

This is layer 8 of a stack. It sits on #287.

The recall benchmark's own graph is fictional. To check that proof find and later reads hold up on real memory, the harness must be able to ask questions about private Proof graphs. Their questions and answers must never enter this repository's Git history. This PR adds no data. It only lets the harness read and write somewhere else.

  1. PROOF_RECALL_QUESTIONS names the folder that loadQuestions in check.mjs reads. Every script loads its questions through that one function.
  2. PROOF_RECALL_RUNS names the folder where run.mjs and find-recall.mjs write run files.
  3. With neither set, nothing changes. check.mjs still prints ok: 270 questions, 498 records, and the full Haiku comparison still prints 221.42 against 226.25.
  4. README: the new settings, and a rule to use such a set only as a held-out check. Never read its cards or outputs to choose a change.

run.mjs also stops recording a dead server's pid. Two runs started 20 seconds apart each tried to start the shared read server. The second was refused, but its pid replaced the live server's, so --stop-server would have missed the live server. Two changes fix this, matching #287:

  • The probe counts a timeout as a live server.
  • The pid is written only when the run's own child is the server that answers.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • I added doc comments to any new public exports, and inline comments to any hard-to-understand areas
  • My changes generate no new console errors locally
  • If applicable, try to include a test that fails without this PR but passes with it. These are benchmark scripts with no test suite. I checked the defaults by hand, and pointed both settings at private sets outside the repository.

Does this introduce any non-backwards compatible changes?

  • Yes
  • No

Does this include any user config changes?

  • Yes: two optional environment variables, PROOF_RECALL_QUESTIONS and PROOF_RECALL_RUNS.
    • If so, I have updated the relevant areas of documentation
  • No

Generated by Claude Code

Two settings let the recall benchmark test a private Proof graph
without its questions or answers ever entering Git:

- `PROOF_RECALL_QUESTIONS` names the folder `loadQuestions` reads.
- `PROOF_RECALL_RUNS` names the folder `run.mjs` and `find-recall.mjs`
  write run files to.

With neither set, nothing changes. The README says to use such a set
only as a held-out check.

`run.mjs` also stops recording a dead server's pid. A run that probed
a server busy loading a large graph started a second one, which was
refused, yet its pid replaced the live server's, so `--stop-server`
missed the live one. The probe now counts a timeout as a live server,
and the pid is written only when the run's own child is the server
that answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVVnhYL1Bz7mCfEKDoyLMW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants