Skip to content

Room archive and search: a local memory of a room's whole history - #137

Merged
ThinkOffApp merged 6 commits into
mainfrom
feat/room-archive-search
Oct 4, 2026
Merged

Room archive and search: a local memory of a room's whole history#137
ThinkOffApp merged 6 commits into
mainfrom
feat/room-archive-search

Conversation

@ThinkOffApp

@ThinkOffApp ThinkOffApp commented Oct 1, 2026

Copy link
Copy Markdown
Owner

petrus asked on 1 Oct 2026: "can you index the room so you have a memory here? add it to IAK".

What it adds

  • src/room-archive.mjs: a durable local archive of a configured GroupMind room.
    • One append-only JSONL per room, in ~/.ide-agent-kit/room-archive/<room>.jsonl (or room_archive.dir). It is deduped by id on load and tolerates a torn last line.
    • Sync pages backwards with a URL-encoded before=. Pages are newest-first, 100 each, and an unencoded +00:00 makes the server answer 500.
      • An incremental sync stops when a page overlaps the archive.
      • Backfill walks to the start of the room.
      • If the server ignores before=, the sync stops instead of looping.
    • Search needs all words (or a regex), with from / since / until filters, newest first. It reports its scope, so "0 hits" reads "not in the archive", never "never said".
  • MCP tools room_search and room_archive_sync.
  • CLI ide-agent-kit rooms search "words" and rooms archive [--backfill].
  • README rows for both.

Safety (from @codexmb's review notes)

  • Only rooms in this agent's config are archived (poller.rooms, mcp.confirmations.room). A slug the config doesn't know is refused, and DMs are never archived.
  • Tool output is labelled as untrusted evidence, never instructions or approvals.
  • Credential-looking values (IAK's SECRET_PATTERNS) are redacted in tool output, and the kinds are listed, so redaction is never silent. The archive file itself stays verbatim as evidence, and it lives outside the repo.

Measured

  • Backfill: 40,000 messages of thinkoff-development archived back to 2026-02-27 in 590 s (400 pages).
  • Incremental sync: one page.
  • Search example: rooms search "5 inch display" --from petrus found 6 matches across 40,004 messages, with ids and timestamps.

Tests

  • 9 new in test/room-archive.test.mjs:
    • backfill to the room start;
    • an incremental sync stopping after one page;
    • no loop when the server ignores before=;
    • search by words, regex, author and date, with scope;
    • persistence, dedup, edits and a torn last line;
    • slug traversal refused;
    • redaction;
    • the configured-rooms-only rule;
    • before= URL encoding.
  • npm test: 823/823 pass.

🤖 Generated with Claude Code

Petrus Pennanen @claude
feat: room archive and search, a local memory of a room's whole history
f318eb0 
petrus, 1 Oct 2026: "can you index the room so you have a memory here? add it to IAK".

- src/room-archive.mjs: one append-only JSONL per room (~/.ide-agent-kit/room-archive, or
  room_archive.dir), deduped by id on load, tolerant of a torn last line. Sync pages backwards
  with an URL-encoded before= (newest-first pages of 100), stops on overlap with the archive,
  and refuses to loop when the server ignores before=. Search: all words or a regex, from /
  since / until filters, newest first, and a scope report (0 hits = not in the archive).
- MCP tools room_search and room_archive_sync; CLI `rooms search` and `rooms archive`.
- Safety (codexmb review notes): only rooms in this config are archived, never DMs; output is
  labelled untrusted evidence; credential-looking values are redacted in tool output and the
  kinds are reported, while the archive itself stays verbatim; nothing lands in the repo.

Measured: 40,000 messages of thinkoff-development archived back to 2026-02-27 in 590 s; an
incremental sync takes one page. Tests: 9 new, 823/823 total pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

Petrus Pennanen and others added 5 commits October 1, 2026 02:39
Petrus Pennanen @claude
fix(room-archive): heal page-budget gaps, survive torn tails, redact …
d5a9acf 
…all fields, owner-only files, fail closed on refused access

All from @codexmb's review of #137 at f318eb0; each has a test that fails on that head:
- A sync whose page budget ran out before reaching the archive left m100..m199 missing forever.
  It now records where it stopped (<room>.state.json) and the next sync walks on until it meets
  the older block; gapPending reports an open hole.
- A crash fragment without a newline swallowed the next appended record; a newline now closes
  the fragment before appending.
- room_search redacted only the body; every string field of a hit is redacted now.
- New files were 0644 under the default umask; directory 0700 and files 0600, and existing
  archives are tightened on load.
- A sync that the server refused (401/403/404) used to fall back to cached hits; it now returns
  no archived text. sync:false still checks access with a one-message request. A network
  failure still returns the archive, labelled stale.
- README and the tool note now say the archive is a snapshot (later edits and deletions are not
  reconciled).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petrus Pennanen @claude
fix(room-archive): keep every open gap; test the access policy end to…
c4f2669 
… end

From @codexmb's re-review of d5a9acf:
- A second page-budget exhaustion before an older gap healed overwrote the single gapCursor, so
  the older hole (m100..m199 in his reproduction) never filled. Gaps are now a list in the state
  file (the old single-cursor format is migrated); each heals on its own. The reproduction is a
  test and fails on d5a9acf.
- The refused-access path was only inspected, not exercised. The room_search body moved into
  an exported roomSearchTool(), and tests drive it with a fake server: 401/403/404 with sync on
  and off return an error and no archived text; a network failure returns the archive labelled
  stale; an unconfigured room is refused before any request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petrus Pennanen @claude
fix(room-archive): persist gap state per page, in crash-safe order
4554d22 
From @codexmb's third review (c4f2669): the resume state was saved only when a sync finished,
so a network error after page 1 left that page archived above an unrecorded hole; the next sync
overlapped it and stopped, and m100..m599 never came back.

Gap state is now written per page, in the order that survives a crash at any point:
- opening a gap records it BEFORE the page is appended (the hole lies below that page);
- healing appends the page BEFORE the cursor moves (a crash re-fetches the page, appends are
  idempotent by id; moving first would skip the page for good).
Tests: his failure-after-page-1 reproduction (fails on c4f2669), and a crash mid-heal, which
fails on the cursor-first ordering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petrus Pennanen @claude
fix(room-archive): time-based gap healing; crash-safe at every page a…
fa4dfa4 
…nd state save

From @codexmb's fourth review (4554d22): stage 1 advanced an existing gap cursor BEFORE the
page was appended, so a crash in between skipped that page (m500..m599 in his reproduction).

Reworking the order exposed a deeper flaw: healing stopped when a page held an id already in
the archive. A crash after a page was saved but before its cursor moved made the next run
re-fetch that page, see known ids, and close the gap with the hole below still open.

- A gap is now { before, until }: `until` is the newest message of the archived block below the
  hole, fixed when the gap opens, and healing stops by time, so re-fetching a saved page can
  never close a gap early. Bare-cursor state files from earlier commits are migrated.
- A gap is opened before its first page is saved and advanced only after each further page is
  on disk; healing saves the page before moving the cursor.
- New exhaustive test: a crash injected at every add() and every saveState() call of a
  gap-producing workload (28 crash points), restart, and require all 900 messages. It fails on
  4554d22 at add #2; the two targeted reproductions are kept too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petrus Pennanen @claude
fix(room-archive): a one-page budget still heals gaps; state the cras…
e3c905b 
…h-safety scope precisely

From @codexmb's fifth review (fa4dfa4): with maxPages 1 the only page always went to the
newest-end overlap, so a pending gap never healed. With gaps pending, one page per call is now
kept for healing (budget >= 2), or newest and healing calls alternate (budget 1). Test: reload,
then maxPages 1 calls heal m100..m199 and still pick up a new message at the newest end; it
fails on fa4dfa4.

The module header now says exactly what the crash tests prove: an exception at every page save
and state save recovers fully; that is not a power-loss/fsync guarantee (writes are not
fsynced). It also restates the snapshot, offline-stale and refusal policies and that gaps from
older state files keep the id-based stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ThinkOffApp
ThinkOffApp merged commit 963785d into main Oct 4, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ThinkOffApp