Skip to content

fix(runtime): recover retired sources and preserve replay authority - #4395

Merged
ScriptedAlchemy merged 29 commits into
masterfrom
fix/4371-revive-retired-source-record
Oct 23, 2026
Merged

ScriptedAlchemy merged 29 commits into
masterfrom
fix/4371-revive-retired-source-record

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

A JSONL source restored to an older byte layout must prove its prefix against that recorded generation's own checkpoint. Matching a different older generation no longer suppresses the restored records. An accepted offer from the exact same source and scope transactionally revives a SourceRecordRetired observation, preserving isolation and newer-rewrite supersession across projection drain, restart, and rebuild.

Exact observation retries retain the first accepted retrieval anchor, projection generation, and repository provenance, including earlier writes in the same batch. Later projection clocks or Git captures neither replace immutable authority nor cause a false collision. Accepted-source revival still uses the canonical writer transaction, and in-batch duplicates remain one admitted writer operation.

The candidate also corrects shared fixtures: establish a real committed text-projection slice before overflow assertions, retry typed pre-admission readiness within existing deadlines, and verify filesystem sync barriers directly. It includes master through d889804. Projection rebuild cleanup preserves both terminal retry entries and source-revival offers.

Validation:

  • Current integration: all 476 global-DB unit tests passed with uncached remote execution and local fallback disabled (73.73s test runtime). Invocation: 376a0b33-0fb2-4a83-a94e-a6bc1e435e10. All 60 generated BUILD.bazel files passed the canonical generation check (f8253301-73e2-43c5-af8e-b632b474b951).

  • All 471 global-DB unit tests passed with the exact-retry authority repair, covering revival, collision refusal, rollback, restart/rebuild, and one writer operation per batch. The batch regression was then extended to vary derived projection metadata while requiring identical retained authority; the extended test passed in the combined run below.

  • Nine exact regressions across six Bazel targets passed on c839a87: batch duplicate, host-provenance replay, detached warmup, immutable session-store duplicate, physical daemon restart, graph-off text overflow, and three Claude/Codex restore/upgrade journeys. Each target reported a nonzero test count. Invocation: d1726fce-1f1b-4824-9fc8-8aa10ff04d7d.

  • After merging master, the reconciled detached-warmup regression and all ten managed-process tests passed on 3391962. Invocation: a13e8adb-922d-4cfc-aafd-910f1eed1e4e.

  • The analyzer/process repair previously passed all 477 MCP unit tests, 231 LSP unit tests, and 57 LSP integration tests on its source branch. These earlier results do not establish a full current-head pass.

  • Rust compilation used remote workers with local fallback disabled; local tests ran uncached in sandboxes. Pinned nightly formatting and whitespace checks passed for the reconciled files; commitlint passed for the repair commits.

Current integration head: 38cc6b0. The earlier targeted results above predate this integration; full hosted CI must pass on the updated head before merge.

Fixes #4371.

Linux explorer investigation on exact head 4e701e7a0dfc7ad2625c8ee37261606e141ce031:

  • Fleet run 39853959490 passed 168 of 169 targets. The dashboard target passed 86 tests and failed only the explorer coordination test: after 1.19s, code and session sources were complete while knowledge remained pending.
  • No failure reproduced and no speculative code change made. The exact test passed (1 passed, 86 filtered, 1.58s): invocation.
  • The full unmodified dashboard target passed all 87 tests at normal concurrency in 107.77s: invocation. A temporary tracing run also passed all 87 in 118.61s: invocation. All ran through remote Rustc/TestRunner, no local fallback, priority 10, uncached tests.
  • Measured query completion was 79.1ms, observed at 85.7ms; knowledge search took 77.3ms, including 72.1ms graph assistance. Fixture setup took 2.23s before polling began. The longest of 6,655 observed write-lock spans was 1.13ms. Temporary tracing was removed and the checkout is clean.
  • This exercises target concurrency, not contention from all 169 fleet targets; the original stall remains unexplained. At that assessment GitHub rejected rerunning the job/run because its workflow was still running. That older run is now superseded by the master integration below; fresh current-head CI is required.

Master integration at bce42ac041627f401327f44dfe96422f80ca1568:

  • Merged master e72ff8786c5566e854a45e04ddd7c799da94056d, preserving all source-revival changes. Only crates/tracedecay-private-fs/tests/framed_log.rs conflicted: retained master's Windows publication tests alongside the Linux durability-barrier tests. The resolved file matches master exactly; no source-revival production code required manual resolution.
  • All 12 source-revival regressions passed, 0 failed, 464 filtered, 6.14s (invocation), covering rebuild/restart, foreign-source isolation, repeated retirement, queue upgrade, and rollback. Global DB and private-filesystem Bazel formatting checks passed.
  • All 6 Linux filesystem tests passed, 0 failed (invocation). Preserved Windows cases require hosted Windows execution.
  • The previously failing real explorer journey passed on this merge: 1 passed, 0 failed, 86 filtered, 1.45s (invocation). This exact run does not recreate full-fleet contention or establish that the prior timing failure is resolved.
  • All test runs required remote Rustc/TestRunner, local fallback disabled, priority 10, minimal outputs, and uncached test execution. Frozen pnpm install and scoped whitespace checks passed. The normal push passed a fresh ancestry guard and the retained checkout is clean. Fresh hosted CI on this pushed head is required before merge.

ScriptedAlchemy and others added 4 commits October 21, 2026 06:00
A source rewrite retires every record the new layout does not offer.
When a later layout offers such a record again, it must become
searchable again: through Codex and Claude ingest when a rewritten
transcript is restored, and in the global database across a projection
rebuild, a restart, a rebuild that already staged the retirement, and a
retirement that landed before the drain. A newer layout that drops the
record again retires it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A record a source rewrite retired never projected again, for two
reasons.

The JSONL scanner proved a rewrite's retained prefix with checkpoints
of any earlier generation. When a file returned to an older layout,
an older checkpoint matched, the scan skipped those bytes, and the
rewrite kept the previous layout's records below the prefix. The
restored records were never offered again. Only the recorded
generation's own checkpoints now prove a retained prefix, so a restored
layout is scanned and its records are offered again.

An offered record that a rewrite had retired kept its terminal
source_record_retired disposition. Recording source presence now
removes that disposition in the same transaction and restores the
projection: it applies the effect when the drain already passed the
record, and it restages a rebuild that already staged the retirement.
A later layout that drops the record retires it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exercise both rollout discovery orders from an explicit production rewrite: first retire the thread records under the shipped v2 source identity, then restore their file before the upgraded ingest pass. This keeps the fixture historical without relying on current source discovery to recreate the old retirement bug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: bce42ac

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/4371-revive-retired-source-record branch from c1a0306 to 83ee6e4 Compare October 21, 2026 16:19
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

Final fix-forward 2/2: 56b7837 corrects the owned regression helper that failed in fleet job 119341737896. observation_source_presence.generation is a durable TEXT column and production writes generation_id().to_string(); the helper incorrectly requested i64, causing the reported TypeMismatch after all preceding collision/claimant/receipt/reset assertions passed. It now reads String and parses to u64 before asserting generation 3, preserving the repeated-presence and restart checks. Local exact Bazel command remained blocked before target loading by the cached crate-universe splice timeout (invocation 43a71ae1-e9ae-4a2b-aa58-97051e45ab2, 0 tests); no build overrides or unrelated edits. Pushed by normal fast-forward from exact remote 83ee6e4.

@ScriptedAlchemy
ScriptedAlchemy enabled auto-merge (squash) October 22, 2026 00:26
@ScriptedAlchemy
ScriptedAlchemy enabled auto-merge (squash) October 22, 2026 04:16
@ScriptedAlchemy ScriptedAlchemy changed the title fix(sessions): revive retired source records offered again fix(runtime): recover retired sources and preserve replay authority Oct 22, 2026
@ScriptedAlchemy
ScriptedAlchemy merged commit 15f2d41 into master Oct 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Projection never revives a retired source record its source offers again

1 participant