Repository navigation
backup: decode the .fsm files a running node persists - #1276
Merged
Merged
Conversation
The decoder as first shipped read only the raw store stream. A fsm-snap/<index>.fsm carries the FSM snapshot header in front and the engine's CRC32C footer behind; the store streams every MVCC version, so routing each one wrote stale values and resurrected deleted keys; the SST ingest format and the store's stored-key budget were refused; and the engine's hex file names were parsed as decimal. Add a dated addendum to the decoder design doc recording each gap and the fix the following commits implement, and point the Background section's framing description at it. Found in review on #1275. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
Each test fails on main: - TestRun_DecodesOnDiskFSMFile persists a real KV FSM snapshot through the Raft engine's file writer (FSM snapshot header, store stream, CRC32C footer) and decodes it with the CLI; main rejects the file on its EKVTHLC1 magic. - TestDecodeSnapshot_LiveStoreKeepsNewestVersion decodes a live store with an overwrite and a delete; main writes the stale "old" value and resurrects the deleted key. - TestDecodeSnapshot_SSTIngestSnapshot decodes the EKVSSTI1 bundle a store emits with SST ingest snapshots on; main rejects its magic. - TestMaxSnapshotEncodedKeySize_MatchesStore and TestReadSnapshot_AcceptsStoredKeyBudget pin the reader's key budget to the store's stored-key budget; main refuses a staged maximum-size key. - TestSnapshotIndexFromPath and the two CLI golden tests use the engine's %016x file names; main parses them as decimal (index 64 is 0000000000000040.fsm and read back as 40). Found in review on #1275. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
The reader capped encoded keys at the 1 MiB logical size plus the timestamp, while a live store streams stored keys up to MaxSnapshotStoredKeySize (1 MiB plus a 32-byte internal envelope, such as a migration-staged key) plus the timestamp; raise MaxSnapshotEncodedKeySize to match. The encoder writes user keys, so it keeps the logical cap under its own constant. Split ReadSnapshotWithHeader into readPebbleStream and emitRawEntry so the framed-file and SST-ingest readers that follow can reuse the EKVPBBL1 loop and the per-entry decode (metadata-key skip included). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
…west version ReadSnapshotFile accepts a store payload with or without the FSM snapshot header (EKVTHLC1/EKVTHLC2) a node writes in front of it, and withholds and verifies the engine's CRC32C footer. It dispatches the payload to the EKVPBBL1 stream reader or to the new EKVSSTI1 bundle reader, which checks the manifest and per-file SHA-256 before parsing each SST. latestVersionFilter keeps the newest version of each user key from a store that streams every physical MVCC version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
DecodeSnapshot now reads through ReadSnapshotFile, so it accepts the fsm-snap/<index>.fsm file a node persists and the SST ingest bundle, and filters versions through latestVersionFilter: a key's older MVCC versions count as Superseded instead of overwriting the newest one. DecodeResult carries the input's framing, including the CRC32C footer the reader verified. SnapshotIndexFromPath parses the live writer's %016x stem as hex; any other stem still parses as decimal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
…footer The CLI logs the superseded count and how the input was framed, and stamps source.fsm_crc32c in MANIFEST.json from the CRC32C footer the reader verified. Adds unit and property tests for the framing reader, the SST ingest bundle's integrity checks, the newest-version filter, and the Superseded counter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
📒 Files selected for processing: 15
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
bootjp
marked this pull request as ready for review
October 16, 2026 08:34
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
bootjp
added a commit
that referenced
this pull request
Oct 16, 2026
The decoder now reads the fsm-snap/<index>.fsm file a node persists, in either store snapshot format, as well as encoder output. CONTEXT.md and the roadmap's section 3 inventory, section 4 row, and section 6.1 README note say so. The section 4 row keeps open what remains: encrypted entries, and committed transactions whose secondary locks are still unresolved when the snapshot is taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
bootjp
pushed a commit
that referenced
this pull request
Oct 16, 2026
The A0 paragraph now says #1267 removed the first failure and #1276 closed the rest: the decoder reads an on-disk .fsm, keeps only the newest version of each key, and accepts the SST-ingest format and the store's stored-key budget. Encrypted entries are still rejected, and the roadmap's section 4 row records that limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
bootjp
added a commit
that referenced
this pull request
Oct 16, 2026
Design: docs/design/2026_09_26_proposed_positioning_and_roadmap.md ## Summary Follow-up to #1267 (merged as `95889d1f`) and #1276 (merged as `184351ae`). The docs merged with #1256 said `elastickv-snapshot-decode` could not read a live-store snapshot, and named #1267's metadata-key fix as the cure. Review on this PR showed #1267 removed only the first failure, so the decoder fix went to #1276. With #1276 merged, this change records what the two PRs fixed and what remains. It scopes the decoder to one group's DynamoDB, S3, Redis, and SQS keyspaces, and says a dump is not a complete restore source. - **`CONTEXT.md` "Backup and snapshots":** a scope statement only, pointing at the roadmap's §4 row. It says: - the decoder reads the `fsm-snap/<index>.fsm` file a node persists as well as encoder output, and dumps that one group's four adapter keyspaces; - encrypted entries are not decodable offline; - a dump can omit committed data: an unresolved secondary lock's write, a value staged by an in-flight range migration, and keyspaces outside the four adapters such as gRPC RawKV and filesystem keys; - a dump ignores storage-level expiry; - a dump reflects the snapshot's applied index, not the Raft log after it; - dumps of several groups do not share a point in time. - **Roadmap §3 inventory:** the same scope, pointing at §4. - **Roadmap §4 row, what each PR fixed:** - #1267 skips the reserved metadata keys. - #1276 strips the FSM header and verifies the CRC32C footer. - #1276 keeps the newest version of each key from MVCC history. - #1276 accepts `EKVSSTI1` and the stored-key budget, and parses hex `%016x.fsm` names. - `TestRun_DecodesOnDiskFSMFile` pins the on-disk path. - **Roadmap §4 row, open limits, each with its code path:** - encrypted entries (`ErrSnapshotEncryptedEntry`); - unresolved secondary locks (`!txn` records dropped); - staged migration values (`getAtWithStagedVisibility` versus the `!dist` drop); - the gRPC and filesystem keyspaces (`dispatcher.route` counts them as `Unknown`); - storage-level expiry (`dispatcher.handleEntry` drops `ExpireAt`); - the Raft log tail and a cross-group point in time, both non-goals of the decoder design and Phase 1 logical backup's job. - **Roadmap §6.1:** the README backup section may describe the decoder for one group's four adapter keyspaces. It must name the limits the §4 row keeps open and say a dump is not a complete restore source. - **Audit A0 status:** the decoder-rejection defect is closed for cleartext stores. #1267 and #1276 are credited, and the paragraph points at the §4 row. Besides the docs, there is one test-only hunk ported from #1236 in 5d39cc3 to unblock CI: `internal/backup/live_producer_test.go` raises `TestRunLiveBackupRenewalFailureCancelsStreamAndReleasesPin`'s TTL from 9 ms to 300 ms. It becomes a no-op once #1236 lands. a83e602 merges `main` into this branch, so the reviewed tree carries #1276's code. ## Behavior change None. The change is documentation plus one test fixture value. ## Risk None beyond a stale statement, which this change removes. The ported hunk is test-only and identical to #1236's. Every symbol and test the §4 row cites exists on `main` at `184351ae` and in this branch's tree. ## Test evidence - Docs are Markdown only. `git diff -U0 | grep '^+[+-] '` shows no stray diff markers. - Every row in the roadmap's §4 table has three cells. Names that contain a pipe (`!txn|`, `!dist|`, `!fs|`) are written without it, so the row does not split. - Each commit was fetched back and diffed against the local worktree: identical. - Checked on `main` at `184351ae`: - `ReadSnapshotFile`, `latestVersionFilter`, `DecodeSnapshot`, and `SnapshotIndexFromPath` (hex parse at `internal/backup/source.go:49`). - `TestRun_DecodesOnDiskFSMFile`, `ErrSnapshotEncryptedEntry`, and `TestReadSnapshot_RejectsEncryptedEntry`. - The `!txn|` and `!dist|` internal-drop routes, and `dispatcher.route`'s `Unknown` path. - `dispatcher.handleEntry` dropping `SnapshotEntry.ExpireAt` (`internal/backup/snapshot_reader.go:529`, `internal/backup/decode.go:297`), with the store's `PutWithTTLAt` / `ExpireAt` and `kv/fsm_migration_import.go:127` as the writers. - `getAtWithStagedVisibility` and `distribution.MigrationStagedDataKey` in `kv/shard_store.go`. - The unprefixed gRPC raw keys (`adapter/grpc.go`) and the `!fs|` prefixes (`internal/fskeys`). - The decoder design's non-goals: the Raft log tail and the cross-group point in time. - The unresolved-secondary limit, against `kv/lock_resolver_test.go`. - Ported test hunk, run as `go test -race -count=400` under CPU load: - 1 failure before the change and 0 after; - `go test -race ./internal/backup/` passes; - lint reports 0 issues. - No Jepsen runs apply. ## Review rounds - **Codex P2 (cleartext-only):** addressed in bd7d7df / b445803, then superseded by e97663f / f8968bd and 68cfb1d / 4d5c8ae. Resolved. - **CodeRabbit (`CONTEXT.md` glossary-only):** fixed in 1d8c950. Two later Minors, which asked to remove the scope statement, were answered as scope rather than mechanism; CodeRabbit withdrew both and recorded learnings. Resolved. - **Codex P1 / P2 (MVCC versions, SST-ingest, FSM header and CRC footer, stored-key budget):** recorded as open in e97663f / f8968bd, fixed in code by #1276, and recorded as closed in 68cfb1d / 4d5c8ae. Resolved. - **Codex P1 (unresolved secondary locks in the summaries):** fixed in 563b01d. Resolved. - **Codex P2 (`SnapshotIndexFromPath` still parses decimal):** the branch predated #1276; a83e602 merges `main`. Resolved. - **Codex P1 ×3 (staged migration values, unsupported keyspaces, cross-group point in time):** fixed in ba6b729. Resolved. - **Codex P2 ×2 (storage-level expiry, Raft log tail):** fixed in 0ccc0d4. §6.1 now says "the limits" rather than "every limit". Resolved. - **Reviews on 0ccc0d4:** Codex found nothing in its code and security reviews; CodeRabbit generated no actionable comments. ## CI On ba6b729, `test (ubuntu-latest)` failed in two tests this PR does not touch: - The backup renewal test: #1236's fix is ported in 5d39cc3. - `TestRedis_LuaXAddRecreatesTTLExpiredString`: `NOTLEADER` during leader loss in its 3-node cluster. It passed on the rerun. On 0ccc0d4, `test (encryption=on)` failed once in the filesystem Jepsen setup with `reg-0 (File exists)`. `RawFileSystem.Create` returns `EEXIST` for a non-exclusive `O_CREAT` that races another mount; that is existing FUSE behaviour outside this docs PR, so no fix is carried here. It passed on the rerun. 0ccc0d4 is green on every job: build, lint, three test jobs, and CodeQL. ## Self-review lenses 1. **Data loss:** no production code change. The docs list the ways a dump can be incomplete or wrong without an error, so an operator does not treat a dump as a full restore source. 2. **Concurrency / distributed failures:** no production code change. The docs now name the 2PC secondary-resolution window, the staged-visibility window, the log tail, and cross-group consistency. The ported test change only widens a fixture TTL. 3. **Performance:** no production code change. The ported test runs about 100 ms instead of a few milliseconds. 4. **Data consistency:** `CONTEXT.md`, the roadmap (§3, §4, §6.1), and the audit's A0 paragraph agree with `main` at `184351ae`. 5. **Test coverage:** the §4 row names the tests that pin the on-disk path and the encrypted-entry limit. The ported hunk makes an existing test deterministic under load. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **ドキュメント** * 保存済みスナップショットから、単一RaftグループのDynamoDB、S3、Redis、SQSのデータをダンプできることを明記しました。 * 暗号化されたエントリーはオフラインでデコードできないこと、未解決のロックや移行中データなど一部の内容はダンプに含まれないことを説明しました。 * ダンプは適用済みインデックス時点の内容であり、複数グループ間で同一時点のデータを示すものではないことを追記しました。 <!-- end of auto-generated comment: release notes by coderabbit.ai -->
bootjp
pushed a commit
that referenced
this pull request
Oct 17, 2026
TestBackupLinksNoLiveClusterCode fails on main since two merges met: 3f097df (#1276) taught internal/backup to read snapshot SST ingest bundles with github.com/cockroachdb/pebble/v2/sstable, and 5678081 (#1277) forbade every github.com/cockroachdb/pebble package there. Allow exactly the sstable package. The bundles are files in pebble's sstable format, and the reader is a file-format dependency, not the live store the boundary keeps out; the rest of pebble stays forbidden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019P5kzueGgfe3F5iSe8TMtC
bootjp
pushed a commit
that referenced
this pull request
Oct 17, 2026
TestBackupLinksNoLiveClusterCode fails on main since two merges met: 3f097df (#1276) taught internal/backup to read snapshot SST ingest bundles with github.com/cockroachdb/pebble/v2/sstable, and 5678081 (#1277) forbade every github.com/cockroachdb/pebble package there. Allow exactly the sstable package. The bundles are files in pebble's sstable format, and the reader is a file-format dependency, not the live store the boundary keeps out; the rest of pebble stays forbidden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019P5kzueGgfe3F5iSe8TMtC
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
elastickv-snapshot-decodecould only read snapshots the encoder produced. Review of #1275 found that it could not read thefsm-snap/<index>.fsmfile a running node writes, and that on a raw store stream with MVCC history it wrote a wrong dump. This PR closes the four gaps Codex reported plus one more found while reproducing them. The design doc addendum (first commit) records each one:EKVTHLC1/EKVTHLC2FSM header in front of the store payload was rejected as a bad magic (got "EKVTHLC1"), and the engine's 4-byte CRC32C footer was never checkedbackup.ReadSnapshotFileaccepts a persisted file, a raw store payload (encoder output), and a raw payload followed by the footer. With a header the footer is required and verified. The verified value is written to MANIFESTsource.fsm_crc32c, a field the format defined and the CLI never filledErrSnapshotUnsorted) on out-of-order input.DecodeCounters.Supersededcounts the hidden versionsEKVSSTI1(the payload whenELASTICKV_PEBBLE_SST_INGEST_SNAPSHOT=true) was rejectedSetentries are accepted, and SSTs are read with Pebble'ssstablereader. The entries match the legacy streamMaxSnapshotStoredKeySize(1 MiB + 32-byte envelope)%016x.fsm, butSnapshotIndexFromPathparsed the stem as decimal, so0000000000000040.fsmgave index 40Commits (made through the GitHub API, test-first per CLAUDE.md):
3aeede92docs: addendum in2026_04_29_implemented_snapshot_logical_decoder.md32198b50failing tests only (compiles on main, 8 tests red)cd94fcfareader: stored-key budget, shared entry path3f097df6framing reader, SST bundle reader, newest-version filterbfd7ff9cDecodeSnapshotwired through them, plus the hex.fsmindexc5b0889cCLI log fields andsource.fsm_crc32c, plus unit and property testsBehavior change
DecodeSnapshotand the CLI now accept persisted.fsmfiles and SST bundles. Encoder-produced streams decode exactly as before (one version per key, soSuperseded= 0).internal/backup:ReadSnapshotFile,SnapshotFraming,DecodeResult.Framing,DecodeCounters.Superseded,MaxSnapshotEncodedKeySize, and the errorsErrSnapshotFSMHeaderUnknown,ErrSnapshotFSMHeaderInvalid,ErrSnapshotFooterMismatch,ErrSnapshotTrailingBytes,ErrSnapshotSSTInvalid,ErrSnapshotUnsorted. The invariant is nowTotal = Superseded + Tombstone + DynamoDB + S3 + Redis + SQS + Internal + Unknown..fsmwhose stem is exactly 16 lowercase hex digits now yields a hexsnapshot_index. A hand-named 16-digit decimal stem would change value, but the live writer never produces one.superseded,fsm_header,store_format, andfooter_verified.Risk
Low. Only the offline decoder (
internal/backup,cmd/elastickv-snapshot-decode) changes. No server, Raft, FSM, store, or adapter code changes.internal/backupnow importsgithub.com/cockroachdb/pebble/v2/sstable, which is already a module dependency throughstore/. The footer is optional on a headerless payload: it is recognised only when the last four bytes equal the CRC32C of everything before them. On a footer-less payload where that happens by chance, the stream reads as truncated, never as different data.Test evidence
32198b50(tests only):TestDecodeSnapshot_LiveStoreKeepsNewestVersion,TestDecodeSnapshot_SSTIngestSnapshot,TestMaxSnapshotEncodedKeySize_MatchesStore,TestReadSnapshot_AcceptsStoredKeyBudget,TestSnapshotIndexFromPath,TestRun_DecodesOnDiskFSMFile,TestRun_DecodesLivePebbleSnapshot,TestRun_DecodesSimpleRedisStringSnapshot.c5b0889c:go test -race -count=1 ./internal/backup/... ./cmd/elastickv-snapshot-decode/... ./cmd/elastickv-snapshot-encode/... ./cmd/elastickv-snapshot-prepare-restore/...all ok.go build ./...andgo vetpass on every commit of the branch.golangci-lintv2.9.0 (.golangci.yaml) on./internal/backup/... ./cmd/...reports 0 issues. No new//nolintexcept twoos.ReadFiletest-path suppressions that follow the package's existing test idiom.TestRun_DecodesOnDiskFSMFilepersists a real.fsmthrough the engine's own writer (kv.NewKvFSMWithHLC+PreparePhysicalSnapshotRestore) and checkssnapshot_index64 andsource.fsm_crc32c. This pins the duplicated header/footer constants to the live writer.Self-review (five lenses)
finalize()still runs on error, so S3 scratch is cleaned up. The live write path is untouched..fsmfails the footer check rather than decoding a prefix.go test -racepasses.last_commit_ts; tombstone-newest keys are dropped. The order check fails closed, so a non-store-ordered input can never pick a stale version. SST bundle bounds and digests mirrorstore/'s own validation. Encoder output round-trips unchanged.32198b50) for every gap. Unit tests cover all framings, each through whole and one-byte readers, plus header, footer, magic, and SST corruption cases with typed errors. A rapid property test (TestLatestVersionFilter_KeepsNewestPerKey) runs over keys with shared prefixes and 0xFF bytes, alongside a deterministic interleaving case, the counter invariant, and an end-to-end CLI test on an engine-written.fsm.Known limitations (unchanged by this PR)
ErrSnapshotEncryptedEntry); the decoder links no keyring.!txn|records are dropped as internal. A committed 2PC transaction whose secondary locks were still unresolved at snapshot time is not applied to the dump.Follow-up
#1275 (positioning roadmap) records this gap as open in
CONTEXT.md, roadmap §3/§4/§6.1, and the audit. Once this lands, those lines can say the decoder reads persisted.fsmfiles.🤖 Generated with Claude Code
https://claude.ai/code/session_01S5GoEpeK4aYNkUxWxutJTz
Generated by Claude Code
Summary by CodeRabbit