review: audit after the node-loss, reader and S3 chunk merges

[CMGS]

Whole-repo audit after the 2026-10-17/18 merges (#339, #349, #352–#366), on main 68f34d1.

What changed

• reader_safe omits false (types.Sandbox, the claim DTO). encoding/json/v2 applies omitempty only to empty JSON values, so the bool was written as "reader_safe":false on every claim row and registry claim; omitzero, which the sibling flags use, omits it. A test pins the encoding.
• metastore: concurrent cold start. Nodes of a fresh cell starting together each saw a meta-store table missing and ran its schema; PostgreSQL serializes the CREATEs on its catalog index, so every loser failed with SQLSTATE 23505 and sandboxd exited at startup. Found by the hardware regression (two nodes started right after a table drop). A failed schema now re-checks the table, and a table that exists after the failure is the winner's complete one, since the schema runs as one implicit transaction. A new PG test opens the registry from 8 goroutines at once: it fails on the old code and passed 20 runs on the fix.
• cut(s3): referencedChunks collects one lineage's chunks into a mutex-guarded set. chunkRefs kept a per-lineage map for the sweep store/s3: sparse-aware, chunked, content-addressed checkpoint transfer #364 removed; the lineage check stays.
• cut(pool): auditFrame drops Method, Decision, Secret and Upstream, which no writer sets since egress audit records moved to egressFrame (egress and disk hardening: own-address guard, audit attribution, per-claim connection rate, disk reserve #352).
• review(pool): SetPools refuses config-owned pool fields from one name/predicate table, the shape config.ReloadDiff uses for the same fields, so a new config-owned field is one row. The egress event name becomes a constant (goconst). The test now covers egress_upstream_env.
• docs: README names the chunked S3 store, node leases and running-claim replication. The docs index points Clusters at node-loss survival. The security model no longer says a dead node's sandboxes always die with it, and notes that archives and replicas carry guest memory like checkpoints.

Prod Go net −13 (+65 −78), tests +34, docs +15 −10. No behavior change except the dropped false flag in persisted JSON (readers treat absent as false). Zero hot-path cost.

Audit scope

Read in full or as the diff since the hygiene ledger's last verdict:

• the 146 files the ledger listed as changed: all of sandboxd/pool, store/{s3,dir,peer,sparse}, archives, server, egress, outbound, engine, tenants, config, types, protocol/wire, mcp, the Python SDK and silkd;
• against /code, /code-rs and /code-py, the use-modern-go catalog, /simplify and /loc-justify.

Kept, with reasons, in the ledger:

• the putArchive/replaceArchive/putReplica/replaceReplica forwarders name putRow's two flags;
• store.isZero names its shifted-compare trick;
• wire.IsContinuation and RequestOp both parse the op, but folding them would make the relay audit tap allocate per line.

LOC (loc-justify)

	prod	test	prod comments	prod blank
Go	32,448 (effective 27,819)	42,895	1,757 (5.4%)	2,872
Rust	4,857	3,738	305
Python	2,653	2,648	10	429

Go prod includes 4,666 lines of test scaffolding (e2e drivers 3,632, archivestest 548, storetest 268, tenantstest 218).

Since 2026-09-17 (de42fd5), prod Go grew +11,610 over 134 commits, almost all features; the 29 review and cut commits sum to −484.

commit	net prod Go
#353 archived claims survive node loss	+1,289
#339 live read-only volumes	+1,071
#365 reader	+1,010
#295 tenants.Source	+1,010
#366 replication + node lease	+922

Gates

• GOWORK=off make go-lint: 10 × 0 issues. (5 modules × linux/darwin, fmt included).
• asl -forwarder=false ./... clean on darwin and linux in every module.
• go test -race -count=1 ./... green in every module.
• Rust make lint and cargo test green.
• Python ruff + pytest + mypy green.
• PG-gated suites (SANDBOXD_TEST_PG_URL on the testbed PostgreSQL) green: archives/pg (with the new concurrent-start test) and poolset/pg. In tenants/pg, TestARoleWithoutCreateServesAPreCreatedTable could not run: the testbed role may not CREATE ROLE, so its setup fails.

Hardware

Full E2E and performance regression on two bare-metal nodes (one cell, PostgreSQL, SeaweedFS, cocoon v0.7.3, rt:24.04), three builds interleaved over two waves (A → B → C → C → B → A each):

• A = b030571, main at the start of 2026-10-18;
• B = 68f34d1, main now;
• C = b4ee3fa, this PR's head.

Functional: every item passed on every build in every run.

• sandboxd-e2e.sh with the full smoke on both nodes;
• meshsmoke (cross-node template redirect);
• crossnode (checkpoint on A, claimed on B, base pulled by digest);
• lifecycle (idle, hibernate, archive on S3, wake);
• checkpoint and branch on both nodes;
• archive, then cross-node adopt and wake, with checksums;
• for B and C: a reader read with no wake, a replication round, and adoption after SIGKILL.

Skipped: volumesmoke (no volume catalog on the kit) and the egress smokes (no egress attachment).

Cold start: with the tables dropped and both nodes started at the same instant (pids 4–21 ms apart), C started cleanly 5 of 5 times. A and B lost the race twice in the waves, the loser exiting with SQLSTATE 23505 on sandboxd_archives and sandboxd_tenants.

Performance (C against B: no confirmed regression; every delta over 5 % went away in interleaved reruns):

metric	A	B	C
warm claim c=32 p50 (rerun, n=256)	3.6 ms	3.9 ms	3.9 ms
exec on a warm claim (rerun, n=60)	3.8 ms	4.55 ms	3.4 ms
hibernate / wake (rerun)	939 / 151 ms	936 / 150 ms	945 / 154 ms
checkpoint capture (rerun, n=18)	9244 ms	2086 ms	1800 ms
branch claim on A (rerun, n=18)	14555 ms	811 ms	742 ms
cross-node wake from S3	8379 ms	607 ms	601 ms
push / pull 128 MiB (rerun, n=20)	161 / 265 MiB/s	163 / 289 MiB/s	160 / 304 MiB/s
dial-per-RPC p50, every round	0.35–0.39 ms	0.34–0.40 ms	0.35–0.39 ms
reader read (rerun, n=15, 0 wakes in 30 reads)	–	253 ms	231 ms

Notes:

• Adoption of a stopped node's claim takes 12–13 s on B and C (lease expiry) against 29 ms on A (memberlist leave), by design of Running claims survive node loss: periodic snapshot replication behind a node lease #366.
• Dial-per-RPC p99 under host load moved in no consistent direction across four rounds; in the last one C was lowest.
• sandboxd RSS after three cross-node wakes and re-archives over S3 is a bounded high-water about 40 % above A at the median (A 430–629 MB, B 613–727, C 630–663), flat afterwards. Each chunked publish holds up to 12 pooled 16 MiB buffers. Without S3 activity RSS is 48–55 MB on every build.
• The standalone smoke's fork step (child 0 machine-id not distinct) is flaky on every build: A 7/16, B 3/16, C 9/16 in repeated runs. sandboxd-e2e.sh's smoke passed 12/12 in the suite runs. It predates these merges.