feat(sandboxd): tenant managed volumes with hibernate and online growth

docs/deploy.md

@@ -112,6 +112,7 @@ gets the intersection of the two.
 | `cocoon_bin` | `cocoon` | cocoon CLI binary |
 | `restore_mode` | unset | clone and wake-restore memory mode: `copy`, `ondemand`, or `mmap`; use `mmap` for dense pools |
 | `no_direct_io` | false | use buffered writable disks for Cloud Hypervisor cold boots and clones; recommended for dense ephemeral pools to avoid direct-I/O CoW journal contention |
+| `managed_volume_hibernate` | absent | Enable explicit hibernate for one mounted managed writable volume with `hook_argv` and `hook_timeout_seconds` from 1 through 300. Requires `managed_volume_dir` and a guest-installed hook. Changes require restart. See [managed volumes](managed-volumes.md#hibernate-one-mounted-managed-volume). |
 | `sync_claims` | false | fsync `claims.json` and its directory on every commit (claim, release, renew, hibernate, wake, fork, archive, reap), about 0.3 ms each on NVMe; without it a host power loss can leave an empty journal, which fails the next start's reconcile until the file is removed, after which hibernate snapshots are reaped as orphans and archive checkpoints stay stranded. Off by default; turn it on when running claims must come back from a power loss (see `vmm_restart`) |
 | `cocoond_socket` | unset | `cocoon daemon`'s read-only API socket (by default `<cocoon run_dir>/cocoond.sock`); set, the node detects VMM exits from its event stream and polls only as a 60 s backstop. See [dead VMMs](#dead-vmms-and-host-restarts) |
 | `vmm_restart` | `cold` | what the node does with a running claim whose VMM exited outside a transition (a VMM crash, or a host restart that took it): `cold` cold-boots the claim's own VM from its disk under the same sandbox id and token (guest memory, processes, and the instance-metadata document are lost), `none` keeps the claim listed as `failed` for the caller to release or wake. Egress-lane and volume claims, a VM record gone from cocoon, and a VMM that exits more than 3 times in 10 minutes are marked `failed` under either value. See [dead VMMs](#dead-vmms-and-host-restarts) |

docs/index.md

@@ -29,6 +29,8 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC.
   online pool reconfigure, metrics, the usage journal
 - [Replace a dataset in a running sandbox](live-volumes.md) — swapping a
   read-only catalog volume on a running claim with the volume API
+- [Tenant managed persistent volumes](managed-volumes.md) — node-local tenant
+  resources, warm attachment, hibernate, online growth, and recovery limits
 - [Go SDK](sdk.md) — connecting (single node and clusters), every option,
   the full sandbox surface, error handling
 - [Python SDK](sdk-python.md) — the guest and data-plane surface for the

docs/sandboxd-api.md

@@ -19,8 +19,10 @@ All bodies are JSON. Three token kinds:
 
 Endpoints below that say "node API token" accept root or tenant unless
 marked root-only. Errors are `{"error": "message"}` with the status codes
-listed per endpoint. Request bodies are capped at 1 MiB; a larger one is
-rejected with `400 {"error": "invalid request body"}`.
+listed per endpoint. Managed volume routes also return a `code` field and
+may return a `volume` name and `holders` array. Request bodies are capped
+at 1 MiB; a larger one is rejected with 400. Managed routes use their
+endpoint-specific error bodies.
 
 ## POST /v1/claim
 
@@ -233,7 +235,11 @@ unclean `rw` release (needs recovery — one `rw` claim must replay and cleanly
 release before `ro` claims resume; see
 [deploy](deploy.md#writable-dataset-volumes)); 429 node at `max_claims`, the
 calling tenant at its own `max_claims`, or the node draining; 500
-provisioning failed.
+provisioning failed. Managed volumes use the managed volume error body: a
+volume that is growing returns 409 `volume_busy` with its public holder list;
+a missing or replaced backing returns 503 `volume_recovery_required` without
+`Retry-After`; unavailable managed storage returns 503
+`volume_storage_unavailable` with `Retry-After`.
 
 ## GET /v1/volumes
 
@@ -255,6 +261,129 @@ list; the field is emitted (as `true`) only for a writable entry this node
 declares and omitted otherwise (peer-only rows never carry it), so a read-only
 entry's response is byte-identical to v1.
 
+When `managed_volume_dir` is enabled, this list also includes caller-visible
+managed summaries and requires an explicit root or tenant token. Individual
+managed GET responses carry holder details, `capacity_generation` (including
+initial zero), and the latest growth operation. Catalog summaries retain their
+existing shape; use the individual GET for the authoritative capacity generation.
+
+## PUT /v1/volumes/{name}
+
+Creates a node-local managed volume. Auth requires an explicit root or tenant
+bearer token. A tenant token makes that tenant the owner. A root token must
+name an existing tenant in `tenant`. A tenant cannot name another owner.
+
+The strict JSON body requires `size_gib`, an integer from 1 through 1024.
+`tenant` is optional for a tenant token. The name must start with a lowercase
+letter and contain at most 19 more lowercase letters, digits, underscores, or
+hyphens. It must not start with `cocoon-`.
+
+```json
+{"size_gib":20}
+```
+
+`201` returns the `volume` envelope and an `ETag` containing the quoted,
+immutable `volume.id`. For example:
+
+```json
+{"volume":{"kind":"managed","id":"vol_1234567890abcdef1234567890abcdef",
+ "tenant":"acme","node":"node-1","state":"ready",
+ "created_at":"2026-10-20T00:00:00Z","name":"work",
+ "default_mount":"/volumes/work","size_bytes":21474836480,
+ "available":true,"nodes":1,"writable":true,"capacity_generation":0}}
+```
+
+An identical request for the same owner and size returns `200` with the same
+ID and ETag. The volume is a sparse raw disk. Creation does not format a
+filesystem. `holders` is present when sandboxes hold the volume. `growth` is
+present when a growth operation exists.
+
+Errors use `{"error":"message","code":"volume_code"}`. They include `400`
+for an invalid body, name, size, or owner; `401` for a missing or invalid token;
+`403` when a tenant selects another owner; `404` for another tenant's existing
+volume; `409` for a catalog name, different size or owner, or pending deletion;
+and `503` for unavailable storage or tenant lookup. A create interrupted after
+allocation may return `503` with code `volume_create_incomplete` and require
+operator recovery. A volume whose backing identity changed settles in state
+`recovery_required`, and PUT and claims of it return `503` with code
+`volume_recovery_required`. `volume_create_incomplete`, `volume_backing_mismatch`,
+and `volume_recovery_required` omit `Retry-After`. Transient storage failures
+include a retry hint.
+
+## GET /v1/volumes/{name}
+
+Auth requires an explicit root or tenant bearer token. Root can inspect any
+managed volume. A tenant can inspect only its own. An unknown or foreign name
+returns the same `404` body with code `volume_not_found`.
+
+`200` returns `{"volume": {...}}` and the quoted immutable ID in `ETag`.
+The volume includes `capacity_generation` and, when present, `growth` with its
+latest operation. `holders` contains public `sandbox_id`, `mode`, and `state`
+values when the volume is held. Host backing paths and process IDs are absent.
+GET observes growth progress without advancing it.
+
+Other errors include `401` for a missing or invalid token and `503` when the
+managed store or backing inspection is unavailable. A `503` includes
+`Retry-After`.
+
+## DELETE /v1/volumes/{name}
+
+Auth requires an explicit root or tenant bearer token. Send the quoted ID
+from the volume's `ETag` as `If-Match`. Root can delete any managed volume;
+a tenant can delete only its own.
+
+`204` has no body. Repeating a completed deletion with the same name and ID
+also returns `204`. A new resource with the same name has a new ID, so replaying
+the old deletion does not remove it.
+
+Errors use the managed volume error body. They include `401` for a missing or
+invalid token; `404` for an unknown or foreign name or tombstone; `409` for a
+held volume or pending growth; `412` for a malformed or mismatched `If-Match`;
+`428` for a missing `If-Match`; and `503` for unavailable storage or backing
+identity that requires operator recovery. A busy response has code
+`volume_busy` and includes the volume name and public holder list. A missing
+backing returns `volume_recovery_required`. A backing mismatch and
+`volume_recovery_required` omit `Retry-After`; transient storage failures
+include a retry hint.
+
+## POST /v1/volumes/{name}/grow
+
+Auth requires an explicit root or tenant management token. The node owning the
+managed volume handles the request. `If-Match` contains its quoted immutable
+resource ID. The strict JSON body requires `size_gib`, `expected_generation`,
+and `operation_id`. Capacity generation is independent of the identity ETag.
+
+```json
+{"size_gib":40,"expected_generation":0,"operation_id":"efbf4748-1c72-43ee-8304-b4e1a91322b5"}
+```
+
+The target is absolute GiB, from 1 through 1024, and must increase committed
+capacity for a fresh operation. The operation ID is a canonical lowercase
+nonzero UUID. `200` returns the `volume` envelope with `growth.state` completed.
+`202` returns the same envelope with pending progress. Only completion commits
+`size_bytes` and increments `capacity_generation`. The ETag remains the resource ID.
+
+`growth` reports `operation_id`, `expected_generation`, `target_bytes`, `state`,
+`runtime_outcome`, `guest_size_bytes`, and `filesystem_status`. State is pending,
+completed, or recovery_required. Runtime outcome is unobserved, outcome_unknown,
+acknowledged, or old_vmm_gone. Guest bytes remain zero until observed.
+Filesystem status is not_managed. Completion proves guest block capacity and
+does not extend the filesystem.
+
+GET detail exposes the latest operation without driving it. After a pending
+result or a lost response, repeat the original POST, including its original
+generation. An exact latest-operation replay is checked before the current
+generation. No permanent operation history is provided.
+
+Errors are `400` invalid arguments, `401` missing or invalid management token,
+`404` foreign or unknown volume, `409` conflicting request or lifecycle,
+`412` wrong identity or stale generation, `428` missing `If-Match`, and `503`
+unsupported runtime or unavailable storage. A terminal recovery result returns
+`409` with code `volume_grow_recovery_required`; GET retains its progress.
+An unsupported growth capability omits `Retry-After`.
+See [Grow a running volume](managed-volumes.md#grow-a-running-volume) for the
+supported workload and recovery limits.
+
 ## GET /v1/sandboxes/{id}/volumes
 
 Returns the persisted desired set (`volumes_want`), confirmed applied set
@@ -396,12 +525,18 @@ freeing its memory; the next agent access restores it transparently
 snapshot point and the stop coincident), except a read-only file call that a
 [reader](#get-v1sandboxesidagent) serves without waking it. Idempotent on an already-hibernated
 sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and
-snapshot) at its deadline, unless it claimed `on_expire: archive` or its pool
-archives. When to hibernate is the caller's policy — the
-node only provides the transition. 204 on success, 404 unknown id or wrong
-token, 409 on the egress lane or when volumes are attached (neither kind of
-sandbox hibernates; see [egress](egress.md)), 429 while the disk is under
-[`disk_reserve`](deploy.md).
+snapshot) at its deadline. Claims without volume holds are excepted when they
+claimed `on_expire: archive` or their pool archives. Hibernated managed-volume
+claims expire through release, which preserves the backing disk. When to hibernate
+is the caller's policy. The node only provides the transition. 204 on success, 404 unknown id or wrong
+token, 409 on the egress lane or unsupported volume claims, 429 while the disk is under
+[`disk_reserve`](deploy.md). An opt-in node hook supports one mounted writable
+managed volume on the `none` lane. See [managed volume hibernation](managed-volumes.md#hibernate-one-mounted-managed-volume)
+for hook installation, exclusive ownership, and failure recovery. Other volume
+capture operations remain blocked. A managed hibernate also returns 409 when a
+guest mount blocks the unmount (close guest references and retry), 409
+`managed_sleep_recovery_required` when the guest could not be resumed after a
+failed preparation, and 503 without `Retry-After` below Cocoon v0.7.2.
 
 ## POST /v1/sandboxes/{id}/wake
 
@@ -416,6 +551,8 @@ while stored, and waking it starts a fresh lease of the length the claim was gra
 its own disk (memory is lost) and its restart budget resets. 204 on
 success, 404 unknown id or wrong token, 409 a failed egress-lane or volume
 sandbox, or one whose VM record is gone, 500 when the cold boot itself fails.
+A managed wake also returns 409 when a guest mount conflicts with the remount
+and 503 without `Retry-After` below Cocoon v0.7.2.
 
 ## POST /v1/sandboxes/{id}/renew
 

docs/sdk-python.md

@@ -21,6 +21,9 @@ wire is chunked, the caller's copy is not). Wire fidelity is
 pinned by the shared protocol fixture corpus that the Rust guest, Go, and
 Python all round-trip in CI.
 
+For node-local `create_volume`, `get_volume`, `delete_volume` and the format-once
+workflow, see [managed volumes](managed-volumes.md).
+
 ## A complete example
 
 Claim a sandbox, push a project in, run a build, then freeze the built state
@@ -123,7 +126,9 @@ branched). `sb.template_digest` is the exact content identity when the claim
 cloned a promoted template; it is empty for other sources. `Sandbox` is a
 context manager; `sb.close()` releases it (releasing one already gone is not
 an error — double-release and reap races stay silent).
-Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Checkpoint
+Volume sandboxes cannot fork, checkpoint, or promote. Explicit hibernate supports
+one mounted managed writable volume when the node enables a workload hook.
+See [managed volume hibernation](managed-volumes.md#hibernate-one-mounted-managed-volume). Checkpoint
 branches do not accept volumes of either mode in this version.
 
 `mount=False` on `client.new` or `template.new` claims the same volumes

docs/sdk.md

@@ -9,6 +9,9 @@ module). One `Client` talks to one entry node; sandbox handles dial their
 owning node directly, so a client works unchanged against a single node or a
 cluster.
 
+For node-local `CreateVolume`, `GetVolume`, `DeleteVolume`, managed detail
+types and structured errors, see [managed volumes](managed-volumes.md).
+
 ## A complete example
 
 Claim a sandbox, push a project into it, run a build, then freeze the built
@@ -227,7 +230,9 @@ promoted template; it is empty for other sources. `Owner()` names the owning
 node, and `Token()` returns the per-sandbox bearer to persist with `ID` for a
 later `Lookup`; `Close()` releases the sandbox (releasing one already gone is
 not an error, and `Close` is bounded internally so it stays defer-friendly).
-Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Passing
+Volume sandboxes cannot fork, checkpoint, or promote. Explicit hibernate supports
+one mounted managed writable volume when the node enables a workload hook.
+See [managed volume hibernation](managed-volumes.md#hibernate-one-mounted-managed-volume). Passing
 `WithVolumes` or `WithClaimRef` to `Checkpoint.New` returns a local error:
 checkpoint branches support neither volumes nor a claim reference in this version.
 `WithMetadata`, `WithArchiveOnExpire` and `WithEnv` apply to `New`, `Template.New`, and

docs/security.md

@@ -130,6 +130,31 @@ to every other tenant's next read. A dataset that genuinely needs multiple
 writers needs an out-of-band process for who writes when, which the catalog
 ACL does not provide.
 
+Managed volumes are node-local tenant resources, separate from the operator's
+catalog. Management requires an explicit root or tenant bearer token; a
+sandbox token cannot create, inspect, grow, or delete a resource. Tenant
+listings omit other tenants' resources. Individual reads and mutations return
+the same 404 for a foreign or unknown volume. Root can manage resources for
+any tenant, but must name an existing tenant when creating one. API responses
+expose resource IDs and public holder IDs, modes, and states. They do not
+expose backing paths or process IDs.
+
+The managed directory and its ancestors must stay under trusted host control.
+The daemon records the backing file's device, inode, and size; an identity
+mismatch blocks use and requires operator recovery. Deletion requires the
+quoted immutable resource ID in `If-Match`. A replay of a completed deletion
+cannot delete a new resource created under the same name. A held volume
+cannot be deleted, and only its tenant or root can request deletion.
+
+Online growth has a separate capacity generation and an operation ID. An
+uncertain runtime result retains its growth record and blocks conflicting
+mutations until the outcome is resolved. The API can report pending progress
+or operator recovery; a 503 and `Retry-After` alone do not establish that an
+immediate retry is safe. Use the same operation ID and original generation
+when reconciling a lost growth response. See [Tenant managed persistent
+volumes](managed-volumes.md#grow-a-running-volume) for the supported workload
+and recovery rules.
+
 ## Known limitations
 
 Facts to plan around, stated so the boundary is honest:

e2e/fakeengine_test.go

@@ -47,6 +47,18 @@ func newFakeEngine(dir string) *fakeEngine {
 
 func (f *fakeEngine) CheckLiveVolumeSupport(context.Context) error { return nil }
 
+func (f *fakeEngine) RunVolumeHook(context.Context, string, engine.VolumeHookCall) error {
+	return errors.New("fakeEngine has no volume hook")
+}
+
+func (f *fakeEngine) VerifyMountedVolume(context.Context, string, string, string) error {
+	return errors.New("fakeEngine cannot verify mounted volumes")
+}
+
+func (f *fakeEngine) InspectHibernateSource(context.Context, string) (engine.HibernateSource, bool, error) {
+	return engine.HibernateSource{}, false, errors.New("fakeEngine cannot inspect hibernate sources")
+}
+
 func (f *fakeEngine) Clone(_ context.Context, _, name string, _ types.PoolKey) (types.VMRecord, error) {
 	return f.createRecord(name)
 }