LakeCat

LakeCat is a Rust-native Iceberg REST catalog and QueryGraph foundation.

It keeps standard Iceberg clients on ordinary REST catalog paths while binding catalog state, governed Sail planning, TypeSec receipts, Grust projection, and QueryGraph handoff to the same accepted table transition. LakeCat is release ready as a locally verified catalog substrate; typed Iceberg v4 remains an upstream Sail task until Apache Iceberg formally adopts the specification.

Start here: architecture, living design, release checklist, current status, contributor guidance, Iceberg namespace behavior, Iceberg table lifecycle, Iceberg commit correctness, release map, and the LakeCat book.

Run Locally With Sail

In one terminal, start the Rust service with the real local integrations:

LAKECAT_BIND_ADDR=127.0.0.1:8181 \
LAKECAT_TURSO_PATH=target/local/catalog.turso \
LAKECAT_GRUST_TURSO_PATH=target/local/catalog-graph.turso \
cargo run -p lakecat-service --features sail-local,typesec-local,grust-turso-local,turso-local

In another terminal, exercise the standard catalog config route and then the full LakeCat-to-QueryGraph acceptance path:

cargo run -p lakecat-cli -- config --catalog http://127.0.0.1:8181
scripts/qglake-handoff-local.sh

The first command proves the Iceberg REST catalog is reachable. The handoff harness creates a local fixture, plans through Sail, writes Turso catalog and Grust graph state, verifies replay and OpenLineage evidence, and runs QueryGraph's locked verify/import commands. It writes disposable artifacts under target/qglake-handoff/ by default.

The current implementation exposes an Iceberg REST-compatible catalog surface under /catalog/v1 and a QueryGraph bootstrap bundle at /querygraph/v1/bootstrap. The bootstrap bundle projects live catalog tables into Croissant, CDIF, OSI, ODRL, OpenLineage, and a Grust-ready graph envelope. Standard table metadata can be registered through POST /catalog/v1[/<warehouse>]/namespaces/<namespace>/register; LakeCat reads and validates the referenced metadata object, preserves its identity and pointer, and records governed registration evidence. Registration is currently false-overwrite only. Tables can be renamed within the served warehouse through POST /catalog/v1[/<warehouse>]/tables/rename using the standard Iceberg source/destination identifier body. Rename preserves the metadata pointer, UUID, version, and creation stamp; atomically retargets commit history and table-scoped policy bindings; and emits source-to-destination graph and OpenLineage evidence. Name-bound commit idempotency records are intentionally retired so an old request cannot replay under a new REST identity. The full local release-readiness gate is green as of July 4, 2026 (see the recorded proof ref below); keep that local proof green before making release or cloud-automation claims. Use RELEASE.md for the release checklist.

Scan planning already routes through the Sail-facing engine. Point-in-time scans produce opaque Iceberg REST plan-task tokens from stable Sail metadata, and append-only incremental scans over a parent snapshot chain use Sail's manifest list reader to plan only manifests added in (start-snapshot-id, end-snapshot-id] when the table metadata and manifests are locally readable. Added delete manifests are expanded through Sail's delete-file index so file scan tasks carry Iceberg delete-file references. Non-append snapshot operations intentionally fail until overwrite/delete incremental semantics are planned end to end.

REST scan filters are validated against Sail's generated Iceberg expression models and stable table schema before planning. The accepted expression bundle is preserved in structured opaque plan-task tokens, which are bound to the planned table for stateless fetchScanTasks calls. During local manifest expansion, simple predicates are applied conservatively to Iceberg file bounds when metrics are present; missing metrics keep the file.

HTTP handlers resolve principals from x-lakecat-principal, x-lakecat-agent-did, or bearer authorization headers before calling the governance engine; absent credentials remain anonymous for local compatibility. The service binary exposes sail-local, typesec-local, grust-local, grust-turso-local, and turso-local feature gates so local real integrations can be activated without code edits. LAKECAT_WAREHOUSE selects the served warehouse, and LAKECAT_BIND_ADDR selects the listen address; defaults are local and 127.0.0.1:8181. LAKECAT_WAREHOUSE_LOCATION optionally selects a validated file:// or s3:// root for standard creates that omit an explicit table location. LAKECAT_CATALOG_IDENTITY optionally sets the stable governed-scan identity from trusted process configuration; it defaults to lakecat://<warehouse> (or a bounded hash form for an unusually long warehouse name) and must never come from request data. With the turso-local feature, LAKECAT_TURSO_PATH selects a Turso-backed TursoCatalogStore for namespaces, table records, metadata pointer history, audit/outbox rows, and idempotent commit replay; without it the binary keeps the in-memory store.

The Grust feature gates consume the published Grust 0.13.0 crates so LakeCat can use the dedicated grust-turso crate for durable catalog graph projection. Plain grust-local keeps the fast memory-backed Grust sink; grust-turso-local constructs a bootstrapped grust_turso::TursoGraphStore, using LAKECAT_GRUST_TURSO_PATH when set and an in-memory Turso graph database otherwise. Startup connect/bootstrap failures for that graph sink are reported with graph-store-path-hash and backend-error-hash evidence, not raw graph database paths or backend text. TypeSec remains on the published typesec 0.12.0 crate, and Sail integration builds from a Cargo git dependency on the lakecat branch of github.com/querygraph/sail (see LAKECAT-SAIL.md) until the required Sail APIs are published.

The local QueryGraph handoff path has a separate compatibility contract: /Users/alexy/src/querygraph (0.4.2) consumes the released lakecat-core and qglake-bundle 0.3.0 crates plus Grust 0.12.1 for lakecat-verify and lakecat-import. The handoff harness starts LakeCat with grust-turso-local plus LAKECAT_GRUST_TURSO_PATH, so the end-to-end QueryGraph acceptance path exercises Grust's Turso-backed catalog graph sink. The handoff summary carries hash-only graphProjectionProof evidence for that backend, including the configured lakecat_graph table prefix, and the Rust verifier rejects missing or drifted graph-backend proof before accepting saved artifacts. The dependency contract keeps that harness aligned with the active local Grust graph implementation while graph persistence, traversal, and Cypher-over-Turso work remain Grust-owned. LakeCat's grust-turso-local graph tests cover writing catalog events, traversing the projection, and querying/mutating it through Grust Cypher over the same Turso-backed store. They also prove Grust's matched-node mutation plan can patch a projected LakeCat table node in Turso, keeping QueryGraph readiness updates in Grust rather than turning LakeCat into a graph database.

Useful local checks:

cargo run -p lakecat-cli -- config
cargo run -p lakecat-cli -- storage-profile-list
cargo run -p lakecat-cli -- storage-profile-upsert \
  --profile local-events \
  --location-prefix file:///tmp/events \
  --provider file \
  --issuance-mode local-file-no-secret
cargo run -p lakecat-cli -- policy-list
cargo run -p lakecat-cli -- policy-upsert \
  --policy agent-read \
  --namespace default \
  --table events \
  --odrl-file ./policy.odrl.json
cargo run -p lakecat-cli --features qglake-fixture -- qglake-fixture \
  --output target/qglake/lakecat-bootstrap.json \
  --drain-output target/qglake/lineage-drain.json \
  --principal did:example:agent
cargo run -p lakecat-cli -- qglake-verify-replay \
  --bundle target/qglake/lakecat-bootstrap.json \
  --drain target/qglake/lineage-drain.json \
  --principal did:example:agent
scripts/qglake-handoff-local.sh
scripts/check-release-readiness.sh --quick
cargo run -p lakecat-cli -- bootstrap-export --output lakecat-bootstrap.json

scripts/qglake-handoff-local.sh is the local-first end-to-end handoff proof: it starts LakeCat on 127.0.0.1:18181, generates paired QGLake bootstrap and lineage-drain artifacts, verifies saved replay with LakeCat, then runs QueryGraph's lakecat-verify and lakecat-import over the same bundle while writing all generated artifacts under target/qglake-handoff/. The script owns that default target directory for each run: it clears stale Turso WAL/SHM files and generated fixture table storage, fails fast if the handoff bind address is already occupied, and stops the spawned LakeCat service tree on exit. It also writes target/qglake-handoff/handoff-summary.json, a lakecat.qglake.handoff-summary.v1 contract which records the verified LakeCat replay status from lakecat.qglake.replay-verification.v1, QueryGraph table/view counts, semantic hashes, and standards after LakeCat replay, lakecat-verify, and lakecat-import agree, structured scan/management/credential/commit replay evidence, artifact paths, raw file hashes, captured LakeCat replay output, QueryGraph verify output, QueryGraph import output, and service log path for automation. Handoff verification keeps those artifacts bundle-local and schema-closed: the handoff summary root rejects unexpected top-level fields, paths must resolve under the handoff summary directory before LakeCat hashes or parses them, and the primary artifacts manifest, nested capturedOutputs manifest, and individual bundle/lineage/import/captured-output artifact objects reject unexpected fields beside the checked path and sha256 evidence. A saved handoff summary cannot attach alternate hashes, mirror artifacts, unverified root proof claims, or unverified captured-output claims beside otherwise valid files; the compact querygraphVerification, querygraphImportVerification, and lakecatReplayVerification roots are also schema-closed before archived QueryGraph/import/replay proof is accepted. The saved captured LakeCat replay output and QueryGraph verify/import output roots are schema-closed as well, so matching captured-output hashes cannot carry unchecked replay or QueryGraph claims. View receipt-chain proof is structural in this local gate as well: the script walks verified chains and receipts, checks version-1 upsert heads, previous receipt links, supported operations, version transitions, identity binding, tombstone posture, and tombstone receipt coverage before compact handoff proof is accepted.

For release readiness, run the local release gate instead of relying on cloud CI. The full release checklist lives in RELEASE.md:

scripts/check-release-readiness.sh --release-candidate

The full gate runs shell syntax checks, the local dependency contract, workflow trigger checks, release version consistency across all LakeCat crates and book artifacts, formatting, default workspace tests, integration feature tests, the Turso store row, service feature rows, Grust/TypeSec/Sail feature rows, the explicit Rust lakecat-cli qglake_handoff verifier row, explicit all-features CLI tests, all-features workspace tests, book rebuild with EPUB metadata and PDF layout validation, QGLake handoff proof, and git diff --check. The current full proof also verifies the Grust Turso graph projection evidence, including graphProjectionProof.backend = grust-turso and graphProjectionProof.tablePrefix = lakecat_graph; the latest clean release-candidate proof was refreshed from head bd5c03ba. scripts/check-release-proof-contract.sh verifies that active release docs agree on that proof commit and that any later commits are limited to documentation and checked-in book artifacts; executable changes after the proof require a fresh full release-candidate run. In --release-candidate mode, book artifacts are built into a temporary dist directory through LAKECAT_BOOK_DIST_DIR; run docs/book/build.sh directly when intentionally refreshing tracked docs/book/dist artifacts. Use --quick for a faster script/contract smoke check while developing a narrow slice.

First-release scope is intentionally narrower than the long-term architecture: standard Iceberg REST behavior, the Rust/Turso catalog spine, CAS/idempotency, audit/outbox replay, governed Sail-planned access, redacted credentials, OpenLineage/Grust projection boundaries, and QGLake handoff proof are in scope. REST commit exact retry accepts Idempotency-Key and x-lakecat-idempotency-key, with duplicate/conflict guards before authorization, Sail validation, or side effects. Typed Iceberg v4 semantics, richer reusable graph mechanics, cloud SDK secret managers, and full QueryGraph product semantics remain Sail, Grust, TypeSec, and QueryGraph follow-on work rather than release blockers for LakeCat's catalog substrate.