diff --git a/AGENTS.md b/AGENTS.md index f834b3c..b7cea6d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,9 +2,8 @@ Read `CLAUDE.md`, `CHANGELOG.md`, and `TODO.md` before making substantive changes in this repository. `CLAUDE.md` is the full contributor guide and the -primary source of truth; the changelog provides recent historical context and -the TODO file records possible, interesting, and deliberately deferred -improvements. +primary source of truth. The changelog explains completed changes, and the TODO +file records deferred work. This file is intentionally short so agent tooling can find the critical rules quickly, then defer to `CLAUDE.md` for complete repository guidance. @@ -23,16 +22,18 @@ quickly, then defer to `CLAUDE.md` for complete repository guidance. answer that, run Spec Intake first (see `CLAUDE.md`). Then use test-driven development to satisfy each criterion. Run the relevant offline pytest suite and Kubernetes renderer/manifest dry-run checks described in - `CLAUDE.md`. Automated tests must not contact live ClickHouse or object - storage, or delete objects. + `CLAUDE.md`. Purely editorial changes are exempt from spec and TDD. + Default tests must stay offline and must not delete objects. Tests marked + `dev_cluster` require explicit opt-in and an isolated development environment. + They never run in CI. 4. **No secrets or customer data in Git.** Never commit credentials, customer configuration, target-cluster details, or rendered customer manifests. Use Kubernetes Secrets or workload identity for production credentials. 5. **Keep deployments immutable and least-privileged.** Preserve digest-pinned images, non-root/read-only container settings, and renderer validation. -6. **Keep dependencies deliberate.** Production dependencies belong in - `requirements.txt`; testing-only dependencies belong in - `requirements-dev.txt`. Do not add either casually. +6. **Keep dependencies deliberate.** Pin production dependencies in + `requirements.txt` and testing-only dependencies in `requirements-dev.txt`. + Add or update one only when the task needs it, and verify the workflow. ## Working rule diff --git a/CHANGELOG.md b/CHANGELOG.md index e810fdb..ecd5510 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,20 +2,27 @@ Notable changes to `s3gc`, newest first. -This file records **what changed and why**, with enough context that someone -picking the repository up later — human or agent — can tell a deliberate design -decision from an accident. Defects found by running the tool against real -clusters carry their evidence, because the reasoning is usually the expensive -part to reconstruct. +This file records **what changed and why**. Preserve the evidence for defects +found on real clusters so later readers can understand the decisions. Format loosely follows [Keep a Changelog](https://keepachangelog.com/). -Customer names, cluster identifiers and credentials never appear here (see -`CLAUDE.md`, rule 3); findings are described in terms of the behaviour they -expose. +Keep customer names, cluster identifiers, and credentials out of this file. +See `CLAUDE.md`, rule 4. Describe findings by the behavior they expose. ## [Unreleased] -Nothing yet. +### Documentation + +- Added a complete CLI and configuration reference, including aliases, + environment variables, defaults, precedence, and phase examples. Explained + how `--usecollected` reuses an inventory and documented a manually reviewed + shared-bucket workflow, because a single configured cluster cannot identify + references held by independent clusters. Clarified that `USETOTAL` applies + per sample rather than as a global Job limit. +- Simplified current operator and contributor guidance while preserving release + history. Aligned logging names, listing permissions, inventory recovery, + preflight limitations, and Job retry wording. Connected the packaged skill + to the manual shared-bucket workflow and clarified its table-creation examples. ## [0.8.0] - 2026-10-25 diff --git a/CLAUDE.md b/CLAUDE.md index 090591c..b782cce 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,35 +1,34 @@ # Contributor guide — s3gc -`s3gc` is a Python command-line tool and Kubernetes Job workflow for finding -and deleting orphaned objects from ClickHouse S3 disks and compatible object -storage. It is an operationally destructive tool: quality is held by offline -tests, explicit delete controls, and conservative deployment defaults. +`s3gc` finds and deletes orphaned objects from ClickHouse S3 disks and compatible +object storage. It provides a Python CLI and a Kubernetes Job workflow. Deletion +is irreversible, so preserve the offline tests and operational safeguards. ## Hard rules -1. **The deletion lifecycle is non-negotiable.** The normal production flow is +1. **Preserve the deletion lifecycle.** The normal production flow is `collect → dry-run → explicit customer approval → delete → verify`. Preserve the delete confirmation token, ClickHouse cluster and expected-replica - preflight, deletion checkpoints, and the rule that a failed delete job does - not retry automatically. A behavior change in this path needs regression - tests and matching README/deployment documentation. + preflight, deletion checkpoints, and the rule that a failed delete Job + is not rerun automatically. A behavior change in this path needs regression + tests and matching README and deployment documentation. Job reruns are + separate from HTTP retries within a running process. 2. **Spec-first TDD is mandatory.** For every new feature, bug fix, or material operational behaviour change, start with a short spec in the - commit that begins the change: one or two sentences on the problem or - intent, plus a short bullet list of acceptance criteria — what must be true - when it's done. If the request doesn't already answer these, run Spec - Intake first (see below). Then use TDD: for each acceptance criterion, - write the failing test that covers it before making it pass. Add a - regression test for every defect. Purely editorial changes are exempt from - both the spec and TDD. + commit that begins the change. State the problem in one or two sentences + and list the acceptance criteria. Run Spec Intake below if details are + missing. For each criterion, write a failing test before implementing the + change. Add a regression test for every defect. Purely editorial changes + are exempt from both the spec and TDD. 3. **Tests stay offline by default.** Install both requirements files, then - run `pytest -v` for relevant changes. Tests must use fakes, local - subprocesses, and manifest dry-runs; they must not contact live ClickHouse, - S3-compatible storage, or delete objects. The `dev_cluster` marker is - explicitly environment-dependent and never runs in CI. Do not add live - credentials or a live-delete test path to the default suite. + run the relevant offline suite with `-m "not dev_cluster"`. Default tests + use fakes, local subprocesses, and manifest dry-runs. They must not contact + live ClickHouse or object storage, or delete objects. Tests marked + `dev_cluster` require explicit opt-in and an isolated development environment. + They never run in CI. Do not add live credentials or live-delete paths to + the default suite. 4. **No secrets or customer artifacts in Git.** Do not commit S3 keys, ClickHouse passwords, customer `.env` files, rendered customer manifests, @@ -40,19 +39,19 @@ tests, explicit delete controls, and conservative deployment defaults. 5. **Kubernetes deployment stays immutable and least-privileged.** `deploy/kubernetes/render.py` must continue to reject unpinned images and invalid phase/confirmation input. Use image digests, never mutable tags. - Preserve the Job template's non-root user and read-only root filesystem; - do not embed credentials in the image or manifest. + Preserve the Job template's non-root user and read-only root filesystem. + Do not embed credentials in the image or manifest. 6. **Dependencies are deliberate and reproducible.** Runtime dependencies are - pinned in `requirements.txt`; test-only dependencies are pinned in + pinned in `requirements.txt`. Test-only dependencies are pinned in `requirements-dev.txt`. Use both files when preparing a development or CI environment. Add or update a dependency only when it is necessary for the requested capability, and test the resulting workflow. -7. **Read project history and backlog before substantive changes.** Review - `CHANGELOG.md` for recent behaviour and operational evidence, then `TODO.md` - for possible, interesting, and deliberately deferred improvements. Do not - treat a TODO item as already implemented or use the changelog as a backlog. +7. **Read the history and backlog before substantive changes.** Read + `CHANGELOG.md` for behavior changes and operational evidence, then `TODO.md` + for deferred work. A TODO item is not implemented. The changelog records + completed work, not the backlog. ## Spec Intake @@ -73,15 +72,16 @@ already answers these, skip straight to the spec. | Path | Purpose | | --- | --- | | `s3gc.py` | CLI arguments, ClickHouse inventory/anti-join, S3 collection and deletion, safety preflight, and checkpoints. | +| `docs/cli-reference.md` | Parameter details, configuration, phase examples, and manual shared-bucket inventory. | | `tests/` | Pytest regression tests using fakes and local subprocesses. | | `docker/Dockerfile` | Minimal Python 3.11 production image. | | `docker/kubernetes-entrypoint.sh` | Phase dispatcher and delete/dev-automation confirmation gate. | | `deploy/kubernetes/render.py` | Validates a non-secret environment file and renders the Job manifest. | | `deploy/kubernetes/job.yaml.tmpl` | Kubernetes Job template with security context and environment wiring. | -| `deploy/kubernetes/example.env` | Non-secret rendering example; copy it outside the repository for a real run. | +| `deploy/kubernetes/example.env` | Non-secret rendering example. Copy it outside the repository for a real run. | | `.github/workflows/container.yml` | CI test, render, manifest validation, and container publication workflow. | -| `CHANGELOG.md` | Shipped behaviour and **why**, including evidence for defects found in production use. Update it in the same change as any behaviour, safety, or deployment change. | -| `TODO.md` | Possible, interesting, and deliberately deferred engineering or operational improvements. Move completed work to the changelog when it lands. | +| `CHANGELOG.md` | Completed changes and their reasons, including production evidence. Update it with behavior, safety, and deployment changes. | +| `TODO.md` | Deferred engineering and operational work. Move completed items to the changelog. | ## Required checks @@ -109,16 +109,12 @@ percentage until coverage tooling and an enforceable threshold are introduced. - Keep the command-line and `S3GC_*` environment interfaces compatible unless the task explicitly authorizes a breaking operational change. -- Record behaviour, safety and deployment changes in `CHANGELOG.md` as part of - the same change. Write down *why*, and keep the evidence for defects found in - production — the reasoning is the expensive part to reconstruct later. Never - put customer names, cluster identifiers or credentials there. -- Record possible, interesting, or pending engineering and operational - improvements in `TODO.md`, not in the changelog. Remove or update the TODO - item when the work lands. -- Treat the renderer, entrypoint, README, and Kubernetes guide as part of the - same operator-facing contract. Update the affected documentation in the same - change as an operational behavior change. +- Record behavior, safety, and deployment changes in `CHANGELOG.md` with the + change itself. Explain why and preserve production evidence without customer + names, cluster identifiers, or credentials. +- Put deferred work in `TODO.md`. Remove or update the item when it lands. +- Keep the renderer, entrypoint, README, CLI reference, and Kubernetes guide in + agreement. Update the affected docs with each operational behavior change. - Surface out-of-scope safety defects rather than silently changing them. State the file and risk, and defer the fix unless it is necessary to keep the current task safe. diff --git a/README.md b/README.md index 84ed368..16e6f6c 100644 --- a/README.md +++ b/README.md @@ -11,41 +11,42 @@ For customer and production work, use the Kubernetes runbook: collect → dry-run → explicit human approval → delete → verify ``` -Read the [Kubernetes Job runner](deploy/kubernetes/README.md) before you run a -cleanup. It is the authoritative production procedure. +Read the [Kubernetes Job runner](deploy/kubernetes/README.md) before a production +cleanup. Use the [CLI reference](docs/cli-reference.md) for parameter details. ## Safety boundary Deleting an object is irreversible. `s3gc` preserves the following controls: - A delete Job needs the `DELETE_ORPHANS` confirmation token. -- A clustered delete checks the configured cluster and expected replica count - before it calls S3. -- When the auxiliary database uses `ENGINE = Replicated`, every phase checks - that all of its replicas are active in Keeper before it runs any DDL. With - `S3GC_REPLICA_LIVENESS_RECHECK=true` it checks again after the Keeper - session timeout (about 30 s, plus `S3GC_REPLICA_LIVENESS_MARGIN_MS`), to - catch a replica that crashed just before the run. A down replica would otherwise stall - each `CREATE TABLE` for `distributed_ddl_task_timeout` and surface only as - `IncompleteRead`. +- A production delete checks the configured cluster and expected replica count + before deleting objects and before each sample. +- Each phase checks replica liveness before DDL in a Replicated auxiliary + database. If ClickHouse cannot report liveness, the tool warns and skips that + check. See the [preflight options](docs/cli-reference.md#clickhouse-connection-and-topology) + for the optional delayed re-check. - A failed delete Job does not retry automatically. Confirmed batches remain checkpointed, so a replacement Job can resume with the same collection-table prefix. -- `USEAGE` must be at least 24 hours outside development automation. This age - window protects a part that S3 has received before ClickHouse registers it. +- Set `USEAGE_HOURS` to at least 24 in the renderer, or `S3GC_USEAGE` for a + direct run. This window protects objects uploaded before ClickHouse registers + their paths. Shorter windows are for isolated development fixtures only. - The Kubernetes renderer requires a digest-pinned image and keeps credentials outside the manifest. -For an installation-wide cleanup, collect the parent prefix that holds every -replica's objects and set the cluster name, so the comparison covers all -replicas. A single-replica prefix misses the other replicas' orphans. Only -collect a parent prefix that holds one `S3DISKNAME`'s objects — the orphan -check matches by disk name, so another disk's objects under the same prefix -would look orphaned. +For a cleanup across one installation, collect the parent prefix that holds +every replica's objects and set the cluster name. The comparison then covers +all replicas. That prefix must contain objects for only the selected disk. +Objects from another disk could otherwise appear orphaned. Use a unique collection-table prefix for each bucket and prefix. Never commit credentials, rendered manifests, target-cluster details, or run output. +If independent clusters share an overlapping bucket or prefix, review references +from every consumer before importing a candidate inventory. The +[shared-bucket workflow](docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters) +explains this manual preparation and the limits of the configured-cluster check. + ## Local setup and read-only preview Install the pinned dependencies and inspect the available options: @@ -77,13 +78,21 @@ export S3GC_COLLECTTABLEPREFIX='s3gc_example_' export S3GC_USEAGE=24 ``` -Inject passwords and keys from a secret manager or your shell. Do not save them -in a file or command history. Then run a preview: +Provide passwords and keys through a secret manager or a private runtime +environment. Keep them out of shell history and repository files. Create the +auxiliary database and grants first, as described in the +[runbook](deploy/kubernetes/README.md#3-provide-credentials-and-identity). +Then run a preview: ```bash .venv/bin/python s3gc.py --verbose --dry-run ``` +This preview collects an inventory first. To preview an existing inventory +without listing again, use `--use-collected --dry-run`. See the complete +[CLI and configuration reference](docs/cli-reference.md) for every parameter, +aliases, environment variables, defaults, and phase examples. + Use direct deletion only for controlled development work. The Kubernetes runbook separates collection, review, approval, deletion, and verification. @@ -93,19 +102,20 @@ Choose one S3 authentication mode with `S3GC_S3AUTH`: | Mode | Credentials | Use it for | | --- | --- | --- | -| `static` | `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`; optional session token | explicit credentials from a secret manager | -| `aws` | boto3 chain; optional `S3GC_S3PROFILE` | AWS SSO or a named workstation profile | +| `static` | `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`, with an optional session token | explicit credentials from a secret manager | +| `aws` | boto3 chain, with optional `S3GC_S3PROFILE` | AWS SSO or a named workstation profile | | `iam` | MinIO workload-identity provider | EKS IRSA, EC2 instance profiles, or ECS task roles | Static mode does not read `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY`. Pass those values as `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`. -`S3GC_S3PROFILE` selects `aws`; the tool rejects contradictory settings. An -`aws` preview still needs `s3:ListBucket` for the configured bucket and prefix. +`S3GC_S3PROFILE` selects `aws`. The tool rejects conflicting settings. +Collection needs `s3:ListBucket`, including a preview that collects first. +A preview with `--use-collected` reads the inventory without listing S3. GCS uses HMAC interoperability keys and usually names the object disk `gcs`. It does not support S3 batch deletion, so `s3gc` falls back to slower -per-object deletion. Set `S3GC_S3DISKNAME=gcs`; do not use a `*_cache` disk as +per-object deletion. Set `S3GC_S3DISKNAME=gcs`. Do not use a `*_cache` disk as the reference scope. ## Kubernetes deployment @@ -123,7 +133,7 @@ Start with these links: Released images are public at `ghcr.io/altinity/s3gc`. Always use a digest: ```bash -docker pull ghcr.io/altinity/s3gc@sha256: +docker pull 'ghcr.io/altinity/s3gc@sha256:' ``` CI publishes the exact `IMAGE=` value in its job summary. The renderer rejects @@ -136,17 +146,10 @@ By default, each run writes structured events to starts, collect progress, deletion checkpoints, totals, warnings, and errors. It outlives the Job and rotated pod logs. -Query one run on the same replica-pinned ClickHouse host: - -```sql -SELECT event_time, phase, event, objects, bytes, message -FROM ._log -WHERE run_id = '' -ORDER BY event_time; -``` - -Set `RUNLOG=false` only when stdout is an adequate record. A run-log failure -falls back to stdout and does not stop a cleanup. +See the [runbook](deploy/kubernetes/README.md#durable-run-history) for the query +to inspect a run. Disable durable logging only when stdout is an adequate record. +Use `RUNLOG=false` in renderer input or `S3GC_RUNLOG_FLAG=false` for a direct run. +A run-log failure falls back to stdout and does not stop the cleanup. ## AI-agent skill @@ -169,12 +172,10 @@ approve the reviewed dry-run result. `CLAUDE.md` (full guide) and `AGENTS.md` (short pointer) tell AI coding agents how to work here. For a new feature, bug fix, or behaviour change: -1. **Spec Intake** — if the request is underspecified, the agent asks what - should happen instead, how to trigger it, whether it touches the delete - lifecycle, and known edge cases. -2. **Spec** — a short problem statement and acceptance criteria, in the - commit that starts the change. -3. **TDD** — a failing test per acceptance criterion, then made to pass. +1. **Spec Intake:** ask for missing details, including expected behavior, + reproduction steps, safety impact, and edge cases. +2. **Spec:** put the problem and acceptance criteria in the starting commit. +3. **TDD:** write a failing test for each criterion, then make it pass. Editorial changes (docs, comments) skip this. See `CLAUDE.md` for the full rules. diff --git a/TODO.md b/TODO.md index e76f536..1368f22 100644 --- a/TODO.md +++ b/TODO.md @@ -1,34 +1,23 @@ # TODO -Possible, interesting, and deliberately deferred engineering or operational -improvements. Completed behaviour belongs in `CHANGELOG.md`; this file is the -forward-looking backlog. +Deferred engineering and operational work. Record completed changes in +`CHANGELOG.md` and remove or update the item here. -- [ ] In a separate documentation PR, document every CLI parameter: its purpose, - accepted values, default, aliases, exact `S3GC_*` environment variable name, - and interactions or constraints. Prefer existing kebab-case aliases in - examples, explain CLI/environment/configuration precedence, and include - examples for collect, dry-run, delete and verify. Keep existing option names - and aliases; a naming overhaul is unnecessary. -- [ ] Record collect progress so a crashed `--collectonly` run can resume. - `--collectafter` already starts listing after a given key, but nothing - records the last key listed, so after a crash the operator cannot tell what - to pass and restarts from the beginning. Write the last listed key - durably (for example in the run log) at each progress step. +- [ ] Record the last listed key so an interrupted collect can resume. + `--collect-after` can start after a key, but collect does not save that key. + Write it durably at each progress step, for example in the run log. - [ ] Add an opt-in `dev_cluster` GCS end-to-end test. It must require explicit environment configuration, create an isolated tiered-policy fixture, run `collect → dry-run → delete → verify`, clean up ClickHouse and object-storage fixtures, and remain excluded from CI. -- [ ] Require the `Container / test` GitHub Actions check before pull-request - merges in the repository branch-protection settings. `master` has no branch - protection or ruleset today; this needs a repository admin. +- [ ] Have a repository admin require the `Container / test` GitHub Actions + check before pull-request merges. This item was recorded when `master` had + no branch protection or ruleset. Verify the current settings before changing them. - [ ] Surface a DDL `TIMEOUT_EXCEEDED` (159) instead of `IncompleteRead`. The server sends `200 OK` before a distributed DDL times out, so the HTTP client - hides the cause. The Replicated-database preflight covers a replica that is - already down; a replica that dies after the preflight still surfaces as + hides the cause. The Replicated-database preflight catches a replica already + reported down. A replica that dies after the check can still cause `IncompleteRead` after `distributed_ddl_task_timeout`. -- [ ] Consider a `TTL` on the run-log table. Volume is a few hundred rows per - run so it is not urgent, but it grows without bound across many cleanups. - The run log is the durable record of what was deleted, so a TTL works - against its purpose: keep it off by default, or make it long (a year or - more). +- [ ] Consider a run-log `TTL`. Each run adds a few hundred rows, and the table + grows across cleanups. It is the durable deletion record, so keep any TTL + disabled by default or retain at least a year of history. diff --git a/deploy/kubernetes/README.md b/deploy/kubernetes/README.md index 80c8dbc..3bbac70 100644 --- a/deploy/kubernetes/README.md +++ b/deploy/kubernetes/README.md @@ -13,8 +13,8 @@ python3.11 -m venv .venv .venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt ``` -`render.py` uses only the Python standard library. The runbook still uses -`.venv/bin/python` so every command runs with the supported Python version. +Use `.venv/bin/python` for these commands. The renderer itself needs only the +Python standard library. ## Runbook @@ -45,35 +45,43 @@ contains no credentials. ### 2. Confirm the cleanup scope -Set the values in `s3gc.env`, then check these five items before rendering: - -1. Set `CHHOST` to one per-replica ClickHouse Service and use the same host for every phase. The inventory table is node-local; a load-balanced Service can send a later phase to a replica without that table. -2. Set the exact bucket, prefix, and underlying object-disk name. `S3PATH` may be empty. GCS commonly uses `gcs`; never use a `*_cache` disk. -3. Set the actual `CLUSTERNAME` and `EXPECTED_REPLICAS` for clustered cleanup. Delete fails closed if this preflight does not match. -4. Set `COLLECTDATABASE` to the auxiliary database, and choose a unique `COLLECTTABLEPREFIX` such as `s3gc__`. Keep the prefix after a partial delete so the replacement Job can use the deletion checkpoints. Without `COLLECTDATABASE`, the auxiliary table lands in the ClickHouse user's current database, which can differ from `default` and cause a grant failure; `COLLECTTABLEPREFIX` still accepts an embedded `db.prefix_` form for compatibility, but a value there that disagrees with `COLLECTDATABASE` is rejected at startup. -5. Set `USEAGE_HOURS` to 24 or more. Raise it for slow merges or long mutations; production phases cannot lower it. - -For an installation-wide cleanup, set `S3PATH` to the parent prefix that -contains every replica's objects, such as `clickhouse//`, not a -single `//` prefix — but only when that parent prefix -holds objects for the one `S3DISKNAME` being cleaned up. The orphan check -matches ClickHouse's `system.remote_data_paths` by `disk_name`, so an object -that belongs to a different disk stored under the same parent prefix would -never match and would look orphaned. When disks share a parent prefix, collect -a disk-scoped prefix instead, or give each disk its own S3 path layout. -Collect the chosen prefix once into one inventory table on the pinned host. -With `CLUSTERNAME` set, dry-run and delete compare -the inventory with `system.remote_data_paths` on all replicas, so an object -that any replica still references is never a candidate. A single-replica -prefix misses the other replicas' orphans. Collect per-replica prefixes only -when you intend to, and give each one its own `COLLECTTABLEPREFIX`. +Set these values in `s3gc.env` before rendering: + +| Setting | What to check | +| --- | --- | +| `CHHOST` | Use one per-replica ClickHouse Service for every phase. The default inventory is node-local. A load-balanced Service can reach a replica without it. | +| `S3BUCKET`, `S3PATH`, `S3DISKNAME` | Match the exact bucket, object prefix, and underlying disk. An empty `S3PATH` selects the bucket root. Use `gcs` for a disk named `gcs`, never a `*_cache` wrapper. | +| `CLUSTERNAME`, `EXPECTED_REPLICAS` | Match the actual cluster and total replica count. Delete stops if the preflight result differs. | +| `COLLECTDATABASE`, `COLLECTTABLEPREFIX` | Use the auxiliary database and a unique prefix such as `s3gc__`. Keep both unchanged during recovery. | +| `USEAGE_HOURS` | Use 24 or more. Raise it for slow merges or long mutations. Production phases reject a lower value. | + +Without `COLLECTDATABASE`, the inventory uses the ClickHouse user's current +database. That database may differ from the one you granted access to. +The older `db.prefix_` form still works in `COLLECTTABLEPREFIX`, but its database +must agree with `COLLECTDATABASE` when both are set. + +For a cleanup across one installation, collect the parent prefix that holds all +replicas' objects, such as `clickhouse//`. Set `CLUSTERNAME` so +the comparison reads references from every configured replica. Collecting only +`//` misses the other replicas' orphans. + +The selected prefix must contain objects for only the chosen disk. References +are matched by disk name, so another disk's objects could look orphaned. Use a +disk-specific prefix when possible. For intentional per-replica runs, give each +prefix its own inventory table. + +If independent clusters or other consumers share the selected scope, follow +the [manual inventory workflow](../../docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters). +Check every consumer before importing candidates. The configured cluster check +cannot see references in other clusters. After importing the reviewed inventory, +start with the dry-run phase and skip collection. Use an immutable multi-architecture image digest in `IMAGE`. CI prints the exact value after publishing. Do not use an image tag. Collection has no resume checkpoint. For a large bucket, collect separate -prefix shards and rerun only a failed shard. Re-listing a shard is safe because -the auxiliary table replaces rows by object path. +prefix shards and rerun only a failed shard. The inventory replaces rows with +the same object path. Do this before deletion, not while recovering a delete. ### 3. Provide credentials and identity @@ -85,12 +93,11 @@ Create or reuse the Kubernetes Secret named by `CREDENTIALS_SECRET`: | `iam` | `S3GC_CHUSER` and `S3GC_CHPASS` | | `aws` | `S3GC_CHUSER`, `S3GC_CHPASS`, plus a credential source available to boto3 | -Set `SERVICE_ACCOUNT` to an existing namespace-local ServiceAccount that your -cluster configures for the chosen workload-identity mode. The Job keeps -`automountServiceAccountToken: false` because it does not call the Kubernetes -API. For `iam`, the platform must inject or otherwise provide the workload -identity credentials. Existing ClickHouse or ClickHouse-backup ServiceAccounts -are suitable when they meet that cluster policy. +Set `SERVICE_ACCOUNT` to an existing ServiceAccount in the namespace. For `iam`, +the platform must provide workload-identity credentials for that account. +The Job keeps `automountServiceAccountToken: false` because it does not call the +Kubernetes API. An existing ClickHouse or ClickHouse-backup ServiceAccount is +suitable if it meets the cluster's identity policy. Create a dedicated auxiliary database and ClickHouse user before the first run. Replace ``, ``, and the password placeholder. @@ -105,7 +112,9 @@ ON CLUSTER IDENTIFIED WITH sha256_password BY ''; GRANT ON CLUSTER - SELECT ON system.* TO s3gc; + SELECT ON system.remote_data_paths, + SELECT ON system.one, + SELECT ON system.clusters TO s3gc; GRANT ON CLUSTER SELECT, INSERT, CREATE TABLE ON .* TO s3gc; @@ -114,13 +123,30 @@ GRANT ON CLUSTER REMOTE ON *.* TO s3gc; ``` +The system-table grants are by name, so the user cannot read `system.query_log` +or other tables that may hold restricted data. `system.remote_data_paths` is the +reference s3gc compares the inventory with, `system.one` backs the replica-count +preflight of the delete phase, and `system.clusters` backs the replica-liveness +check for a Replicated auxiliary database. s3gc also reads `system.databases` +and `system.tables`; on the version this was checked on (26.8) any user can +read those two, limited to the objects it has access to, so they need no grant. + The database grant covers the auxiliary inventory and durable run-log tables. `REMOTE` permits the `clusterAllReplicas` reads used by clustered dry-run and -delete preflight. When `` uses `ENGINE = Replicated`, every -phase, including collect, first checks that all of that database's replicas are -active in Keeper (`S3GC_REPLICA_LIVENESS_RECHECK=true` adds a re-check after one -Keeper session timeout, about 30 s); this reads only system tables and needs no extra grant. Do not grant ClickHouse's `S3` source privilege: s3gc uses -its own S3 client and the Secret or workload identity instead. +delete preflight. These grants support the normal phases, which preserve the +inventory after delete. Optional DROP or TRUNCATE operations need their own +grants. + +For a Replicated auxiliary database, every phase checks replica liveness before +DDL. If ClickHouse cannot report liveness or returns no replicas, s3gc warns and +skips that check. An optional delayed re-check uses the Keeper session timeout +reported by the cluster. See the [CLI preflight options](../../docs/cli-reference.md#clickhouse-connection-and-topology). +The optional delayed re-check also reads `system.zookeeper_connection`. Grant +`SELECT ON system.zookeeper_connection` only if you pass +`--replica-liveness-recheck`; without the grant that flag warns and skips the +re-check. + +s3gc uses its own S3 client. It does not need ClickHouse's `S3` source privilege. For `S3AUTH=static`, create the Secret from a private, shell-style credential file. Keep this file outside the repository, restrict it to its owner, and @@ -162,7 +188,7 @@ if [ -n "${S3GC_S3SESSIONTOKEN:-}" ]; then literals+=(--from-literal=S3GC_S3SESSIONTOKEN="$S3GC_S3SESSIONTOKEN") fi -kubectl -n create secret generic \ +kubectl -n '' create secret generic '' \ "${literals[@]}" unset S3GC_CHUSER S3GC_CHPASS S3GC_S3ACCESSKEY S3GC_S3SECRETKEY S3GC_S3SESSIONTOKEN @@ -174,15 +200,16 @@ object keys. Keep delete permission separate until an approved delete phase. ### 4. Run each phase -For each phase, edit only the values shown below, render the Job, server-side -validate it, apply it, and follow its logs: +For each phase, change the values in the table below. Render the Job, validate +it against the API server, apply it, and follow its logs. Replace the +placeholders before running these commands: ```bash .venv/bin/python deploy/kubernetes/render.py "$S3GC_RUN_DIR/s3gc.env" \ > "$S3GC_RUN_DIR/.yaml" kubectl apply --dry-run=server -f "$S3GC_RUN_DIR/.yaml" kubectl apply -f "$S3GC_RUN_DIR/.yaml" -kubectl -n logs -f job/ +kubectl -n '' logs -f 'job/' ``` | Phase | Change in `s3gc.env` | Required outcome | @@ -193,17 +220,34 @@ kubectl -n logs -f job/ | Verify | Set `PHASE=dry-run`, a new `JOB_NAME`, and clear `DELETE_CONFIRMATION`. | Report zero candidates. | Do not run delete from an unreviewed dry-run. Before the first full delete for -a new image or cluster, set `USETOTAL` to a few thousand objects for one -bounded delete. Clear it before the full run. +a new image or cluster, set `USETOTAL` to a few thousand objects per sample +for a bounded trial delete. Clear it before the full run and verification. +The use-phase query applies this limit separately to each sample, so it is not +a global cap on the Job. + +The entrypoint selects the CLI flags for each phase: + +| Phase | CLI flags | +| --- | --- | +| Collect | `--collectonly --keepdata` | +| Dry-run and verify | `--usecollected --dry-run` | +| Delete | `--usecollected --keepdata --non-interactive`, after the token and cluster-input gates | + +See the [CLI reference](../../docs/cli-reference.md) for parameter details. ### 5. Recover or verify If a delete Job fails, create a replacement delete Job with a new `JOB_NAME` -and the same `COLLECTTABLEPREFIX`. Do not re-collect unless the scope changed. -`backoffLimit: 0` prevents automatic retries. +and the same host, scope, and inventory settings. Inspect the failure and confirm +that the approval still applies first. Do not re-collect or drop the inventory +during recovery. A changed scope needs a new inventory, review, and approval. +`backoffLimit: 0` prevents automatic Job reruns. Individual S3 requests can still +retry within the running process. After a successful delete, run the verification dry-run. Do not declare the -cleanup complete until it reports zero candidates. +cleanup complete until it reports zero candidates with `USETOTAL` cleared. +That result covers the reviewed inventory and age window. It does not check +new objects absent from the inventory or prove the bucket is empty. ## Configuration reference @@ -211,6 +255,9 @@ cleanup complete until it reports zero candidates. values, unpinned images, invalid phases, invalid booleans, and unsafe age windows. Start with [example.env](example.env). +Renderer keys differ from CLI environment names. For example, `USEAGE_HOURS` +becomes `S3GC_USEAGE`, and `RUNLOG` becomes `S3GC_RUNLOG_FLAG`. + | Group | Required values | Notes | | --- | --- | --- | | Job | `JOB_NAME`, `NAMESPACE`, `IMAGE`, `IMAGE_PULL_SECRET`, `SERVICE_ACCOUNT`, `CREDENTIALS_SECRET` | `JOB_NAME` is a DNS label and becomes the durable run ID. Leave `IMAGE_PULL_SECRET` empty for the public image. | @@ -219,13 +266,13 @@ windows. Start with [example.env](example.env). | Limits | `SAMPLES`, `DELETE_BATCH_SIZE`, `USEAGE_HOURS`, `ACTIVE_DEADLINE_SECONDS`, `TTL_SECONDS_AFTER_FINISHED`, `MEMORY_REQUEST`, `MEMORY_LIMIT` | Keep `SAMPLES` unchanged after collect for partition pruning. | | Behavior | `PHASE`, `DELETE_CONFIRMATION`, `ORDER_BY_OBJPATH`, `VERBOSE` | The renderer requires the confirmation token for delete and development automation. | -`USETOTAL` is optional and limits one use phase. `RUNLOG` defaults to `true`; -set it to `false` only when stdout is an adequate operational record. +`USETOTAL` is optional and limits each sample in a use phase. `RUNLOG` defaults +to `true`. Set it to `false` only when stdout is an adequate operational record. ## Durable run history -Pod logs are for monitoring, not durable evidence. The kubelet rotates them, -and the Job TTL removes the pod. By default, `s3gc` writes run events to +The kubelet rotates pod logs, and the Job TTL removes the pod. By default, +`s3gc` also writes durable run events to `_log` in ClickHouse. Use the same replica-pinned host to inspect a run: @@ -247,12 +294,14 @@ the cleanup. with a fresh inventory and requires `DELETE_CONFIRMATION=DELETE_ORPHANS`, `CLUSTERNAME`, and `EXPECTED_REPLICAS`. -Use it only for isolated development fixtures. It bypasses the manual review -and approval gate, and it alone may use an age window below 24 hours. +Use it only for isolated development fixtures. It runs all three phases without +a pause for manual review. The renderer allows an age window below 24 hours +only for this phase, and the entrypoint passes the development override. +Never use it for customer or production data. ## Preserved safeguards - The manifest runs as a non-root user with a read-only root filesystem and no Linux capabilities. - Credentials come from `CREDENTIALS_SECRET` or workload identity, never the image or environment file. - The renderer requires a digest-pinned image and validates delete confirmation before the Job reaches the cluster. -- Delete requires the cluster/replica preflight, records checkpoints, and never retries automatically. +- A delete Job requires the cluster/replica preflight and records checkpoints. A failed Job is not rerun automatically. diff --git a/deploy/kubernetes/example.env b/deploy/kubernetes/example.env index 6637aa8..2b91694 100644 --- a/deploy/kubernetes/example.env +++ b/deploy/kubernetes/example.env @@ -5,10 +5,10 @@ # Job identity and lifecycle JOB_NAME=s3gc-example-dry-run NAMESPACE=clickhouse -# Always digest-pinned; CI prints the exact line in its job summary. +# Always pin the image by digest. CI prints the exact line in its job summary. IMAGE=ghcr.io/altinity/s3gc@sha256:0000000000000000000000000000000000000000000000000000000000000000 -# The published image is PUBLIC, so leave this empty: the renderer then omits -# the imagePullSecrets block entirely. Set it only for a private mirror. +# Leave empty for the public image. The renderer omits imagePullSecrets. +# Set a secret name only for a private mirror. IMAGE_PULL_SECRET= PHASE=dry-run # Development automation only: set PHASE=dev-automation and @@ -22,9 +22,8 @@ CHHOST=clickhouse.example.svc.cluster.local CHPORT=8123 CLUSTERNAME=example-cluster EXPECTED_REPLICAS=2 -# COLLECTDATABASE names the dedicated auxiliary database. Without it, the -# auxiliary table lands in the ClickHouse user's current database, which may -# not be the one you granted. +# Use the dedicated auxiliary database. Without it, the inventory uses the +# ClickHouse user's current database, which may not have the required grants. COLLECTDATABASE=s3gc COLLECTTABLEPREFIX=s3gc_example_ @@ -36,8 +35,8 @@ S3PATH=clickhouse-data/ S3REGION=us-east-1 S3SECURE_FLAG=true S3DISKNAME=s3 -# static = explicit keys in the Secret; aws = boto3 chain / SSO profile; iam = IRSA/IMDS. -# Prefer iam in Kubernetes: it refreshes credentials and needs no boto3. +# static uses keys from the Secret. aws uses boto3 credentials or an SSO profile. +# iam uses workload identity. Prefer it for in-cluster AWS runs. S3AUTH=iam # Only for S3AUTH=aws. The session token, if any, belongs in the Secret. S3PROFILE= @@ -45,17 +44,13 @@ S3PROFILE= # Processing limits. Keep SAMPLES unchanged after collection. SAMPLES=4 DELETE_BATCH_SIZE=1000 -# Minimum 24, and the renderer refuses less for every phase except -# dev-automation. This window is the only protection against deleting a part -# between its upload to S3 and its registration in system.remote_data_paths. -# Raising it is fine and sometimes wise; lowering it is not. +# Use at least 24 hours for production. This window protects objects uploaded +# before ClickHouse registers their paths. Only dev-automation may use less. USEAGE_HOURS=24 -# Optional cap on how many collected objects a use phase processes. Leave empty -# for a full run. Set it to bound the first delete against a new image or a new -# cluster: a few thousand objects exercise collect -> delete -> tombstone -# end to end in one sample instead of committing to a multi-hour run. +# Optional limit per sample. Use it for a trial delete with a new image or +# cluster. Leave empty for a full run and verification. USETOTAL= -# Leave false for Kubernetes Jobs; global ordering is unnecessary for deletion. +# Leave false for Kubernetes Jobs. Sorting candidates can be expensive. ORDER_BY_OBJPATH=false # Pod limits and observability. @@ -64,9 +59,7 @@ TTL_SECONDS_AFTER_FINISHED=604800 MEMORY_REQUEST=1Gi MEMORY_LIMIT=4Gi VERBOSE=true -# Record durable run history in _log in ClickHouse. -# Pod logs are rotated by the kubelet and deleted with the Job, so this table is -# the only evidence of what a cleanup did once TTL_SECONDS_AFTER_FINISHED -# expires. Needs the same CREATE TABLE grant as the auxiliary table; a missing -# grant degrades to stdout only instead of failing the run. +# Write durable run history to _log in ClickHouse. +# It survives pod-log rotation and Job deletion. It needs the same CREATE TABLE +# grant as the inventory. A logging failure warns and falls back to stdout. RUNLOG=true diff --git a/docs/cli-reference.md b/docs/cli-reference.md new file mode 100644 index 0000000..7c665d1 --- /dev/null +++ b/docs/cli-reference.md @@ -0,0 +1,403 @@ +# CLI and configuration reference + +`s3gc` collects an object inventory and compares it with ClickHouse references. +For production, follow the [Kubernetes runbook](../deploy/kubernetes/README.md): +collect → dry-run → explicit human approval → delete → verify. This reference +describes the CLI, including options that the Kubernetes renderer does not expose. + +## Main parameters + +Start with these settings for a cleanup. The linked tables describe their +aliases, defaults, and environment variables. + +- [ClickHouse host, cluster, and expected replicas](#clickhouse-connection-and-topology) identify the reference scope and gate a clustered delete. +- [Bucket, object prefix, and disk name](#object-store-and-authentication) identify the object-store scope and the ClickHouse disk to compare. +- [Collection database, table prefix, and samples](#collection-and-inventory) identify the inventory. Use `--collect-only` to build it and `--use-collected` to reuse it. +- [Dry-run, use age, and deletion limits](#candidate-processing-and-deletion) control the review and use phases. + +Boolean environment and explicit-value options accept `true`/`false`, +`yes`/`no`, `on`/`off`, `1`/`0`, and the shorter `y`/`n` and `t`/`f`, ignoring +case. A bare flag such as `--dry-run` sets its value to true. Use its +explicit-value alias, such as `--dry-run-flag false`, to set it false. Where +aliases share a destination, the later CLI assignment wins. An unset optional +value is shown as `none`. An empty string default is shown as `empty`. + +The following tables give every application parameter. The first spelling is +the preferred CLI form where a kebab-case alias exists. All other accepted +aliases are listed in the same cell. Environment variables use the destination +name, which can differ from the flag spelling. String parameters take a string. +Integer parameters take an integer. Defaults shown are CLI defaults, not values +from the Kubernetes example or renderer. + +## ClickHouse connection and topology + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--ch-host`, `--chhost` | `S3GC_CHHOST` / `chhost` | `localhost`. String host. Pin every phase to the same replica, especially with the default node-local inventory. | +| `--ch-port`, `--chport` | `S3GC_CHPORT` / `chport` | `8123`. ClickHouse HTTP port. | +| `--ch-user-name`, `--chuser`, `--chusername` | `S3GC_CHUSER` / `chuser` | `default`. ClickHouse user. | +| `--ch-password`, `--chpass`, `--ch-pass` | `S3GC_CHPASS` / `chpass` | empty. ClickHouse password. Supply securely. | +| `--ch-timeout`, `--chtimeout`, `--send-receive-timeout`, `--ch-send-receive-timeout` | `S3GC_CHTIMEOUT` / `chtimeout` | `1800`. Integer ClickHouse send/receive timeout in seconds. | +| `--cluster-name`, `--cluster`, `--clustername` | `S3GC_CLUSTERNAME` / `clustername` | empty. String cluster whose replicas are queried for references. Empty uses the connected server only. It does not cover independent clusters. | +| `--expected-replicas` | `S3GC_EXPECTED_REPLICAS` / `expected_replicas` | none. Integer expected replica count. With a nonzero value, delete checks the cluster macro and reachable count before use and each sample. It requires a cluster name. Set a positive value for production. | +| `--replica-liveness-recheck` | `S3GC_REPLICA_LIVENESS_RECHECK` / `replica_liveness_recheck` | `false`. Bare boolean flag. For a Replicated auxiliary database, wait past the Keeper session timeout and re-check registered replica liveness before DDL. | +| `--replica-liveness-margin-ms` | `S3GC_REPLICA_LIVENESS_MARGIN_MS` / `replica_liveness_margin_ms` | `1000`. Nonnegative integer milliseconds added to that optional re-check. | + +The Replicated-database preflight is distinct from the delete-time topology +preflight. If ClickHouse cannot report database liveness, or returns no replica +rows, it logs a warning and skips that database check. The Kubernetes delete +entrypoint requires both cluster name and expected replica count. A direct CLI +run does not impose that requirement when the count is unset. + +## Object store and authentication + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--s3-ip`, `--s3ip` | `S3GC_S3IP` / `s3ip` | `127.0.0.1`. String S3-compatible API hostname or IP. | +| `--s3-port`, `--s3port` | `S3GC_S3PORT` / `s3port` | `9001`. API port. | +| `--s3-bucket`, `--s3bucket` | `S3GC_S3BUCKET` / `s3bucket` | `root`. String bucket name. | +| `--s3-path`, `--s3path` | `S3GC_S3PATH` / `s3path` | `data/`. String listing prefix. Empty selects the bucket root. Set the scope consistently across phases. | +| `--s3-disk-name`, `--s3diskname` | `S3GC_S3DISKNAME` / `s3diskname` | `s3`. String ClickHouse disk name used in the reference join and inventory table name. Select the underlying object disk, not its cache disk. | +| `--s3-auth`, `--s3auth` | `S3GC_S3AUTH` / `s3auth` | `static`. `static`, `aws`, or `iam` (case-insensitive). Static uses explicit keys or anonymous access if both are empty. AWS uses boto3. IAM uses MinIO's refreshable workload-identity provider. | +| `--s3-access-key`, `--s3accesskey` | `S3GC_S3ACCESSKEY` / `s3accesskey` | empty. Static access key. Supply with the secret key. | +| `--s3-secret-key`, `--s3secretkey` | `S3GC_S3SECRETKEY` / `s3secretkey` | empty. Static secret key. Supply securely. | +| `--s3-session-token`, `--s3sessiontoken` | `S3GC_S3SESSIONTOKEN` / `s3sessiontoken` | empty. Optional static session token. Requires both static keys. | +| `--s3-profile`, `--s3profile` | `S3GC_S3PROFILE` / `s3profile` | empty. AWS profile name. A profile selects `aws` and conflicts with `iam`. AWS mode rejects explicit S3 keys. | +| `--s3-region`, `--s3region` | `S3GC_S3REGION` / `s3region` | none. Optional string region. AWS mode can use its session region. | +| `--s3-secure`, `--s3secure`. Explicit-value forms: `--s3-secure-flag`, `--s3secureflag` | `S3GC_S3SECURE_FLAG` / `s3secure_flag` | `false`. Boolean HTTPS selection. The current HTTP client is configured with certificate verification disabled. Do not interpret this option as a certificate-verification guarantee. | +| `--s3-ssl-cert-file`, `--s3sslcertfile` | `S3GC_S3SSLCERTFILE` / `s3sslcertfile` | empty. String path assigned to `SSL_CERT_FILE` when secure mode is enabled. See the certificate-verification limitation above. | +| `--s3-connect-timeout` | `S3GC_S3_CONNECT_TIMEOUT` / `s3_connect_timeout` | `15`. Integer S3 connection timeout in seconds. | +| `--s3-read-timeout` | `S3GC_S3_READ_TIMEOUT` / `s3_read_timeout` | `120`. Integer S3 read timeout in seconds. | +| `--s3-retries` | `S3GC_S3_RETRIES` / `s3_retries` | `3`. Integer HTTP retry count for transient S3 errors. | + +HTTP retries happen within a running process. Batch deletion can also reconnect +and retry once after a transport failure. These retries are separate from +Kubernetes Job reruns, which the manifest disables. + +Static mode does not automatically read the standard `AWS_ACCESS_KEY_ID` and +`AWS_SECRET_ACCESS_KEY` variables. Supply the corresponding `S3GC_*` values +through a secret manager, or choose an appropriate AWS/IAM mode. Collection +requires listing permission. A dry-run using an existing inventory does not list +S3. A delete requires object-deletion permission. + +## Collection and inventory + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--collect-only`, `--collectonly`. Explicit-value forms: `--collect-only-flag`, `--collectonlyflag` | `S3GC_COLLECTONLY_FLAG` / `collectonly_flag` | `false`. Boolean. List objects and populate the inventory without running the use phase. | +| `--use-collected`, `--usecollected`. Explicit-value forms: `--use-collected-flag`, `--usecollectedflag` | `S3GC_USECOLLECTED_FLAG` / `usecollected_flag` | `false`. Boolean. Skip listing and use the existing inventory. See [Using a collected inventory](#using-a-collected-inventory). | +| `--collect-table-prefix`, `--collecttableprefix` | `S3GC_COLLECTTABLEPREFIX` / `collecttableprefix` | `s3objects_for_`. String prefix for the inventory table. The disk name is appended. A `database.prefix_` form can qualify it. Use a unique prefix per cleanup scope. | +| `--collect-database`, `--collectdatabase` | `S3GC_COLLECTDATABASE` / `collectdatabase` | empty. Optional database for inventory and run log. If the prefix embeds a database, both names must agree. Without either, the connection's current database is used. | +| `--collect-batch-size`, `--collectbatchsize` | `S3GC_COLLECTBATCHSIZE` / `collectbatchsize` | `1024`. Integer listing rows considered per insert batch. Use a positive value. | +| `--collect-total`, `--total`, `--collecttotal`, `--total-num` | `S3GC_TOTAL` / `total` | none. Optional integer target for collected rows. The code subtracts whole batches and stops on exactly zero, so this is not a reliable strict limit when a batch overshoots. | +| `--collect-after`, `--collectafter` | `S3GC_COLLECTAFTER` / `collectafter` | none. Optional string key passed as the listing start-after marker. Collect does not persist a resume marker. | +| `--age-hours`, `--age`, `--hours`, `--collectage`, `--collecthours` | `S3GC_AGE` / `age` | `0`. Integer hours. Collection includes listed objects whose age is at least this value. Separate from the use-phase age floor. | +| `--samples` | `S3GC_SAMPLES` / `samples` | `4`. Integer number of inventory partitions and use-phase samples. Use a positive value and keep it consistent across phases for partition pruning. | +| `--create-database`, `--createdatabase`. Explicit-value forms: `--create-database-flag`, `--createdatabase-flag` | `S3GC_CREATEDATABASE_FLAG` / `createdatabase_flag` | `false`. Boolean. Create the qualified inventory database during collection. A database name must be supplied. | +| `--drop-collecttable`, `--dropcollecttable`. Explicit-value forms: `--drop-collecttable-flag`, `--dropcollecttable-flag` | `S3GC_DROP_COLLECTTABLE_FLAG` / `drop_collecttable_flag` | `false`. Boolean. Drop and recreate the inventory during collection, losing prior inventory and checkpoints. Do not use when resuming a delete. | +| `--keep-data`, `--keepdata`. Explicit-value forms: `--keep-data-flag`, `--keepdataflag` | `S3GC_KEEPDATA_FLAG` / `keepdata_flag` | `false`. Boolean. Preserve the inventory after a successful delete. Without it, a successful non-dry use truncates the table. Dry-run does not truncate it. | + +## Candidate processing and deletion + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--dry-run`, `--dryrun`. Explicit-value forms: `--dry-run-flag`, `--dryrun-flag`, `--dryrunflag` | `S3GC_DRYRUN_FLAG` / `dryrun_flag` | `false`. Boolean. Report candidates without deleting objects. Without `--use-collected`, collection still runs and writes the inventory. | +| `--useage-hours`, `--useage`, `--usehours` | `S3GC_USEAGE` / `useage` | `24`. Integer hours. Use selects inventory rows older than this age. Below 24 is refused for preview and delete except with the development-only override. | +| `--use-after`, `--useafter` | `S3GC_USEAFTER` / `useafter` | none. Optional string. Process inventory keys lexically greater than this marker. | +| `--use-total`, `--usetotal` | `S3GC_USETOTAL` / `usetotal` | none. Optional integer SQL limit **per sample**, not a global run limit. A positive value can bound a trial. Unset it for a full run and verification. | +| `--delete-batch-size`, `--deletebatchsize` | `S3GC_DELETEBATCHSIZE` / `deletebatchsize` | `1000`. Positive integer objects per deletion/checkpoint batch. | +| `--order-by-objpath`. Explicit-value forms: `--order-by-objpath-flag` | `S3GC_ORDER_BY_OBJPATH` / `order_by_objpath` | `false`. Boolean. Order candidates by key within each sample. Sorting can be expensive. | +| `--use-remove-objects`, `--useremoveobjects` | `S3GC_USE_REMOVE_OBJECTS` / `use_remove_objects` | `true`. Boolean. This option requires a value. Use batch S3 deletion when true, per-object deletion when false. The recognized GCS endpoint switches to per-object deletion automatically. | +| `--non-interactive`, `--noninteractive`. Explicit-value forms: `--interactive-flag` | `S3GC_INTERACTIVE_FLAG` / `interactive_flag` | `true`. Boolean controls the CLI prompt. `--non-interactive` sets false. The prompt appears only when both stdin and stdout are terminals. A non-terminal run does not prompt even with the default. The Kubernetes token gate is separate. | +| `--dev-allow-short-useage` | `S3GC_DEV_ALLOW_SHORT_USEAGE` / `dev_allow_short_useage` | `false`. Boolean. This option requires a value. Permits a use age below 24 hours for isolated development fixtures only. Never use for customer or production data. | + +## Using a collected inventory + +`--use-collected` skips the S3 listing and reads an existing inventory table. +Use it after a normal collect or after importing a reviewed candidate list. +The [manual shared-bucket example](#manual-inventory-for-a-bucket-shared-by-independent-clusters) +shows how to prepare that list when independent clusters share a bucket. + +Each use phase checks the inventory against current ClickHouse references for +the configured cluster and disk. It also applies the age filter to the stored +timestamp. It does not refresh S3 metadata or check independent clusters. New +keys are missing from an old inventory, and overwritten keys can retain old +timestamps. A missing or empty inventory causes the run to fail. + +Use the same pinned ClickHouse host and cleanup settings for every phase. The +default inventory table is node-local. Its name is +`` in the chosen database. Start a new scope +with a unique table because collection adds rows to an existing table. + +These examples are for an isolated development environment. Set the scope as +shown in the [README](../README.md#local-setup-and-read-only-preview) and +provide credentials securely. Use the [Kubernetes runbook](../deploy/kubernetes/README.md) +for production. + +```bash +# 1. Collect into a new, unique inventory table. +.venv/bin/python s3gc.py --collect-only --keep-data --verbose + +# 2. Review candidates in that inventory. +.venv/bin/python s3gc.py --use-collected --dry-run --verbose + +# 3. Run from a terminal after review and explicit approval. +.venv/bin/python s3gc.py --use-collected --keep-data --verbose + +# 4. Verify the same inventory, without use-total or use-after limits. +.venv/bin/python s3gc.py --use-collected --dry-run --verbose +``` + +The CLI prompts only when stdin and stdout are terminals. Get explicit human +approval before a non-interactive delete too. Keep the table after delete so +its checkpoints remain available for recovery and verification. If delete +fails, inspect the run log and confirm that the scope and approval still apply +before retrying. Do not re-collect or drop the table during recovery. + +A zero-candidate verification covers only this inventory and its filters. It +does not check objects absent from the inventory or prove the bucket is empty. + +## Manual inventory for a bucket shared by independent clusters + +When different ClickHouse clusters share an overlapping bucket/prefix, use a +manually reviewed inventory. A normal clustered s3gc comparison covers the +configured cluster, not other independent clusters. Putting the full shared +listing directly into the s3gc inventory would let objects referenced only by +another cluster appear orphaned. + +First export the listing and compare it with references from every consumer. +Import only the reviewed candidates into the inventory table. Then use +`--use-collected` to preview, delete, and verify that table without listing S3 +again. The steps below show the complete workflow. + +The outline below uses an AWS-compatible listing, CSV staging, and a dedicated +replicated candidate table. Adapt endpoints, authentication, names, and SQL to +the installation. It requires a complete view of every cluster, replica, disk, +and any other consumer of the shared scope. If that view cannot be established, +stop. Coordinate all consumers so that references cannot change between the +final cross-cluster review and deletion. This coordination is outside s3gc. +Work in a private directory outside Git, use least-privilege credentials, and +keep the raw listing separate from the candidate table. + +### 1. Export the full scoped listing + +For an AWS-compatible endpoint, the following example uses the installed boto3 +dependency and its paginated `list_objects_v2` API. Configure the SDK's +credentials and region outside the script. For another S3-compatible endpoint, +configure the session/endpoint as appropriate. Set `S3GC_RUN_DIR` to an existing +private directory outside the repository, and set `S3GC_S3BUCKET` and +`S3GC_S3PATH` to the reviewed scope. An empty prefix lists the bucket root. +The CSV has one row per object and preserves the object-store size and UTC +last-modified time. Do not fabricate or backdate timestamps: s3gc's age guard +uses the imported value. + +```bash +.venv/bin/python - > "$S3GC_RUN_DIR/listing.csv" <<'PY' +import csv, datetime, os, sys +import boto3 + +writer = csv.writer(sys.stdout) +writer.writerow(['objpath', 'size', 'last_modified']) +s3 = boto3.client('s3') +pages = s3.get_paginator('list_objects_v2').paginate( + Bucket=os.environ['S3GC_S3BUCKET'], Prefix=os.environ['S3GC_S3PATH']) +for page in pages: + for obj in page.get('Contents', []): + modified = obj['LastModified'].astimezone(datetime.timezone.utc) + writer.writerow([obj['Key'], obj['Size'], modified.strftime('%Y-%m-%d %H:%M:%S')]) +PY +``` + +Check that the command succeeded, pagination completed, and the row count, +bucket, prefix, object keys, sizes, and timestamps are credible. The export is +a point-in-time listing, not an atomic snapshot of all consumers. Preserve its +provenance and obtain a new listing if the scope or assumptions change. + +### 2. Stage the listing and collect all references + +The examples use a pre-existing `s3gc` database and a private, per-replica +ClickHouse connection configured in your client. Use unique staging names for +the run. Create staging tables on the chosen host. These are **not** the table +that s3gc will use: + +```sql +CREATE TABLE s3gc.manual_listing + (objpath String, size Int64, last_modified DateTime('UTC')) + ENGINE = MergeTree ORDER BY objpath; +CREATE TABLE s3gc.manual_references (objpath String) + ENGINE = MergeTree ORDER BY objpath; +``` + +Import the listing into staging through the same pinned host: + +```bash +clickhouse-client --host '' \ + --query 'INSERT INTO s3gc.manual_listing FORMAT CSVWithNames' \ + < "$S3GC_RUN_DIR/listing.csv" +``` + +For **each** independent cluster, connect to that cluster and export references +from every replica, using the disk names that map to this bucket/prefix on that +cluster. The names can differ between clusters. For example, after replacing +the placeholders with the cluster's actual names: + +```bash +clickhouse-client --host '' \ + --query "SELECT DISTINCT remote_path AS objpath + FROM clusterAllReplicas('', system.remote_data_paths) + WHERE disk_name IN ('') + SETTINGS skip_unavailable_shards = 0 FORMAT CSVWithNames" \ + > "$S3GC_RUN_DIR/references-cluster-a.csv" +``` + +Check the expected replica count, successful query completion, disk-to-bucket +mapping, and key format for each cluster. Include every relevant disk and +non-ClickHouse consumer in the review. If a replica or consumer is unavailable, +the exports are incomplete. Stop rather than treating missing references as +orphans. Import each verified reference CSV into `manual_references` on the +pinned staging host, using a distinct file per source: + +```bash +clickhouse-client --host '' \ + --query 'INSERT INTO s3gc.manual_references FORMAT CSVWithNames' \ + < "$S3GC_RUN_DIR/references-cluster-a.csv" +``` + +Repeat for every cluster. Add references from other consumers in the same +one-column `objpath` format. Confirm that keys are comparable to the S3 listing +and verify import counts. Duplicate references are harmless for the anti-join. + +### 3. Review candidates and load a replicated inventory + +Generate a candidate CSV from the staging tables. Only rows absent from the +union of all reference exports enter it. Preserve listing metadata and mark rows +active. Review keys, counts, total bytes, timestamps, scope, and the source +exports before importing. Exclude anything whose ownership is uncertain. + +```bash +clickhouse-client --host '' \ + --query "SELECT l.objpath, l.size, l.last_modified, true AS active + FROM s3gc.manual_listing AS l + LEFT ANTI JOIN s3gc.manual_references AS r ON l.objpath = r.objpath + FORMAT CSVWithNames" > "$S3GC_RUN_DIR/reviewed-candidates.csv" +``` + +Create a **new, empty** auxiliary table with the schema and replacement key +expected by s3gc. This example uses `S3GC_COLLECTDATABASE=s3gc`, +`S3GC_COLLECTTABLEPREFIX=manual_cleanup_`, `S3GC_S3DISKNAME=s3`, and +`S3GC_SAMPLES=4`. Use a unique Keeper path and a cluster with correctly +configured `{shard}` and `{replica}` macros. Review the DDL for the installation. + +Check the database engine before choosing how to create the table: + +```sql +SELECT engine FROM system.databases WHERE name = 's3gc'; +``` + +The `CREATE TABLE` example below uses `ON CLUSTER` for an Atomic database. +If `s3gc` uses `ENGINE = Replicated`, omit `ON CLUSTER ''`. +That database distributes table DDL itself. See the +[Replicated database documentation](https://clickhouse.com/docs/engines/database-engines/replicated). + +The table engine replicates data within a shard. It does not automatically +share rows across independent shards or clusters. Table replication is separate +from a Replicated database's DDL replication. See ClickHouse's +[replicated table engine documentation](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replication). + +```sql +CREATE TABLE s3gc.manual_cleanup_s3 ON CLUSTER '' +( + objpath String, + size Int64, + last_modified DateTime('UTC'), + active Bool +) +ENGINE = ReplicatedReplacingMergeTree( + '/clickhouse/tables/{shard}/s3gc_manual_cleanup_s3', '{replica}') +PARTITION BY CRC32(objpath) % 4 +ORDER BY objpath; +``` + +Import only the reviewed candidate CSV on the pinned execution replica: + +```bash +clickhouse-client --host '' \ + --query 'INSERT INTO s3gc.manual_cleanup_s3 FORMAT CSVWithNames' \ + < "$S3GC_RUN_DIR/reviewed-candidates.csv" +``` + +Compare the imported keys, metadata, count, and total bytes with the reviewed +CSV. Confirm replication has caught up where the table will be read. Replication +is asynchronous, so keep all s3gc phases pinned to the import replica. +Resolve any partial import before proceeding. An empty candidate table cannot +be passed to the use phase. If the review found no candidates, record that +result instead of starting delete. + +The normal s3gc collector creates a node-local table. You must create the +replicated table yourself for this workflow. + +### 4. Preview, approve, delete, and verify + +Keep the same database, table prefix, disk, bucket, object prefix, execution +cluster, expected replicas, sample count, and age window. Refresh references +from **all** consumers immediately before delete, and verify that candidate +objects still have the reviewed metadata. Maintain the agreed period in which +references cannot change and candidate keys cannot be overwritten or reused. +Then obtain explicit approval for the reviewed candidate set. If any +cross-cluster reference, candidate metadata, or scope changes, stop and rebuild +and re-review the candidate inventory. s3gc rechecks only the configured +cluster during use. It does not recheck other clusters or S3 metadata for you. + +Run the Kubernetes `dry-run`, approved `delete`, and `dry-run` verification +phases against the existing table. They already select `--usecollected`. Do not +run the Kubernetes collect phase or set the fresh-run drop flag on this table. +Keep the same pinned host and inventory for recovery and verification. A zero +result applies to the reviewed imported inventory and current configured-cluster +filters, not to a newly generated bucket listing. Preserve review evidence and +clean up private staging artifacts according to your retention policy. + +## Advanced: configuration files and precedence + +The tables above list each parameter's configuration key. Use those keys in a +JSON or YAML file passed with `--cfg`. For example, this file selects a replica +and enables a preview: + +```json +{ + "chhost": "replica.example.invalid", + "collectdatabase": "s3gc", + "collecttableprefix": "example_", + "dryrun_flag": true +} +``` + +Pass `--cfg /path/to/config.json`. The parser starts with defaults, applies +environment values, then processes command-line arguments in order. A `--cfg` +file is applied at its position on the command line: it can override an earlier +CLI option, and a later CLI option can override the file. Multiple files and +options follow the same ordering. `S3GC_CFG` can also select a config file. +The parser applies other environment variables after that environment-selected +file. Keep secrets outside config files and shell history. + +`--help` (`-h`) prints CLI help. `--print_config` prints the effective parsed +configuration and exits. **Its output can contain credentials**, so do not +share or commit it. `--list-options` (`--listoptions`) prints environment-style +options and redacts configured credentials. These inspection options do not +contact ClickHouse or S3. `--cfg` has environment variable `S3GC_CFG`. The help +and print-config actions have no documented environment equivalent. The +list-options variable is `S3GC_LISTOPTIONS` (boolean, default `false`). + + +## Advanced: logging and inspection + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--verbose`. Explicit-value forms: `--verbose-flag`, `--verboseflag` | `S3GC_VERBOSE_FLAG` / `verbose_flag` | `false`. Boolean. Enable informational logs. | +| `--debug`. Explicit-value forms: `--debug-flag`, `--debugflag` | `S3GC_DEBUG_FLAG` / `debug_flag` | `false`. Boolean. Enable more detailed logs, including per-object output. | +| `--silent`. Explicit-value forms: `--silent-flag`, `--silentflag` | `S3GC_SILENT_FLAG` / `silent_flag` | `false`. Boolean. Restrict console logging to critical messages. | +| `--run-log`, `--runlog` | `S3GC_RUNLOG_FLAG` / `runlog_flag` | `true`. Boolean. This option requires a value. Write durable run events alongside the inventory. Logging failure warns and falls back to stdout. | +| `--run-id`, `--runid` | `S3GC_RUNID` / `runid` | empty. String identifier for run-log rows. Empty generates a timestamped ID. Kubernetes supplies the Job name. | +| `--list-options`, `--listoptions` | `S3GC_LISTOPTIONS` / `listoptions` | `false`. Boolean. Print environment-style options with credentials redacted and exit. | +| `--cfg` | `S3GC_CFG` / `cfg` | none. Configuration file path. Place before CLI overrides. | +| `--help`, `-h` | none | Print help and exit. | +| `--print_config` | none | Print effective configuration and exit. Output may contain secrets. | diff --git a/docs/plans/cli-parameter-documentation.md b/docs/plans/cli-parameter-documentation.md new file mode 100644 index 0000000..4c93dca --- /dev/null +++ b/docs/plans/cli-parameter-documentation.md @@ -0,0 +1,168 @@ +# Plan: complete CLI parameter documentation + +Status: implemented on `docs/cli-parameter-reference`. +This completed plan records the original work. Use the +[CLI reference](../cli-reference.md) and +[Kubernetes runbook](../../deploy/kubernetes/README.md) for current instructions. + +Planning branch: `docs/cli-reference-plan`. +Proposed implementation branch: `docs/cli-parameter-reference`. + +## Problem and intent + +The README introduces common settings, but operators need a complete reference +that explains each parameter and how parameters work together across a cleanup. +Complete the first documentation item in `TODO.md`, with particular attention to +`--usecollected` and the separation of collection, review, deletion, and verification. + +This is an editorial change. Preserve CLI names, aliases, defaults, environment +variables, runtime behavior, deployment settings, and deletion safeguards. Exclude +the deleted manifest feature. Do not fix runtime defects as part of this work. + +## Reference style + +Use Altinity's [clickhouse-backup configurable parameters](https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md#configurable-parameters) +as a presentation reference: group related settings and put environment names, +value meanings, defaults, and constraints near their explanations. Its +[CLI manual](https://github.com/Altinity/clickhouse-backup/blob/master/Manual.md) +also puts usage, aliases, and option descriptions together. References inspected +on 2026-10-27. + +Adapt that structure to s3gc's existing interfaces. Distinguish illustrative +values from actual defaults and provide operational explanations beyond help text. + +## Deliverables + +| File | Planned change | +| --- | --- | +| `docs/cli-reference.md` | Complete parameter reference, configuration rules, phase walkthrough, and examples. | +| `README.md` | Link the reference beside local setup and introduce the collect/use distinction briefly. | +| `deploy/kubernetes/README.md` | Cross-link the reference and explain how entrypoint phases select CLI flags; retain the production runbook. | +| `deploy/kubernetes/example.env` | Clarify that the documented use-phase limit applies to each sample. | +| `CHANGELOG.md` | Record the completed documentation under Unreleased. | +| `TODO.md` | Remove the completed documentation item only after its acceptance criteria are met. | + +## Implementation sequence + +1. Read `CLAUDE.md`, `CHANGELOG.md`, `TODO.md`, and the existing operator guides. + Inventory every argument declaration in `s3gc.py`, including aliases, shared + destinations, explicit-value boolean forms, and parser-provided help/config + options. Consult runtime consumers, the entrypoint, renderer, and template to + distinguish declared defaults from effective behavior. Use local sources; + consult the clickhouse-reference skill if a ClickHouse claim needs verification. +2. Build the complete reference, grouped into ClickHouse connection/topology, + object storage/authentication, collection/inventory, candidate processing and + deletion, timeouts/retries, and logging/configuration/development controls. + Give each setting a stable anchor. For each setting document purpose, all + aliases, value type and accepted values, units, actual default, exact + `S3GC_*` variable, affected phases, and interactions. State explicitly when a + value has no default or a parser utility has no environment equivalent. + Document enforced restrictions separately from operational recommendations. +3. Explain configuration files through `--cfg`, configuration destination names, + booleans, and precedence among defaults, environment, files, and CLI arguments. + Verify precedence and shared-destination alias behavior with the pinned + jsonargparse version using offline parser probes; do not assume precedence + matches clickhouse-backup. Include a non-secret configuration example and + explain `--help`, `--print_config`, and the redacted `--list-options` output. +4. Write the collect/use explanation and examples described below. Explain + defaults that can start collection and deletion in one invocation; distinguish + auxiliary-table writes during preview from S3 object deletion. Check CLI + confirmation behavior separately from the Kubernetes confirmation token. +5. Add README/runbook links, remove unnecessary duplicate explanations, and + reconcile any conflicting guidance against the implementation. Preserve the + production approval sequence and existing runbook requirements. Finish the + changelog and TODO edits after validation. + +## Required explanation of `--usecollected` + +- Document `--usecollected`, its `--use-collected` alias, the explicit-value forms + `--usecollectedflag` and `--use-collected-flag`, default `false`, and the exact + variable `S3GC_USECOLLECTED_FLAG`. +- Explain that it skips collection and reads the existing auxiliary inventory. + It re-evaluates references and the age filter during the use phase; the + inventory is not a saved final deletion decision and is not refreshed by an + S3 listing. New keys are absent until another collect, and overwritten keys + can retain stale metadata. +- Show the distinction between `--collect-only`, `--use-collected --dry-run`, + and a delete using the collected inventory. Explain the effect of omitting + `--use-collected`, and the interaction with `--keep-data`, tombstones, table + truncation, `--drop-collecttable`, and retrying an interrupted delete. +- Explain how the selected database, table prefix, disk, and pinned ClickHouse + host identify the inventory. Cover missing/empty inventory errors and why + every phase must use the same replica and cleanup scope. +- Keep collection age (`--age`) separate from use age (`--useage`), collection + limits/markers separate from use limits/markers, and document sampling and + ordering constraints from the code. Describe actual limit behavior rather + than promising a global cap without checking it. +- Map Kubernetes collect, dry-run, delete, and dev-automation phases to their + actual entrypoint arguments. Keep renderer keys distinct from `S3GC_*` + variables; do not imply all CLI options are exposed by the renderer. + +## Manual inventory for a shared bucket + +Document the preferred manual workflow when independent clusters share an +overlapping bucket/prefix: export the full scoped listing to CSV, gather live +references from every participating cluster and replica, and exclude those +references before approving candidates. Include example listing, CSV import, +reference-export, and comparison commands. A cluster's own replicas can normally +be covered with `--cluster-name`; independent clusters require this additional +cross-cluster review because s3gc checks only its configured reference scope. + +Import the reviewed candidates into a dedicated replicated auxiliary table with +the schema, naming, replacement semantics, and partitioning expected by s3gc. +Distinguish table data replication from a Replicated database's DDL replication, +and keep the execution host pinned. The full raw listing belongs in a separate +staging table so unrelated objects cannot become deletion candidates. Explain +metadata provenance, complete reference exports, replication readiness, and +freshness checks before delete. Use `--use-collected` for preview, approved +delete, recovery, and verification; do not re-collect into the reviewed table. + +## Examples and safety requirements + +Provide one consistent placeholder configuration and separate examples for +collect, dry-run, approved delete, and verification. Prefer existing kebab-case +aliases. Preserve inventory through the delete/verify sequence as required by +the implementation. State prerequisites next to the commands, including the +same host and scope, cluster and expected replicas, and use age of at least +24 hours. Keep human review and explicit approval before deletion; explain the +actual confirmation mechanism for each execution path. + +Include an interrupted-delete recovery example using the same inventory and a +brief contrast with restarting collection. Verification must explain what a +zero-candidate result establishes for that inventory and time window. Keep +production commands separate from the development-only age override. Use +placeholders and externally injected credentials throughout. Do not execute +any live example while writing or validating the documentation. + +## Acceptance criteria and validation + +- Every current CLI argument and alias appears in the reference, with the fields + above checked against parser declarations and runtime consumers. Review an + inventory-to-documentation checklist to catch omissions, including boolean + destinations ending in `_FLAG` and parser utility options. +- Configuration precedence and representative boolean/alias combinations are + demonstrated with isolated offline probes of the installed pinned parser. + Clear inherited `S3GC_*` values for probes and use dummy configuration only; + do not print the real environment or call `main()`. +- A reader can explain when `--use-collected` is needed, what it reuses, what + gets checked again, and how to preserve the inventory for recovery and verify. +- The shared-bucket example covers full CSV listing/import, all-cluster reference + review, and a replicated candidate table. It states that `--use-collected` + does not discover other clusters or make an unfiltered shared listing safe. +- All four workflow examples use existing options, consistent scope, and the + existing approval, age, topology, and checkpoint safeguards. Kubernetes + explanations agree with the renderer, template, and entrypoint. +- Relative links and anchors resolve, Markdown tables/code blocks render, and + `git diff --check` passes. Run help/config inspection offline only. This + editorial change needs no new runtime tests or live-system validation. + If executable files enter scope, split that work out and apply the repository's + spec-first TDD and required offline pytest/render/Kubeconform checks. +- Only the planned documentation files change; no secrets, customer artifacts, + dependencies, generated configuration dumps, or rendered manifests enter Git. + +## Handoff + +The planning stage ended when implementation was requested. The implementation +branch is `docs/cli-parameter-reference`; carry this plan with the documentation. +Keep the TODO item open until implementation is done. Publishing a PR or merging +is outside this documentation implementation request. diff --git a/skills/altinity-clickhouse-s3gc/SKILL.md b/skills/altinity-clickhouse-s3gc/SKILL.md index eff6650..65f34da 100644 --- a/skills/altinity-clickhouse-s3gc/SKILL.md +++ b/skills/altinity-clickhouse-s3gc/SKILL.md @@ -5,79 +5,102 @@ description: Safely investigate, plan, review, and execute cleanup of orphaned o # ClickHouse S3 Garbage Collection -Use this skill for the `s3gc` workflow only: identifying objects in a configured object-store -bucket/prefix that are absent from ClickHouse `system.remote_data_paths`, reviewing candidates, -and removing approved orphaned objects. It is not a generic S3 cleanup procedure. +Use this skill to find, review, and remove approved orphaned objects with s3gc. +It compares an inventory with ClickHouse `system.remote_data_paths` for the +selected disk and cluster. Use the Kubernetes runbook for production. ## Non-negotiable rules -- Deletion is irreversible. For customer and production work, use the Kubernetes Job workflow: - **collect → dry-run → explicit human approval → delete → verify**. -- Scope the bucket and prefix as narrowly as possible. A wrong prefix can produce a misleading - clean result or expand deletion scope. -- Use the underlying object disk (for example `s3` or `gcs`) for `S3DISKNAME`, never its - `*_cache` wrapper. The cache does not represent a separate object-reference scope. -- Pin `CHHOST` to a per-replica ClickHouse Service, and use the same host for every phase. The - auxiliary inventory table is node-local; a load-balanced Service can send phases to different - replicas and invalidate the result. -- For clustered cleanup, require the actual cluster name and expected replica count. The delete - phase must pass `s3gc`'s cluster/replica preflight before any S3 call. -- Use a unique `COLLECTTABLEPREFIX` for each bucket/prefix cleanup. Preserve it after a partial - delete: confirmed deletions are checkpointed, and a replacement delete Job resumes from that - state without re-collecting. -- Keep `USEAGE_HOURS` at 24 or higher for production phases. This protects objects uploaded to S3 - before ClickHouse has registered their paths. -- An agent must not supply `DELETE_ORPHANS`, create a delete manifest with that token, or bypass - the reviewed dry-run gate. A human must review the immediate dry-run output and explicitly - authorize deletion. -- Never commit or record credentials, customer `.env` files, rendered manifests, bucket names, - cluster identifiers, or command output containing them. Keep run material outside this repo and - use Kubernetes Secrets or workload identity. +- Deletion is irreversible. Follow **collect → dry-run → explicit human approval + → delete → verify** for customer and production work. +- Limit the bucket and prefix to the intended scope. A wrong prefix can hide + orphans or include objects you must not delete. +- Set `S3DISKNAME` to the underlying object disk, such as `s3` or `gcs`. + Never use its `*_cache` wrapper. +- Pin `CHHOST` to one per-replica ClickHouse Service for every phase. The + default inventory is node-local. Keep the same pinned host even when using + a manually created replicated inventory. +- For clustered cleanup, require the actual cluster name and expected replica + count. Delete must pass the cluster/replica preflight before deleting objects. + The Kubernetes delete workflow requires both settings. +- Use a unique `COLLECTTABLEPREFIX` for each cleanup scope. Keep it after a + partial delete so a replacement Job can use the confirmed checkpoints. +- Keep `USEAGE_HOURS` at 24 or more for production. This protects objects + uploaded before ClickHouse registers their paths. +- An agent must not supply `DELETE_ORPHANS`, create a delete manifest with that + token, or bypass dry-run review. A human must review the immediate dry-run + output and explicitly authorize deletion. +- Never record credentials. Keep customer configuration, rendered manifests, + bucket names, cluster identifiers, and their output outside Git. Use a private + run directory and the authorized support record for run evidence. + Provide credentials through Kubernetes Secrets or workload identity. ## Workflow ### 1. Establish the cleanup scope -Before collecting, record the purpose, affected disk, endpoint, bucket, exact prefix, ClickHouse -cluster, expected replica count, namespace, and per-replica `CHHOST`. Confirm the intended S3 -authentication mode and the minimum ClickHouse/S3 permissions. Prefer workload identity (`iam`) -for an in-cluster AWS cleanup; use static credentials only from a runtime Secret. +Record the purpose, disk, endpoint, bucket, prefix, cluster, expected replicas, +namespace, and pinned host in the private run record. Confirm the authentication +mode and required ClickHouse and S3 permissions. Prefer `iam` for in-cluster AWS +work. Supply static credentials only through a runtime Secret. -Read [the Kubernetes operator guide](../../deploy/kubernetes/README.md) and start from -[`deploy/kubernetes/example.env`](../../deploy/kubernetes/example.env) copied to a private run -directory outside the repository. Use an immutable multi-architecture image digest, not a tag. -Select an existing namespace-local ServiceAccount configured for the cluster's intended workload -identity mode; the runner does not need Kubernetes API credentials. +Read the [Kubernetes runbook](../../deploy/kubernetes/README.md). Copy +[example.env](../../deploy/kubernetes/example.env) to the private run directory. +Use a multi-architecture image pinned by digest and an existing ServiceAccount +configured for the intended identity. The Job does not need Kubernetes API access. + +If independent clusters share the scope, follow the +[manual inventory workflow](../../docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters). +Check references from every cluster, replica, relevant disk, and other consumer. +Import only reviewed candidates. s3gc's configured-cluster check cannot see +independent clusters. Stop if the review cannot cover every consumer. ### 2. Collect and review -Render and run a `collect` Job, then a `dry-run` Job against the same `CHHOST` and collection -table. Collect does not delete; dry-run reports the candidate count and size. Review the exact -bucket and prefix too. Capture the rendered configuration location and the non-secret Job/log -evidence in the active support record without copying credentials or customer artifacts into Git. +Choose the inventory path before running a Job: + +- For normal collection, run `collect`, then `dry-run`. +- For a manually imported inventory, skip `collect` and start with `dry-run`. + Never re-collect into the reviewed candidate table. + +Use the same pinned host and inventory for subsequent phases. +`--use-collected` reads the existing table without listing S3 again. For a manual +inventory, keep the all-consumer reference review current through deletion, as +the manual workflow requires. -Treat an empty or missing collected inventory as a failure to investigate, not proof that the -bucket is clean. Verify the prefix, disk name, and replica-pinned service first. For large buckets, -shard collection by prefix; interrupted collection has no checkpoint and must be rerun only for -the affected shard. +Collection does not delete objects. Dry-run reports the candidate count and +size. Review the bucket and prefix too. Store the configuration location and +only redacted, non-secret Job/log evidence in the authorized support record. +Keep credentials and customer artifacts out of Git. + +Investigate a missing or empty inventory before proceeding. Check the prefix, +disk, and pinned host. For large buckets, collect separate prefix shards. +Interrupted collection has no checkpoint. Rerun only the failed shard before +deletion, not while recovering a delete. ### 3. Require approval and delete -Present the dry-run candidate count, size, exact bucket/prefix scope, and any uncertainty to the -authorized customer/operator. Do not proceed without explicit approval. +Present the candidate count, size, exact scope, and any uncertainty to the +authorized operator. Do not proceed without explicit approval. + +After the human supplies the token in the private configuration, render a new +delete Job with the same host, scope, and inventory. Monitor it to completion. +Keep the durable run log enabled unless stdout is the approved record. -After the human supplies the delete confirmation in the private runtime configuration, render a -new delete Job using the same collection-table prefix. Monitor it to completion. A failed delete -Job has no automatic retry; start a replacement delete Job with the same prefix so the confirmed -deletion checkpoints are preserved. Do not re-collect unless the scope itself changed. Keep the -durable run log enabled unless stdout is the approved operational record. +A failed Job is not rerun automatically. Individual S3 requests can retry within +the process. Inspect the failure and confirm that approval still applies before +starting a replacement Job. Preserve the inventory and checkpoints. Do not +re-collect during recovery. A changed scope needs a new inventory, review, and +approval. ### 4. Verify and hand off -Run the final `dry-run` phase against the same host and inventory. It must report zero remaining -candidates before declaring the cleanup complete. Record the approved scope, image digest, -cluster/replica preflight result, candidate and deleted totals, verification result, and any -partial-failure recovery steps in the active ticket's handoff record. +Run a final dry-run against the same host and inventory with `USETOTAL` cleared. +It must report zero candidates before the cleanup is complete. That result +covers the reviewed inventory and age window, not newly uploaded objects. + +Record the approved scope, image digest, preflight result, candidate and deleted +totals, verification result, and recovery steps in the authorized handoff record. ## References @@ -86,4 +109,5 @@ partial-failure recovery steps in the active ticket's handoff record. - [s3gc README](../../README.md) — tool model, authentication modes, direct read-only usage, and local development checks. - [`deploy/kubernetes/example.env`](../../deploy/kubernetes/example.env) — non-secret renderer - template; copy it only to a private run directory. + template. Copy it only to a private run directory. +- [CLI reference](../../docs/cli-reference.md) — parameter details and manual inventory preparation. diff --git a/tests/dev_cluster_e2e_spec.md b/tests/dev_cluster_e2e_spec.md index 329635e..6338974 100644 --- a/tests/dev_cluster_e2e_spec.md +++ b/tests/dev_cluster_e2e_spec.md @@ -1,5 +1,10 @@ # Dev-cluster E2E spec +Historical acceptance criteria for the test now implemented in +[test_s3gc_dev_cluster.py](test_s3gc_dev_cluster.py). See its explicit opt-in +requirements before using an isolated development cluster. These criteria are +not instructions to run a live test as part of the default suite. + Add an opt-in test that validates s3gc's Replicated-database preflight and cross-replica inventory checkpoints against an isolated development ClickHouse cluster and S3 prefix. The test must never run in the default pytest suite.