From baba9413beaa0030af69b70ba709011a75c7409d Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Tue, 27 Oct 2026 15:57:33 +0100 Subject: [PATCH 1/3] docs: add CLI parameter reference and manual inventory workflow --- CHANGELOG.md | 9 +- README.md | 9 + TODO.md | 6 - deploy/kubernetes/README.md | 14 +- deploy/kubernetes/example.env | 7 +- docs/cli-reference.md | 385 ++++++++++++++++++++++ docs/plans/cli-parameter-documentation.md | 164 +++++++++ 7 files changed, 580 insertions(+), 14 deletions(-) create mode 100644 docs/cli-reference.md create mode 100644 docs/plans/cli-parameter-documentation.md diff --git a/CHANGELOG.md b/CHANGELOG.md index e810fdb..d2f0051 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,14 @@ expose. ## [Unreleased] -Nothing yet. +### Documentation + +- Added a complete CLI and configuration reference, including aliases, + environment variables, defaults, precedence, and phase examples. Explained + how `--usecollected` reuses an inventory and documented a manually reviewed + shared-bucket workflow, because a single configured cluster cannot identify + references held by independent clusters. Clarified that `USETOTAL` applies + per sample rather than as a global Job limit. ## [0.8.0] - 2026-10-25 diff --git a/README.md b/README.md index 84ed368..59e6089 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,10 @@ would look orphaned. Use a unique collection-table prefix for each bucket and prefix. Never commit credentials, rendered manifests, target-cluster details, or run output. +If independent clusters share an overlapping bucket or prefix, review references +from every consumer before importing a candidate inventory. The +[shared-bucket workflow](docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters) +explains this manual preparation and the limits of the configured-cluster check. ## Local setup and read-only preview @@ -84,6 +88,11 @@ in a file or command history. Then run a preview: .venv/bin/python s3gc.py --verbose --dry-run ``` +This preview collects an inventory first. To preview an existing inventory +without listing again, use `--use-collected --dry-run`. See the complete +[CLI and configuration reference](docs/cli-reference.md) for every parameter, +aliases, environment variables, defaults, and phase examples. + Use direct deletion only for controlled development work. The Kubernetes runbook separates collection, review, approval, deletion, and verification. diff --git a/TODO.md b/TODO.md index e76f536..84f23e5 100644 --- a/TODO.md +++ b/TODO.md @@ -4,12 +4,6 @@ Possible, interesting, and deliberately deferred engineering or operational improvements. Completed behaviour belongs in `CHANGELOG.md`; this file is the forward-looking backlog. -- [ ] In a separate documentation PR, document every CLI parameter: its purpose, - accepted values, default, aliases, exact `S3GC_*` environment variable name, - and interactions or constraints. Prefer existing kebab-case aliases in - examples, explain CLI/environment/configuration precedence, and include - examples for collect, dry-run, delete and verify. Keep existing option names - and aliases; a naming overhaul is unnecessary. - [ ] Record collect progress so a crashed `--collectonly` run can resume. `--collectafter` already starts listing after a given key, but nothing records the last key listed, so after a crash the operator cannot tell what diff --git a/deploy/kubernetes/README.md b/deploy/kubernetes/README.md index 80c8dbc..8195224 100644 --- a/deploy/kubernetes/README.md +++ b/deploy/kubernetes/README.md @@ -193,8 +193,16 @@ kubectl -n logs -f job/ | Verify | Set `PHASE=dry-run`, a new `JOB_NAME`, and clear `DELETE_CONFIRMATION`. | Report zero candidates. | Do not run delete from an unreviewed dry-run. Before the first full delete for -a new image or cluster, set `USETOTAL` to a few thousand objects for one -bounded delete. Clear it before the full run. +a new image or cluster, set `USETOTAL` to a few thousand objects per sample +for a bounded trial delete. Clear it before the full run and verification. +The use-phase query applies this limit separately to each sample, so it is not +a global cap on the Job. + +The entrypoint uses `--collectonly --keepdata` for collection, +`--usecollected --dry-run` for review and verification, and +`--usecollected --keepdata --non-interactive` for delete after the token and +topology-input gates. See the [CLI reference](../../docs/cli-reference.md) for +all parameters and the [manual shared-bucket workflow](../../docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters). ### 5. Recover or verify @@ -219,7 +227,7 @@ windows. Start with [example.env](example.env). | Limits | `SAMPLES`, `DELETE_BATCH_SIZE`, `USEAGE_HOURS`, `ACTIVE_DEADLINE_SECONDS`, `TTL_SECONDS_AFTER_FINISHED`, `MEMORY_REQUEST`, `MEMORY_LIMIT` | Keep `SAMPLES` unchanged after collect for partition pruning. | | Behavior | `PHASE`, `DELETE_CONFIRMATION`, `ORDER_BY_OBJPATH`, `VERBOSE` | The renderer requires the confirmation token for delete and development automation. | -`USETOTAL` is optional and limits one use phase. `RUNLOG` defaults to `true`; +`USETOTAL` is optional and limits each sample in a use phase. `RUNLOG` defaults to `true`; set it to `false` only when stdout is an adequate operational record. ## Durable run history diff --git a/deploy/kubernetes/example.env b/deploy/kubernetes/example.env index 6637aa8..88c6208 100644 --- a/deploy/kubernetes/example.env +++ b/deploy/kubernetes/example.env @@ -50,10 +50,9 @@ DELETE_BATCH_SIZE=1000 # between its upload to S3 and its registration in system.remote_data_paths. # Raising it is fine and sometimes wise; lowering it is not. USEAGE_HOURS=24 -# Optional cap on how many collected objects a use phase processes. Leave empty -# for a full run. Set it to bound the first delete against a new image or a new -# cluster: a few thousand objects exercise collect -> delete -> tombstone -# end to end in one sample instead of committing to a multi-hour run. +# Optional limit on how many collected objects each sample processes. Leave +# empty for a full run. Set it to bound the first delete against a new image or +# cluster; the whole run can process up to this limit per sample. USETOTAL= # Leave false for Kubernetes Jobs; global ordering is unnecessary for deletion. ORDER_BY_OBJPATH=false diff --git a/docs/cli-reference.md b/docs/cli-reference.md new file mode 100644 index 0000000..3af38d8 --- /dev/null +++ b/docs/cli-reference.md @@ -0,0 +1,385 @@ +# CLI and configuration reference + +`s3gc` collects an object inventory and compares it with ClickHouse references. +For production, follow the [Kubernetes runbook](../deploy/kubernetes/README.md): +collect → dry-run → explicit human approval → delete → verify. This reference +describes the CLI, including options that the Kubernetes renderer does not expose. + +## Main parameters + +Start with these settings for a cleanup. The linked tables describe their +aliases, defaults, and environment variables. + +- [ClickHouse host, cluster, and expected replicas](#clickhouse-connection-and-topology) identify the reference scope and gate a clustered delete. +- [Bucket, object prefix, and disk name](#object-store-and-authentication) identify the object-store scope and the ClickHouse disk to compare. +- [Collection database, table prefix, and samples](#collection-and-inventory) identify the inventory. Use `--collect-only` to build it and `--use-collected` to reuse it. +- [Dry-run, use age, and deletion limits](#candidate-processing-and-deletion) control the review and use phases. + +Boolean environment and explicit-value options accept `true`/`false`, +`yes`/`no`, `on`/`off`, `1`/`0`, and the shorter `y`/`n` and `t`/`f`, ignoring +case. A bare flag such as `--dry-run` sets its value to true. Use its +explicit-value alias, such as `--dry-run-flag false`, to set it false. Where +aliases share a destination, the later CLI assignment wins. An unset optional +value is shown as `none`. An empty string default is shown as `empty`. + +The following tables give every application parameter. The first spelling is +the preferred CLI form where a kebab-case alias exists. All other accepted +aliases are listed in the same cell. Environment variables use the destination +name, which can differ from the flag spelling. String parameters take a string. +Integer parameters take an integer. Defaults shown are CLI defaults, not values +from the Kubernetes example or renderer. + +## ClickHouse connection and topology + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--ch-host`, `--chhost` | `S3GC_CHHOST` / `chhost` | `localhost`. String host. Pin every phase to the same replica, especially with the default node-local inventory. | +| `--ch-port`, `--chport` | `S3GC_CHPORT` / `chport` | `8123`. ClickHouse HTTP port. | +| `--ch-user-name`, `--chuser`, `--chusername` | `S3GC_CHUSER` / `chuser` | `default`. ClickHouse user. | +| `--ch-password`, `--chpass`, `--ch-pass` | `S3GC_CHPASS` / `chpass` | empty. ClickHouse password. Supply securely. | +| `--ch-timeout`, `--chtimeout`, `--send-receive-timeout`, `--ch-send-receive-timeout` | `S3GC_CHTIMEOUT` / `chtimeout` | `1800`. Integer ClickHouse send/receive timeout in seconds. | +| `--cluster-name`, `--cluster`, `--clustername` | `S3GC_CLUSTERNAME` / `clustername` | empty. String cluster whose replicas are queried for references. Empty uses the connected server only. It does not cover independent clusters. | +| `--expected-replicas` | `S3GC_EXPECTED_REPLICAS` / `expected_replicas` | none. Integer expected replica count. With a nonzero value, delete checks the cluster macro and reachable count before use and each sample. It requires a cluster name. Set a positive value for production. | +| `--replica-liveness-recheck` | `S3GC_REPLICA_LIVENESS_RECHECK` / `replica_liveness_recheck` | `false`. Bare boolean flag. For a Replicated auxiliary database, wait past the Keeper session timeout and re-check registered replica liveness before DDL. | +| `--replica-liveness-margin-ms` | `S3GC_REPLICA_LIVENESS_MARGIN_MS` / `replica_liveness_margin_ms` | `1000`. Nonnegative integer milliseconds added to that optional re-check. | + +The Replicated-database preflight is distinct from the delete-time topology +preflight. If ClickHouse cannot report database liveness, or returns no replica +rows, it logs a warning and skips that database check. The Kubernetes delete +entrypoint requires both cluster name and expected replica count. A direct CLI +run does not impose that requirement when the count is unset. + +## Object store and authentication + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--s3-ip`, `--s3ip` | `S3GC_S3IP` / `s3ip` | `127.0.0.1`. String S3-compatible API hostname or IP. | +| `--s3-port`, `--s3port` | `S3GC_S3PORT` / `s3port` | `9001`. API port. | +| `--s3-bucket`, `--s3bucket` | `S3GC_S3BUCKET` / `s3bucket` | `root`. String bucket name. | +| `--s3-path`, `--s3path` | `S3GC_S3PATH` / `s3path` | `data/`. String listing prefix. Empty selects the bucket root. Set the scope consistently across phases. | +| `--s3-disk-name`, `--s3diskname` | `S3GC_S3DISKNAME` / `s3diskname` | `s3`. String ClickHouse disk name used in the reference join and inventory table name. Select the underlying object disk, not its cache disk. | +| `--s3-auth`, `--s3auth` | `S3GC_S3AUTH` / `s3auth` | `static`. `static`, `aws`, or `iam` (case-insensitive). Static uses explicit keys or anonymous access if both are empty. AWS uses boto3. IAM uses MinIO's refreshable workload-identity provider. | +| `--s3-access-key`, `--s3accesskey` | `S3GC_S3ACCESSKEY` / `s3accesskey` | empty. Static access key. Supply with the secret key. | +| `--s3-secret-key`, `--s3secretkey` | `S3GC_S3SECRETKEY` / `s3secretkey` | empty. Static secret key. Supply securely. | +| `--s3-session-token`, `--s3sessiontoken` | `S3GC_S3SESSIONTOKEN` / `s3sessiontoken` | empty. Optional static session token. Requires both static keys. | +| `--s3-profile`, `--s3profile` | `S3GC_S3PROFILE` / `s3profile` | empty. AWS profile name. A profile selects `aws` and conflicts with `iam`. AWS mode rejects explicit S3 keys. | +| `--s3-region`, `--s3region` | `S3GC_S3REGION` / `s3region` | none. Optional string region. AWS mode can use its session region. | +| `--s3-secure`, `--s3secure`. Explicit-value forms: `--s3-secure-flag`, `--s3secureflag` | `S3GC_S3SECURE_FLAG` / `s3secure_flag` | `false`. Boolean HTTPS selection. The current HTTP client is configured with certificate verification disabled. Do not interpret this option as a certificate-verification guarantee. | +| `--s3-ssl-cert-file`, `--s3sslcertfile` | `S3GC_S3SSLCERTFILE` / `s3sslcertfile` | empty. String path assigned to `SSL_CERT_FILE` when secure mode is enabled. See the certificate-verification limitation above. | +| `--s3-connect-timeout` | `S3GC_S3_CONNECT_TIMEOUT` / `s3_connect_timeout` | `15`. Integer S3 connection timeout in seconds. | +| `--s3-read-timeout` | `S3GC_S3_READ_TIMEOUT` / `s3_read_timeout` | `120`. Integer S3 read timeout in seconds. | +| `--s3-retries` | `S3GC_S3_RETRIES` / `s3_retries` | `3`. Integer HTTP retry count for transient S3 errors. | + +Static mode does not automatically read the standard `AWS_ACCESS_KEY_ID` and +`AWS_SECRET_ACCESS_KEY` variables. Supply the corresponding `S3GC_*` values +through a secret manager, or choose an appropriate AWS/IAM mode. Collection +requires listing permission. A dry-run using an existing inventory does not list +S3. A delete requires object-deletion permission. + +## Collection and inventory + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--collect-only`, `--collectonly`. Explicit-value forms: `--collect-only-flag`, `--collectonlyflag` | `S3GC_COLLECTONLY_FLAG` / `collectonly_flag` | `false`. Boolean. List objects and populate the inventory without running the use phase. | +| `--use-collected`, `--usecollected`. Explicit-value forms: `--use-collected-flag`, `--usecollectedflag` | `S3GC_USECOLLECTED_FLAG` / `usecollected_flag` | `false`. Boolean. Skip listing and use the existing inventory. See [Using a collected inventory](#using-a-collected-inventory). | +| `--collect-table-prefix`, `--collecttableprefix` | `S3GC_COLLECTTABLEPREFIX` / `collecttableprefix` | `s3objects_for_`. String prefix for the inventory table. The disk name is appended. A `database.prefix_` form can qualify it. Use a unique prefix per cleanup scope. | +| `--collect-database`, `--collectdatabase` | `S3GC_COLLECTDATABASE` / `collectdatabase` | empty. Optional database for inventory and run log. If the prefix embeds a database, both names must agree. Without either, the connection's current database is used. | +| `--collect-batch-size`, `--collectbatchsize` | `S3GC_COLLECTBATCHSIZE` / `collectbatchsize` | `1024`. Integer listing rows considered per insert batch. Use a positive value. | +| `--collect-total`, `--total`, `--collecttotal`, `--total-num` | `S3GC_TOTAL` / `total` | none. Optional integer target for collected rows. The code subtracts whole batches and stops on exactly zero, so this is not a reliable strict limit when a batch overshoots. | +| `--collect-after`, `--collectafter` | `S3GC_COLLECTAFTER` / `collectafter` | none. Optional string key passed as the listing start-after marker. Collect does not persist a resume marker. | +| `--age-hours`, `--age`, `--hours`, `--collectage`, `--collecthours` | `S3GC_AGE` / `age` | `0`. Integer hours. Collection includes listed objects whose age is at least this value. Separate from the use-phase age floor. | +| `--samples` | `S3GC_SAMPLES` / `samples` | `4`. Integer number of inventory partitions and use-phase samples. Use a positive value and keep it consistent across phases for partition pruning. | +| `--create-database`, `--createdatabase`. Explicit-value forms: `--create-database-flag`, `--createdatabase-flag` | `S3GC_CREATEDATABASE_FLAG` / `createdatabase_flag` | `false`. Boolean. Create the qualified inventory database during collection. A database name must be supplied. | +| `--drop-collecttable`, `--dropcollecttable`. Explicit-value forms: `--drop-collecttable-flag`, `--dropcollecttable-flag` | `S3GC_DROP_COLLECTTABLE_FLAG` / `drop_collecttable_flag` | `false`. Boolean. Drop and recreate the inventory during collection, losing prior inventory and checkpoints. Do not use when resuming a delete. | +| `--keep-data`, `--keepdata`. Explicit-value forms: `--keep-data-flag`, `--keepdataflag` | `S3GC_KEEPDATA_FLAG` / `keepdata_flag` | `false`. Boolean. Preserve the inventory after a successful delete. Without it, a successful non-dry use truncates the table. Dry-run does not truncate it. | + +## Candidate processing and deletion + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--dry-run`, `--dryrun`. Explicit-value forms: `--dry-run-flag`, `--dryrun-flag`, `--dryrunflag` | `S3GC_DRYRUN_FLAG` / `dryrun_flag` | `false`. Boolean. Report candidates without deleting objects. Without `--use-collected`, collection still runs and writes the inventory. | +| `--useage-hours`, `--useage`, `--usehours` | `S3GC_USEAGE` / `useage` | `24`. Integer hours. Use selects inventory rows older than this age. Below 24 is refused for preview and delete except with the development-only override. | +| `--use-after`, `--useafter` | `S3GC_USEAFTER` / `useafter` | none. Optional string. Process inventory keys lexically greater than this marker. | +| `--use-total`, `--usetotal` | `S3GC_USETOTAL` / `usetotal` | none. Optional integer SQL limit **per sample**, not a global run limit. A positive value can bound a trial. Unset it for a full run and verification. | +| `--delete-batch-size`, `--deletebatchsize` | `S3GC_DELETEBATCHSIZE` / `deletebatchsize` | `1000`. Positive integer objects per deletion/checkpoint batch. | +| `--order-by-objpath`. Explicit-value forms: `--order-by-objpath-flag` | `S3GC_ORDER_BY_OBJPATH` / `order_by_objpath` | `false`. Boolean. Order candidates by key within each sample. Sorting can be expensive. | +| `--use-remove-objects`, `--useremoveobjects` | `S3GC_USE_REMOVE_OBJECTS` / `use_remove_objects` | `true`. Boolean. This option requires a value. Use batch S3 deletion when true, per-object deletion when false. The recognized GCS endpoint switches to per-object deletion automatically. | +| `--non-interactive`, `--noninteractive`. Explicit-value forms: `--interactive-flag` | `S3GC_INTERACTIVE_FLAG` / `interactive_flag` | `true`. Boolean controls the CLI prompt. `--non-interactive` sets false. The prompt appears only when both stdin and stdout are terminals. A non-terminal run does not prompt even with the default. The Kubernetes token gate is separate. | +| `--dev-allow-short-useage` | `S3GC_DEV_ALLOW_SHORT_USEAGE` / `dev_allow_short_useage` | `false`. Boolean. This option requires a value. Permits a use age below 24 hours for isolated development fixtures only. Never use for customer or production data. | + +## Using a collected inventory + +`--use-collected` skips the S3 listing and reads an existing inventory table. +Use it after a normal collect or after importing a reviewed candidate list. +The [manual shared-bucket example](#manual-inventory-for-a-bucket-shared-by-independent-clusters) +shows how to prepare that list when independent clusters share a bucket. + +Each use phase checks the inventory against current ClickHouse references for +the configured cluster and disk. It also applies the age filter to the stored +timestamp. It does not refresh S3 metadata or check independent clusters. New +keys are missing from an old inventory, and overwritten keys can retain old +timestamps. A missing or empty inventory causes the run to fail. + +Use the same pinned ClickHouse host and cleanup settings for every phase. The +default inventory table is node-local. Its name is +`` in the chosen database. Start a new scope +with a unique table because collection adds rows to an existing table. + +These examples are for an isolated development environment. Set the scope as +shown in the [README](../README.md#local-setup-and-read-only-preview) and +provide credentials securely. Use the [Kubernetes runbook](../deploy/kubernetes/README.md) +for production. + +```bash +# 1. Collect into a new, unique inventory table. +.venv/bin/python s3gc.py --collect-only --keep-data --verbose + +# 2. Review candidates in that inventory. +.venv/bin/python s3gc.py --use-collected --dry-run --verbose + +# 3. Run from a terminal after review and explicit approval. +.venv/bin/python s3gc.py --use-collected --keep-data --verbose + +# 4. Verify the same inventory, without use-total or use-after limits. +.venv/bin/python s3gc.py --use-collected --dry-run --verbose +``` + +The CLI prompts only when stdin and stdout are terminals. Get explicit human +approval before a non-interactive delete too. Keep the table after delete so +its checkpoints remain available for recovery and verification. If delete +fails, inspect the run log and confirm that the scope and approval still apply +before retrying. Do not re-collect or drop the table during recovery. + +A zero-candidate verification covers only this inventory and its filters. It +does not check objects absent from the inventory or prove the bucket is empty. + +## Manual inventory for a bucket shared by independent clusters + +When different ClickHouse clusters share an overlapping bucket/prefix, use a +manually reviewed inventory. A normal clustered s3gc comparison covers the +configured cluster, not other independent clusters. Putting the full shared +listing directly into the s3gc inventory would let objects referenced only by +another cluster appear orphaned. + +First export the listing and compare it with references from every consumer. +Import only the reviewed candidates into the inventory table. Then use +`--use-collected` to preview, delete, and verify that table without listing S3 +again. The steps below show the complete workflow. + +The outline below uses an AWS-compatible listing, CSV staging, and a dedicated +replicated candidate table. Adapt endpoints, authentication, names, and SQL to +the installation. It requires a complete view of every cluster, replica, disk, +and any other consumer of the shared scope. If that view cannot be established, +stop. Coordinate all consumers so that references cannot change between the +final cross-cluster review and deletion. This coordination is outside s3gc. +Work in a private directory outside Git, use least-privilege credentials, and +keep the raw listing separate from the candidate table. + +### 1. Export the full scoped listing + +For an AWS-compatible endpoint, the following example uses the installed boto3 +dependency and its paginated `list_objects_v2` API. Configure the SDK's +credentials and region outside the script. For another S3-compatible endpoint, +configure the session/endpoint as appropriate. Set `S3GC_RUN_DIR` to an existing +private directory outside the repository, and set `S3GC_S3BUCKET` and +`S3GC_S3PATH` to the reviewed scope. An empty prefix lists the bucket root. +The CSV has one row per object and preserves the object-store size and UTC +last-modified time. Do not fabricate or backdate timestamps: s3gc's age guard +uses the imported value. + +```bash +.venv/bin/python - > "$S3GC_RUN_DIR/listing.csv" <<'PY' +import csv, datetime, os, sys +import boto3 + +writer = csv.writer(sys.stdout) +writer.writerow(['objpath', 'size', 'last_modified']) +s3 = boto3.client('s3') +pages = s3.get_paginator('list_objects_v2').paginate( + Bucket=os.environ['S3GC_S3BUCKET'], Prefix=os.environ['S3GC_S3PATH']) +for page in pages: + for obj in page.get('Contents', []): + modified = obj['LastModified'].astimezone(datetime.timezone.utc) + writer.writerow([obj['Key'], obj['Size'], modified.strftime('%Y-%m-%d %H:%M:%S')]) +PY +``` + +Check that the command succeeded, pagination completed, and the row count, +bucket, prefix, object keys, sizes, and timestamps are credible. The export is +a point-in-time listing, not an atomic snapshot of all consumers. Preserve its +provenance and obtain a new listing if the scope or assumptions change. + +### 2. Stage the listing and collect all references + +The examples use a pre-existing `s3gc` database and a private, per-replica +ClickHouse connection configured in your client. Use unique staging names for +the run. Create staging tables on the chosen host. These are **not** the table +that s3gc will use: + +```sql +CREATE TABLE s3gc.manual_listing + (objpath String, size Int64, last_modified DateTime('UTC')) + ENGINE = MergeTree ORDER BY objpath; +CREATE TABLE s3gc.manual_references (objpath String) + ENGINE = MergeTree ORDER BY objpath; +``` + +Import the listing into staging through the same pinned host: + +```bash +clickhouse-client --host '' \ + --query 'INSERT INTO s3gc.manual_listing FORMAT CSVWithNames' \ + < "$S3GC_RUN_DIR/listing.csv" +``` + +For **each** independent cluster, connect to that cluster and export references +from every replica, using the disk names that map to this bucket/prefix on that +cluster. The names can differ between clusters. For example, after replacing +the placeholders with the cluster's actual names: + +```bash +clickhouse-client --host '' \ + --query "SELECT DISTINCT remote_path AS objpath + FROM clusterAllReplicas('', system.remote_data_paths) + WHERE disk_name IN ('') + SETTINGS skip_unavailable_shards = 0 FORMAT CSVWithNames" \ + > "$S3GC_RUN_DIR/references-cluster-a.csv" +``` + +Check the expected replica count, successful query completion, disk-to-bucket +mapping, and key format for each cluster. Include every relevant disk and +non-ClickHouse consumer in the review. If a replica or consumer is unavailable, +the exports are incomplete. Stop rather than treating missing references as +orphans. Import each verified reference CSV into `manual_references` on the +pinned staging host, using a distinct file per source: + +```bash +clickhouse-client --host '' \ + --query 'INSERT INTO s3gc.manual_references FORMAT CSVWithNames' \ + < "$S3GC_RUN_DIR/references-cluster-a.csv" +``` + +Repeat for every cluster. Add references from other consumers in the same +one-column `objpath` format. Confirm that keys are comparable to the S3 listing +and verify import counts. Duplicate references are harmless for the anti-join. + +### 3. Review candidates and load a replicated inventory + +Generate a candidate CSV from the staging tables. Only rows absent from the +union of all reference exports enter it. Preserve listing metadata and mark rows +active. Review keys, counts, total bytes, timestamps, scope, and the source +exports before importing. Exclude anything whose ownership is uncertain. + +```bash +clickhouse-client --host '' \ + --query "SELECT l.objpath, l.size, l.last_modified, true AS active + FROM s3gc.manual_listing AS l + LEFT ANTI JOIN s3gc.manual_references AS r ON l.objpath = r.objpath + FORMAT CSVWithNames" > "$S3GC_RUN_DIR/reviewed-candidates.csv" +``` + +Create a **new, empty** auxiliary table with the schema and replacement key +expected by s3gc. This example uses `S3GC_COLLECTDATABASE=s3gc`, +`S3GC_COLLECTTABLEPREFIX=manual_cleanup_`, `S3GC_S3DISKNAME=s3`, and +`S3GC_SAMPLES=4`. Use a unique Keeper path and a cluster with correctly +configured `{shard}` and `{replica}` macros. Review the DDL for the installation. +The table engine replicates data within a shard. It does not automatically +share rows across independent shards or clusters. Table replication is separate +from a Replicated database's DDL replication. See ClickHouse's +[replicated table engine documentation](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replication). + +```sql +CREATE TABLE s3gc.manual_cleanup_s3 ON CLUSTER '' +( + objpath String, + size Int64, + last_modified DateTime('UTC'), + active Bool +) +ENGINE = ReplicatedReplacingMergeTree( + '/clickhouse/tables/{shard}/s3gc_manual_cleanup_s3', '{replica}') +PARTITION BY CRC32(objpath) % 4 +ORDER BY objpath; +``` + +Import only the reviewed candidate CSV on the pinned execution replica: + +```bash +clickhouse-client --host '' \ + --query 'INSERT INTO s3gc.manual_cleanup_s3 FORMAT CSVWithNames' \ + < "$S3GC_RUN_DIR/reviewed-candidates.csv" +``` + +Verify the imported keys and metadata against the reviewed CSV, including count +and total bytes, and confirm replication has caught up where the table will be read. Replication is +asynchronous. Pinning all s3gc phases to the import replica avoids assuming an +immediate copy elsewhere. A partial import or an empty candidate table is not a +successful cleanup: resolve it before proceeding. The normal s3gc collector +creates a node-local table. It does not create this replicated table for you. + +### 4. Preview, approve, delete, and verify + +Configure the matching database, prefix, disk, bucket, prefix, execution cluster, +expected replicas, sample count, and age window. Refresh and compare references +from **all** consumers immediately before delete, and verify that candidate +objects still have the reviewed metadata. Maintain the agreed period in which +references cannot change and candidate keys cannot be overwritten or reused. +Then obtain explicit approval for the reviewed candidate set. If any +cross-cluster reference, candidate metadata, or scope changes, stop and rebuild +and re-review the candidate inventory. s3gc rechecks only the configured +cluster during use. It does not recheck other clusters or S3 metadata for you. + +Run the Kubernetes `dry-run`, approved `delete`, and `dry-run` verification +phases against the existing table. They already select `--usecollected`. Do not +run the Kubernetes collect phase or set the fresh-run drop flag on this table. +Keep the same pinned host and inventory for recovery and verification. A zero +result applies to the reviewed imported inventory and current configured-cluster +filters, not to a newly generated bucket listing. Preserve review evidence and +clean up private staging artifacts according to your retention policy. + +## Advanced: configuration files and precedence + +Each parameter has a parser destination, shown below as its configuration key. +Use that key in a JSON or YAML file passed with `--cfg`, or set the listed +`S3GC_*` environment variable. For example, this non-secret JSON file selects a +replica and enables a preview: + +```json +{ + "chhost": "replica.example.invalid", + "collectdatabase": "s3gc", + "collecttableprefix": "example_", + "dryrun_flag": true +} +``` + +Pass `--cfg /path/to/config.json`. The parser starts with defaults, applies +environment values, then processes command-line arguments in order. A `--cfg` +file is applied at its position on the command line: it can override an earlier +CLI option, and a later CLI option can override the file. Multiple files and +options follow the same ordering. `S3GC_CFG` can also select a config file. +The parser applies other environment variables after that environment-selected +file. Keep secrets outside config files and shell history. + +`--help` (`-h`) prints CLI help. `--print_config` prints the effective parsed +configuration and exits. **its output can contain credentials**, so do not +share or commit it. `--list-options` (`--listoptions`) prints environment-style +options and redacts configured credentials. These inspection options do not +contact ClickHouse or S3. `--cfg` has environment variable `S3GC_CFG`. The help +and print-config actions have no documented environment equivalent. The +list-options variable is `S3GC_LISTOPTIONS` (boolean, default `false`). + + +## Advanced: logging and inspection + +| Option and aliases | Environment / config key | Default and purpose | +| --- | --- | --- | +| `--verbose`. Explicit-value forms: `--verbose-flag`, `--verboseflag` | `S3GC_VERBOSE_FLAG` / `verbose_flag` | `false`. Boolean. Enable informational logs. | +| `--debug`. Explicit-value forms: `--debug-flag`, `--debugflag` | `S3GC_DEBUG_FLAG` / `debug_flag` | `false`. Boolean. Enable more detailed logs, including per-object output. | +| `--silent`. Explicit-value forms: `--silent-flag`, `--silentflag` | `S3GC_SILENT_FLAG` / `silent_flag` | `false`. Boolean. Restrict console logging to critical messages. | +| `--run-log`, `--runlog` | `S3GC_RUNLOG_FLAG` / `runlog_flag` | `true`. Boolean. This option requires a value. Write durable run events alongside the inventory. Logging failure warns and falls back to stdout. | +| `--run-id`, `--runid` | `S3GC_RUNID` / `runid` | empty. String identifier for run-log rows. Empty generates a timestamped ID. Kubernetes supplies the Job name. | +| `--list-options`, `--listoptions` | `S3GC_LISTOPTIONS` / `listoptions` | `false`. Boolean. Print environment-style options with credentials redacted and exit. | +| `--cfg` | `S3GC_CFG` / `cfg` | none. Configuration file path. Place before CLI overrides. | +| `--help`, `-h` | none | Print help and exit. | +| `--print_config` | none | Print effective configuration and exit. Output may contain secrets. | diff --git a/docs/plans/cli-parameter-documentation.md b/docs/plans/cli-parameter-documentation.md new file mode 100644 index 0000000..04779e0 --- /dev/null +++ b/docs/plans/cli-parameter-documentation.md @@ -0,0 +1,164 @@ +# Plan: complete CLI parameter documentation + +Status: implemented on `docs/cli-parameter-reference`. +Planning branch: `docs/cli-reference-plan`. +Proposed implementation branch: `docs/cli-parameter-reference`. + +## Problem and intent + +The README introduces common settings, but operators need a complete reference +that explains each parameter and how parameters work together across a cleanup. +Complete the first documentation item in `TODO.md`, with particular attention to +`--usecollected` and the separation of collection, review, deletion, and verification. + +This is an editorial change. Preserve CLI names, aliases, defaults, environment +variables, runtime behavior, deployment settings, and deletion safeguards. Exclude +the deleted manifest feature. Do not fix runtime defects as part of this work. + +## Reference style + +Use Altinity's [clickhouse-backup configurable parameters](https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md#configurable-parameters) +as a presentation reference: group related settings and put environment names, +value meanings, defaults, and constraints near their explanations. Its +[CLI manual](https://github.com/Altinity/clickhouse-backup/blob/master/Manual.md) +also puts usage, aliases, and option descriptions together. References inspected +on 2026-10-27. + +Adapt that structure to s3gc's existing interfaces. Distinguish illustrative +values from actual defaults and provide operational explanations beyond help text. + +## Deliverables + +| File | Planned change | +| --- | --- | +| `docs/cli-reference.md` | Complete parameter reference, configuration rules, phase walkthrough, and examples. | +| `README.md` | Link the reference beside local setup and introduce the collect/use distinction briefly. | +| `deploy/kubernetes/README.md` | Cross-link the reference and explain how entrypoint phases select CLI flags; retain the production runbook. | +| `deploy/kubernetes/example.env` | Clarify that the documented use-phase limit applies to each sample. | +| `CHANGELOG.md` | Record the completed documentation under Unreleased. | +| `TODO.md` | Remove the completed documentation item only after its acceptance criteria are met. | + +## Implementation sequence + +1. Read `CLAUDE.md`, `CHANGELOG.md`, `TODO.md`, and the existing operator guides. + Inventory every argument declaration in `s3gc.py`, including aliases, shared + destinations, explicit-value boolean forms, and parser-provided help/config + options. Consult runtime consumers, the entrypoint, renderer, and template to + distinguish declared defaults from effective behavior. Use local sources; + consult the clickhouse-reference skill if a ClickHouse claim needs verification. +2. Build the complete reference, grouped into ClickHouse connection/topology, + object storage/authentication, collection/inventory, candidate processing and + deletion, timeouts/retries, and logging/configuration/development controls. + Give each setting a stable anchor. For each setting document purpose, all + aliases, value type and accepted values, units, actual default, exact + `S3GC_*` variable, affected phases, and interactions. State explicitly when a + value has no default or a parser utility has no environment equivalent. + Document enforced restrictions separately from operational recommendations. +3. Explain configuration files through `--cfg`, configuration destination names, + booleans, and precedence among defaults, environment, files, and CLI arguments. + Verify precedence and shared-destination alias behavior with the pinned + jsonargparse version using offline parser probes; do not assume precedence + matches clickhouse-backup. Include a non-secret configuration example and + explain `--help`, `--print_config`, and the redacted `--list-options` output. +4. Write the collect/use explanation and examples described below. Explain + defaults that can start collection and deletion in one invocation; distinguish + auxiliary-table writes during preview from S3 object deletion. Check CLI + confirmation behavior separately from the Kubernetes confirmation token. +5. Add README/runbook links, remove unnecessary duplicate explanations, and + reconcile any conflicting guidance against the implementation. Preserve the + production approval sequence and existing runbook requirements. Finish the + changelog and TODO edits after validation. + +## Required explanation of `--usecollected` + +- Document `--usecollected`, its `--use-collected` alias, the explicit-value forms + `--usecollectedflag` and `--use-collected-flag`, default `false`, and the exact + variable `S3GC_USECOLLECTED_FLAG`. +- Explain that it skips collection and reads the existing auxiliary inventory. + It re-evaluates references and the age filter during the use phase; the + inventory is not a saved final deletion decision and is not refreshed by an + S3 listing. New keys are absent until another collect, and overwritten keys + can retain stale metadata. +- Show the distinction between `--collect-only`, `--use-collected --dry-run`, + and a delete using the collected inventory. Explain the effect of omitting + `--use-collected`, and the interaction with `--keep-data`, tombstones, table + truncation, `--drop-collecttable`, and retrying an interrupted delete. +- Explain how the selected database, table prefix, disk, and pinned ClickHouse + host identify the inventory. Cover missing/empty inventory errors and why + every phase must use the same replica and cleanup scope. +- Keep collection age (`--age`) separate from use age (`--useage`), collection + limits/markers separate from use limits/markers, and document sampling and + ordering constraints from the code. Describe actual limit behavior rather + than promising a global cap without checking it. +- Map Kubernetes collect, dry-run, delete, and dev-automation phases to their + actual entrypoint arguments. Keep renderer keys distinct from `S3GC_*` + variables; do not imply all CLI options are exposed by the renderer. + +## Manual inventory for a shared bucket + +Document the preferred manual workflow when independent clusters share an +overlapping bucket/prefix: export the full scoped listing to CSV, gather live +references from every participating cluster and replica, and exclude those +references before approving candidates. Include example listing, CSV import, +reference-export, and comparison commands. A cluster's own replicas can normally +be covered with `--cluster-name`; independent clusters require this additional +cross-cluster review because s3gc checks only its configured reference scope. + +Import the reviewed candidates into a dedicated replicated auxiliary table with +the schema, naming, replacement semantics, and partitioning expected by s3gc. +Distinguish table data replication from a Replicated database's DDL replication, +and keep the execution host pinned. The full raw listing belongs in a separate +staging table so unrelated objects cannot become deletion candidates. Explain +metadata provenance, complete reference exports, replication readiness, and +freshness checks before delete. Use `--use-collected` for preview, approved +delete, recovery, and verification; do not re-collect into the reviewed table. + +## Examples and safety requirements + +Provide one consistent placeholder configuration and separate examples for +collect, dry-run, approved delete, and verification. Prefer existing kebab-case +aliases. Preserve inventory through the delete/verify sequence as required by +the implementation. State prerequisites next to the commands, including the +same host and scope, cluster and expected replicas, and use age of at least +24 hours. Keep human review and explicit approval before deletion; explain the +actual confirmation mechanism for each execution path. + +Include an interrupted-delete recovery example using the same inventory and a +brief contrast with restarting collection. Verification must explain what a +zero-candidate result establishes for that inventory and time window. Keep +production commands separate from the development-only age override. Use +placeholders and externally injected credentials throughout. Do not execute +any live example while writing or validating the documentation. + +## Acceptance criteria and validation + +- Every current CLI argument and alias appears in the reference, with the fields + above checked against parser declarations and runtime consumers. Review an + inventory-to-documentation checklist to catch omissions, including boolean + destinations ending in `_FLAG` and parser utility options. +- Configuration precedence and representative boolean/alias combinations are + demonstrated with isolated offline probes of the installed pinned parser. + Clear inherited `S3GC_*` values for probes and use dummy configuration only; + do not print the real environment or call `main()`. +- A reader can explain when `--use-collected` is needed, what it reuses, what + gets checked again, and how to preserve the inventory for recovery and verify. +- The shared-bucket example covers full CSV listing/import, all-cluster reference + review, and a replicated candidate table. It states that `--use-collected` + does not discover other clusters or make an unfiltered shared listing safe. +- All four workflow examples use existing options, consistent scope, and the + existing approval, age, topology, and checkpoint safeguards. Kubernetes + explanations agree with the renderer, template, and entrypoint. +- Relative links and anchors resolve, Markdown tables/code blocks render, and + `git diff --check` passes. Run help/config inspection offline only. This + editorial change needs no new runtime tests or live-system validation. + If executable files enter scope, split that work out and apply the repository's + spec-first TDD and required offline pytest/render/Kubeconform checks. +- Only the planned documentation files change; no secrets, customer artifacts, + dependencies, generated configuration dumps, or rendered manifests enter Git. + +## Handoff + +The planning stage ended when implementation was requested. The implementation +branch is `docs/cli-parameter-reference`; carry this plan with the documentation. +Keep the TODO item open until implementation is done. Publishing a PR or merging +is outside this documentation implementation request. From 9b3e0629c6b5b4184f3b25b7b56029291f4d01fd Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Tue, 27 Oct 2026 20:04:11 +0100 Subject: [PATCH 2/3] docs: simplify guidance and align cleanup workflows --- AGENTS.md | 17 +-- CHANGELOG.md | 16 +-- CLAUDE.md | 76 +++++------ README.md | 80 +++++------ TODO.md | 31 ++--- deploy/kubernetes/README.md | 155 ++++++++++++++-------- deploy/kubernetes/example.env | 36 +++-- docs/cli-reference.md | 44 ++++-- docs/plans/cli-parameter-documentation.md | 4 + skills/altinity-clickhouse-s3gc/SKILL.md | 125 +++++++++-------- tests/dev_cluster_e2e_spec.md | 5 + 11 files changed, 326 insertions(+), 263 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f834b3c..b7cea6d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,9 +2,8 @@ Read `CLAUDE.md`, `CHANGELOG.md`, and `TODO.md` before making substantive changes in this repository. `CLAUDE.md` is the full contributor guide and the -primary source of truth; the changelog provides recent historical context and -the TODO file records possible, interesting, and deliberately deferred -improvements. +primary source of truth. The changelog explains completed changes, and the TODO +file records deferred work. This file is intentionally short so agent tooling can find the critical rules quickly, then defer to `CLAUDE.md` for complete repository guidance. @@ -23,16 +22,18 @@ quickly, then defer to `CLAUDE.md` for complete repository guidance. answer that, run Spec Intake first (see `CLAUDE.md`). Then use test-driven development to satisfy each criterion. Run the relevant offline pytest suite and Kubernetes renderer/manifest dry-run checks described in - `CLAUDE.md`. Automated tests must not contact live ClickHouse or object - storage, or delete objects. + `CLAUDE.md`. Purely editorial changes are exempt from spec and TDD. + Default tests must stay offline and must not delete objects. Tests marked + `dev_cluster` require explicit opt-in and an isolated development environment. + They never run in CI. 4. **No secrets or customer data in Git.** Never commit credentials, customer configuration, target-cluster details, or rendered customer manifests. Use Kubernetes Secrets or workload identity for production credentials. 5. **Keep deployments immutable and least-privileged.** Preserve digest-pinned images, non-root/read-only container settings, and renderer validation. -6. **Keep dependencies deliberate.** Production dependencies belong in - `requirements.txt`; testing-only dependencies belong in - `requirements-dev.txt`. Do not add either casually. +6. **Keep dependencies deliberate.** Pin production dependencies in + `requirements.txt` and testing-only dependencies in `requirements-dev.txt`. + Add or update one only when the task needs it, and verify the workflow. ## Working rule diff --git a/CHANGELOG.md b/CHANGELOG.md index d2f0051..ecd5510 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,16 +2,12 @@ Notable changes to `s3gc`, newest first. -This file records **what changed and why**, with enough context that someone -picking the repository up later — human or agent — can tell a deliberate design -decision from an accident. Defects found by running the tool against real -clusters carry their evidence, because the reasoning is usually the expensive -part to reconstruct. +This file records **what changed and why**. Preserve the evidence for defects +found on real clusters so later readers can understand the decisions. Format loosely follows [Keep a Changelog](https://keepachangelog.com/). -Customer names, cluster identifiers and credentials never appear here (see -`CLAUDE.md`, rule 3); findings are described in terms of the behaviour they -expose. +Keep customer names, cluster identifiers, and credentials out of this file. +See `CLAUDE.md`, rule 4. Describe findings by the behavior they expose. ## [Unreleased] @@ -23,6 +19,10 @@ expose. shared-bucket workflow, because a single configured cluster cannot identify references held by independent clusters. Clarified that `USETOTAL` applies per sample rather than as a global Job limit. +- Simplified current operator and contributor guidance while preserving release + history. Aligned logging names, listing permissions, inventory recovery, + preflight limitations, and Job retry wording. Connected the packaged skill + to the manual shared-bucket workflow and clarified its table-creation examples. ## [0.8.0] - 2026-10-25 diff --git a/CLAUDE.md b/CLAUDE.md index 090591c..b782cce 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,35 +1,34 @@ # Contributor guide — s3gc -`s3gc` is a Python command-line tool and Kubernetes Job workflow for finding -and deleting orphaned objects from ClickHouse S3 disks and compatible object -storage. It is an operationally destructive tool: quality is held by offline -tests, explicit delete controls, and conservative deployment defaults. +`s3gc` finds and deletes orphaned objects from ClickHouse S3 disks and compatible +object storage. It provides a Python CLI and a Kubernetes Job workflow. Deletion +is irreversible, so preserve the offline tests and operational safeguards. ## Hard rules -1. **The deletion lifecycle is non-negotiable.** The normal production flow is +1. **Preserve the deletion lifecycle.** The normal production flow is `collect → dry-run → explicit customer approval → delete → verify`. Preserve the delete confirmation token, ClickHouse cluster and expected-replica - preflight, deletion checkpoints, and the rule that a failed delete job does - not retry automatically. A behavior change in this path needs regression - tests and matching README/deployment documentation. + preflight, deletion checkpoints, and the rule that a failed delete Job + is not rerun automatically. A behavior change in this path needs regression + tests and matching README and deployment documentation. Job reruns are + separate from HTTP retries within a running process. 2. **Spec-first TDD is mandatory.** For every new feature, bug fix, or material operational behaviour change, start with a short spec in the - commit that begins the change: one or two sentences on the problem or - intent, plus a short bullet list of acceptance criteria — what must be true - when it's done. If the request doesn't already answer these, run Spec - Intake first (see below). Then use TDD: for each acceptance criterion, - write the failing test that covers it before making it pass. Add a - regression test for every defect. Purely editorial changes are exempt from - both the spec and TDD. + commit that begins the change. State the problem in one or two sentences + and list the acceptance criteria. Run Spec Intake below if details are + missing. For each criterion, write a failing test before implementing the + change. Add a regression test for every defect. Purely editorial changes + are exempt from both the spec and TDD. 3. **Tests stay offline by default.** Install both requirements files, then - run `pytest -v` for relevant changes. Tests must use fakes, local - subprocesses, and manifest dry-runs; they must not contact live ClickHouse, - S3-compatible storage, or delete objects. The `dev_cluster` marker is - explicitly environment-dependent and never runs in CI. Do not add live - credentials or a live-delete test path to the default suite. + run the relevant offline suite with `-m "not dev_cluster"`. Default tests + use fakes, local subprocesses, and manifest dry-runs. They must not contact + live ClickHouse or object storage, or delete objects. Tests marked + `dev_cluster` require explicit opt-in and an isolated development environment. + They never run in CI. Do not add live credentials or live-delete paths to + the default suite. 4. **No secrets or customer artifacts in Git.** Do not commit S3 keys, ClickHouse passwords, customer `.env` files, rendered customer manifests, @@ -40,19 +39,19 @@ tests, explicit delete controls, and conservative deployment defaults. 5. **Kubernetes deployment stays immutable and least-privileged.** `deploy/kubernetes/render.py` must continue to reject unpinned images and invalid phase/confirmation input. Use image digests, never mutable tags. - Preserve the Job template's non-root user and read-only root filesystem; - do not embed credentials in the image or manifest. + Preserve the Job template's non-root user and read-only root filesystem. + Do not embed credentials in the image or manifest. 6. **Dependencies are deliberate and reproducible.** Runtime dependencies are - pinned in `requirements.txt`; test-only dependencies are pinned in + pinned in `requirements.txt`. Test-only dependencies are pinned in `requirements-dev.txt`. Use both files when preparing a development or CI environment. Add or update a dependency only when it is necessary for the requested capability, and test the resulting workflow. -7. **Read project history and backlog before substantive changes.** Review - `CHANGELOG.md` for recent behaviour and operational evidence, then `TODO.md` - for possible, interesting, and deliberately deferred improvements. Do not - treat a TODO item as already implemented or use the changelog as a backlog. +7. **Read the history and backlog before substantive changes.** Read + `CHANGELOG.md` for behavior changes and operational evidence, then `TODO.md` + for deferred work. A TODO item is not implemented. The changelog records + completed work, not the backlog. ## Spec Intake @@ -73,15 +72,16 @@ already answers these, skip straight to the spec. | Path | Purpose | | --- | --- | | `s3gc.py` | CLI arguments, ClickHouse inventory/anti-join, S3 collection and deletion, safety preflight, and checkpoints. | +| `docs/cli-reference.md` | Parameter details, configuration, phase examples, and manual shared-bucket inventory. | | `tests/` | Pytest regression tests using fakes and local subprocesses. | | `docker/Dockerfile` | Minimal Python 3.11 production image. | | `docker/kubernetes-entrypoint.sh` | Phase dispatcher and delete/dev-automation confirmation gate. | | `deploy/kubernetes/render.py` | Validates a non-secret environment file and renders the Job manifest. | | `deploy/kubernetes/job.yaml.tmpl` | Kubernetes Job template with security context and environment wiring. | -| `deploy/kubernetes/example.env` | Non-secret rendering example; copy it outside the repository for a real run. | +| `deploy/kubernetes/example.env` | Non-secret rendering example. Copy it outside the repository for a real run. | | `.github/workflows/container.yml` | CI test, render, manifest validation, and container publication workflow. | -| `CHANGELOG.md` | Shipped behaviour and **why**, including evidence for defects found in production use. Update it in the same change as any behaviour, safety, or deployment change. | -| `TODO.md` | Possible, interesting, and deliberately deferred engineering or operational improvements. Move completed work to the changelog when it lands. | +| `CHANGELOG.md` | Completed changes and their reasons, including production evidence. Update it with behavior, safety, and deployment changes. | +| `TODO.md` | Deferred engineering and operational work. Move completed items to the changelog. | ## Required checks @@ -109,16 +109,12 @@ percentage until coverage tooling and an enforceable threshold are introduced. - Keep the command-line and `S3GC_*` environment interfaces compatible unless the task explicitly authorizes a breaking operational change. -- Record behaviour, safety and deployment changes in `CHANGELOG.md` as part of - the same change. Write down *why*, and keep the evidence for defects found in - production — the reasoning is the expensive part to reconstruct later. Never - put customer names, cluster identifiers or credentials there. -- Record possible, interesting, or pending engineering and operational - improvements in `TODO.md`, not in the changelog. Remove or update the TODO - item when the work lands. -- Treat the renderer, entrypoint, README, and Kubernetes guide as part of the - same operator-facing contract. Update the affected documentation in the same - change as an operational behavior change. +- Record behavior, safety, and deployment changes in `CHANGELOG.md` with the + change itself. Explain why and preserve production evidence without customer + names, cluster identifiers, or credentials. +- Put deferred work in `TODO.md`. Remove or update the item when it lands. +- Keep the renderer, entrypoint, README, CLI reference, and Kubernetes guide in + agreement. Update the affected docs with each operational behavior change. - Surface out-of-scope safety defects rather than silently changing them. State the file and risk, and defer the fix unless it is necessary to keep the current task safe. diff --git a/README.md b/README.md index 59e6089..16e6f6c 100644 --- a/README.md +++ b/README.md @@ -11,40 +11,37 @@ For customer and production work, use the Kubernetes runbook: collect → dry-run → explicit human approval → delete → verify ``` -Read the [Kubernetes Job runner](deploy/kubernetes/README.md) before you run a -cleanup. It is the authoritative production procedure. +Read the [Kubernetes Job runner](deploy/kubernetes/README.md) before a production +cleanup. Use the [CLI reference](docs/cli-reference.md) for parameter details. ## Safety boundary Deleting an object is irreversible. `s3gc` preserves the following controls: - A delete Job needs the `DELETE_ORPHANS` confirmation token. -- A clustered delete checks the configured cluster and expected replica count - before it calls S3. -- When the auxiliary database uses `ENGINE = Replicated`, every phase checks - that all of its replicas are active in Keeper before it runs any DDL. With - `S3GC_REPLICA_LIVENESS_RECHECK=true` it checks again after the Keeper - session timeout (about 30 s, plus `S3GC_REPLICA_LIVENESS_MARGIN_MS`), to - catch a replica that crashed just before the run. A down replica would otherwise stall - each `CREATE TABLE` for `distributed_ddl_task_timeout` and surface only as - `IncompleteRead`. +- A production delete checks the configured cluster and expected replica count + before deleting objects and before each sample. +- Each phase checks replica liveness before DDL in a Replicated auxiliary + database. If ClickHouse cannot report liveness, the tool warns and skips that + check. See the [preflight options](docs/cli-reference.md#clickhouse-connection-and-topology) + for the optional delayed re-check. - A failed delete Job does not retry automatically. Confirmed batches remain checkpointed, so a replacement Job can resume with the same collection-table prefix. -- `USEAGE` must be at least 24 hours outside development automation. This age - window protects a part that S3 has received before ClickHouse registers it. +- Set `USEAGE_HOURS` to at least 24 in the renderer, or `S3GC_USEAGE` for a + direct run. This window protects objects uploaded before ClickHouse registers + their paths. Shorter windows are for isolated development fixtures only. - The Kubernetes renderer requires a digest-pinned image and keeps credentials outside the manifest. -For an installation-wide cleanup, collect the parent prefix that holds every -replica's objects and set the cluster name, so the comparison covers all -replicas. A single-replica prefix misses the other replicas' orphans. Only -collect a parent prefix that holds one `S3DISKNAME`'s objects — the orphan -check matches by disk name, so another disk's objects under the same prefix -would look orphaned. +For a cleanup across one installation, collect the parent prefix that holds +every replica's objects and set the cluster name. The comparison then covers +all replicas. That prefix must contain objects for only the selected disk. +Objects from another disk could otherwise appear orphaned. Use a unique collection-table prefix for each bucket and prefix. Never commit credentials, rendered manifests, target-cluster details, or run output. + If independent clusters share an overlapping bucket or prefix, review references from every consumer before importing a candidate inventory. The [shared-bucket workflow](docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters) @@ -81,8 +78,11 @@ export S3GC_COLLECTTABLEPREFIX='s3gc_example_' export S3GC_USEAGE=24 ``` -Inject passwords and keys from a secret manager or your shell. Do not save them -in a file or command history. Then run a preview: +Provide passwords and keys through a secret manager or a private runtime +environment. Keep them out of shell history and repository files. Create the +auxiliary database and grants first, as described in the +[runbook](deploy/kubernetes/README.md#3-provide-credentials-and-identity). +Then run a preview: ```bash .venv/bin/python s3gc.py --verbose --dry-run @@ -102,19 +102,20 @@ Choose one S3 authentication mode with `S3GC_S3AUTH`: | Mode | Credentials | Use it for | | --- | --- | --- | -| `static` | `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`; optional session token | explicit credentials from a secret manager | -| `aws` | boto3 chain; optional `S3GC_S3PROFILE` | AWS SSO or a named workstation profile | +| `static` | `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`, with an optional session token | explicit credentials from a secret manager | +| `aws` | boto3 chain, with optional `S3GC_S3PROFILE` | AWS SSO or a named workstation profile | | `iam` | MinIO workload-identity provider | EKS IRSA, EC2 instance profiles, or ECS task roles | Static mode does not read `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY`. Pass those values as `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`. -`S3GC_S3PROFILE` selects `aws`; the tool rejects contradictory settings. An -`aws` preview still needs `s3:ListBucket` for the configured bucket and prefix. +`S3GC_S3PROFILE` selects `aws`. The tool rejects conflicting settings. +Collection needs `s3:ListBucket`, including a preview that collects first. +A preview with `--use-collected` reads the inventory without listing S3. GCS uses HMAC interoperability keys and usually names the object disk `gcs`. It does not support S3 batch deletion, so `s3gc` falls back to slower -per-object deletion. Set `S3GC_S3DISKNAME=gcs`; do not use a `*_cache` disk as +per-object deletion. Set `S3GC_S3DISKNAME=gcs`. Do not use a `*_cache` disk as the reference scope. ## Kubernetes deployment @@ -132,7 +133,7 @@ Start with these links: Released images are public at `ghcr.io/altinity/s3gc`. Always use a digest: ```bash -docker pull ghcr.io/altinity/s3gc@sha256: +docker pull 'ghcr.io/altinity/s3gc@sha256:' ``` CI publishes the exact `IMAGE=` value in its job summary. The renderer rejects @@ -145,17 +146,10 @@ By default, each run writes structured events to starts, collect progress, deletion checkpoints, totals, warnings, and errors. It outlives the Job and rotated pod logs. -Query one run on the same replica-pinned ClickHouse host: - -```sql -SELECT event_time, phase, event, objects, bytes, message -FROM ._log -WHERE run_id = '' -ORDER BY event_time; -``` - -Set `RUNLOG=false` only when stdout is an adequate record. A run-log failure -falls back to stdout and does not stop a cleanup. +See the [runbook](deploy/kubernetes/README.md#durable-run-history) for the query +to inspect a run. Disable durable logging only when stdout is an adequate record. +Use `RUNLOG=false` in renderer input or `S3GC_RUNLOG_FLAG=false` for a direct run. +A run-log failure falls back to stdout and does not stop the cleanup. ## AI-agent skill @@ -178,12 +172,10 @@ approve the reviewed dry-run result. `CLAUDE.md` (full guide) and `AGENTS.md` (short pointer) tell AI coding agents how to work here. For a new feature, bug fix, or behaviour change: -1. **Spec Intake** — if the request is underspecified, the agent asks what - should happen instead, how to trigger it, whether it touches the delete - lifecycle, and known edge cases. -2. **Spec** — a short problem statement and acceptance criteria, in the - commit that starts the change. -3. **TDD** — a failing test per acceptance criterion, then made to pass. +1. **Spec Intake:** ask for missing details, including expected behavior, + reproduction steps, safety impact, and edge cases. +2. **Spec:** put the problem and acceptance criteria in the starting commit. +3. **TDD:** write a failing test for each criterion, then make it pass. Editorial changes (docs, comments) skip this. See `CLAUDE.md` for the full rules. diff --git a/TODO.md b/TODO.md index 84f23e5..1368f22 100644 --- a/TODO.md +++ b/TODO.md @@ -1,28 +1,23 @@ # TODO -Possible, interesting, and deliberately deferred engineering or operational -improvements. Completed behaviour belongs in `CHANGELOG.md`; this file is the -forward-looking backlog. +Deferred engineering and operational work. Record completed changes in +`CHANGELOG.md` and remove or update the item here. -- [ ] Record collect progress so a crashed `--collectonly` run can resume. - `--collectafter` already starts listing after a given key, but nothing - records the last key listed, so after a crash the operator cannot tell what - to pass and restarts from the beginning. Write the last listed key - durably (for example in the run log) at each progress step. +- [ ] Record the last listed key so an interrupted collect can resume. + `--collect-after` can start after a key, but collect does not save that key. + Write it durably at each progress step, for example in the run log. - [ ] Add an opt-in `dev_cluster` GCS end-to-end test. It must require explicit environment configuration, create an isolated tiered-policy fixture, run `collect → dry-run → delete → verify`, clean up ClickHouse and object-storage fixtures, and remain excluded from CI. -- [ ] Require the `Container / test` GitHub Actions check before pull-request - merges in the repository branch-protection settings. `master` has no branch - protection or ruleset today; this needs a repository admin. +- [ ] Have a repository admin require the `Container / test` GitHub Actions + check before pull-request merges. This item was recorded when `master` had + no branch protection or ruleset. Verify the current settings before changing them. - [ ] Surface a DDL `TIMEOUT_EXCEEDED` (159) instead of `IncompleteRead`. The server sends `200 OK` before a distributed DDL times out, so the HTTP client - hides the cause. The Replicated-database preflight covers a replica that is - already down; a replica that dies after the preflight still surfaces as + hides the cause. The Replicated-database preflight catches a replica already + reported down. A replica that dies after the check can still cause `IncompleteRead` after `distributed_ddl_task_timeout`. -- [ ] Consider a `TTL` on the run-log table. Volume is a few hundred rows per - run so it is not urgent, but it grows without bound across many cleanups. - The run log is the durable record of what was deleted, so a TTL works - against its purpose: keep it off by default, or make it long (a year or - more). +- [ ] Consider a run-log `TTL`. Each run adds a few hundred rows, and the table + grows across cleanups. It is the durable deletion record, so keep any TTL + disabled by default or retain at least a year of history. diff --git a/deploy/kubernetes/README.md b/deploy/kubernetes/README.md index 8195224..3bbac70 100644 --- a/deploy/kubernetes/README.md +++ b/deploy/kubernetes/README.md @@ -13,8 +13,8 @@ python3.11 -m venv .venv .venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt ``` -`render.py` uses only the Python standard library. The runbook still uses -`.venv/bin/python` so every command runs with the supported Python version. +Use `.venv/bin/python` for these commands. The renderer itself needs only the +Python standard library. ## Runbook @@ -45,35 +45,43 @@ contains no credentials. ### 2. Confirm the cleanup scope -Set the values in `s3gc.env`, then check these five items before rendering: - -1. Set `CHHOST` to one per-replica ClickHouse Service and use the same host for every phase. The inventory table is node-local; a load-balanced Service can send a later phase to a replica without that table. -2. Set the exact bucket, prefix, and underlying object-disk name. `S3PATH` may be empty. GCS commonly uses `gcs`; never use a `*_cache` disk. -3. Set the actual `CLUSTERNAME` and `EXPECTED_REPLICAS` for clustered cleanup. Delete fails closed if this preflight does not match. -4. Set `COLLECTDATABASE` to the auxiliary database, and choose a unique `COLLECTTABLEPREFIX` such as `s3gc__`. Keep the prefix after a partial delete so the replacement Job can use the deletion checkpoints. Without `COLLECTDATABASE`, the auxiliary table lands in the ClickHouse user's current database, which can differ from `default` and cause a grant failure; `COLLECTTABLEPREFIX` still accepts an embedded `db.prefix_` form for compatibility, but a value there that disagrees with `COLLECTDATABASE` is rejected at startup. -5. Set `USEAGE_HOURS` to 24 or more. Raise it for slow merges or long mutations; production phases cannot lower it. - -For an installation-wide cleanup, set `S3PATH` to the parent prefix that -contains every replica's objects, such as `clickhouse//`, not a -single `//` prefix — but only when that parent prefix -holds objects for the one `S3DISKNAME` being cleaned up. The orphan check -matches ClickHouse's `system.remote_data_paths` by `disk_name`, so an object -that belongs to a different disk stored under the same parent prefix would -never match and would look orphaned. When disks share a parent prefix, collect -a disk-scoped prefix instead, or give each disk its own S3 path layout. -Collect the chosen prefix once into one inventory table on the pinned host. -With `CLUSTERNAME` set, dry-run and delete compare -the inventory with `system.remote_data_paths` on all replicas, so an object -that any replica still references is never a candidate. A single-replica -prefix misses the other replicas' orphans. Collect per-replica prefixes only -when you intend to, and give each one its own `COLLECTTABLEPREFIX`. +Set these values in `s3gc.env` before rendering: + +| Setting | What to check | +| --- | --- | +| `CHHOST` | Use one per-replica ClickHouse Service for every phase. The default inventory is node-local. A load-balanced Service can reach a replica without it. | +| `S3BUCKET`, `S3PATH`, `S3DISKNAME` | Match the exact bucket, object prefix, and underlying disk. An empty `S3PATH` selects the bucket root. Use `gcs` for a disk named `gcs`, never a `*_cache` wrapper. | +| `CLUSTERNAME`, `EXPECTED_REPLICAS` | Match the actual cluster and total replica count. Delete stops if the preflight result differs. | +| `COLLECTDATABASE`, `COLLECTTABLEPREFIX` | Use the auxiliary database and a unique prefix such as `s3gc__`. Keep both unchanged during recovery. | +| `USEAGE_HOURS` | Use 24 or more. Raise it for slow merges or long mutations. Production phases reject a lower value. | + +Without `COLLECTDATABASE`, the inventory uses the ClickHouse user's current +database. That database may differ from the one you granted access to. +The older `db.prefix_` form still works in `COLLECTTABLEPREFIX`, but its database +must agree with `COLLECTDATABASE` when both are set. + +For a cleanup across one installation, collect the parent prefix that holds all +replicas' objects, such as `clickhouse//`. Set `CLUSTERNAME` so +the comparison reads references from every configured replica. Collecting only +`//` misses the other replicas' orphans. + +The selected prefix must contain objects for only the chosen disk. References +are matched by disk name, so another disk's objects could look orphaned. Use a +disk-specific prefix when possible. For intentional per-replica runs, give each +prefix its own inventory table. + +If independent clusters or other consumers share the selected scope, follow +the [manual inventory workflow](../../docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters). +Check every consumer before importing candidates. The configured cluster check +cannot see references in other clusters. After importing the reviewed inventory, +start with the dry-run phase and skip collection. Use an immutable multi-architecture image digest in `IMAGE`. CI prints the exact value after publishing. Do not use an image tag. Collection has no resume checkpoint. For a large bucket, collect separate -prefix shards and rerun only a failed shard. Re-listing a shard is safe because -the auxiliary table replaces rows by object path. +prefix shards and rerun only a failed shard. The inventory replaces rows with +the same object path. Do this before deletion, not while recovering a delete. ### 3. Provide credentials and identity @@ -85,12 +93,11 @@ Create or reuse the Kubernetes Secret named by `CREDENTIALS_SECRET`: | `iam` | `S3GC_CHUSER` and `S3GC_CHPASS` | | `aws` | `S3GC_CHUSER`, `S3GC_CHPASS`, plus a credential source available to boto3 | -Set `SERVICE_ACCOUNT` to an existing namespace-local ServiceAccount that your -cluster configures for the chosen workload-identity mode. The Job keeps -`automountServiceAccountToken: false` because it does not call the Kubernetes -API. For `iam`, the platform must inject or otherwise provide the workload -identity credentials. Existing ClickHouse or ClickHouse-backup ServiceAccounts -are suitable when they meet that cluster policy. +Set `SERVICE_ACCOUNT` to an existing ServiceAccount in the namespace. For `iam`, +the platform must provide workload-identity credentials for that account. +The Job keeps `automountServiceAccountToken: false` because it does not call the +Kubernetes API. An existing ClickHouse or ClickHouse-backup ServiceAccount is +suitable if it meets the cluster's identity policy. Create a dedicated auxiliary database and ClickHouse user before the first run. Replace ``, ``, and the password placeholder. @@ -105,7 +112,9 @@ ON CLUSTER IDENTIFIED WITH sha256_password BY ''; GRANT ON CLUSTER - SELECT ON system.* TO s3gc; + SELECT ON system.remote_data_paths, + SELECT ON system.one, + SELECT ON system.clusters TO s3gc; GRANT ON CLUSTER SELECT, INSERT, CREATE TABLE ON .* TO s3gc; @@ -114,13 +123,30 @@ GRANT ON CLUSTER REMOTE ON *.* TO s3gc; ``` +The system-table grants are by name, so the user cannot read `system.query_log` +or other tables that may hold restricted data. `system.remote_data_paths` is the +reference s3gc compares the inventory with, `system.one` backs the replica-count +preflight of the delete phase, and `system.clusters` backs the replica-liveness +check for a Replicated auxiliary database. s3gc also reads `system.databases` +and `system.tables`; on the version this was checked on (26.8) any user can +read those two, limited to the objects it has access to, so they need no grant. + The database grant covers the auxiliary inventory and durable run-log tables. `REMOTE` permits the `clusterAllReplicas` reads used by clustered dry-run and -delete preflight. When `` uses `ENGINE = Replicated`, every -phase, including collect, first checks that all of that database's replicas are -active in Keeper (`S3GC_REPLICA_LIVENESS_RECHECK=true` adds a re-check after one -Keeper session timeout, about 30 s); this reads only system tables and needs no extra grant. Do not grant ClickHouse's `S3` source privilege: s3gc uses -its own S3 client and the Secret or workload identity instead. +delete preflight. These grants support the normal phases, which preserve the +inventory after delete. Optional DROP or TRUNCATE operations need their own +grants. + +For a Replicated auxiliary database, every phase checks replica liveness before +DDL. If ClickHouse cannot report liveness or returns no replicas, s3gc warns and +skips that check. An optional delayed re-check uses the Keeper session timeout +reported by the cluster. See the [CLI preflight options](../../docs/cli-reference.md#clickhouse-connection-and-topology). +The optional delayed re-check also reads `system.zookeeper_connection`. Grant +`SELECT ON system.zookeeper_connection` only if you pass +`--replica-liveness-recheck`; without the grant that flag warns and skips the +re-check. + +s3gc uses its own S3 client. It does not need ClickHouse's `S3` source privilege. For `S3AUTH=static`, create the Secret from a private, shell-style credential file. Keep this file outside the repository, restrict it to its owner, and @@ -162,7 +188,7 @@ if [ -n "${S3GC_S3SESSIONTOKEN:-}" ]; then literals+=(--from-literal=S3GC_S3SESSIONTOKEN="$S3GC_S3SESSIONTOKEN") fi -kubectl -n create secret generic \ +kubectl -n '' create secret generic '' \ "${literals[@]}" unset S3GC_CHUSER S3GC_CHPASS S3GC_S3ACCESSKEY S3GC_S3SECRETKEY S3GC_S3SESSIONTOKEN @@ -174,15 +200,16 @@ object keys. Keep delete permission separate until an approved delete phase. ### 4. Run each phase -For each phase, edit only the values shown below, render the Job, server-side -validate it, apply it, and follow its logs: +For each phase, change the values in the table below. Render the Job, validate +it against the API server, apply it, and follow its logs. Replace the +placeholders before running these commands: ```bash .venv/bin/python deploy/kubernetes/render.py "$S3GC_RUN_DIR/s3gc.env" \ > "$S3GC_RUN_DIR/.yaml" kubectl apply --dry-run=server -f "$S3GC_RUN_DIR/.yaml" kubectl apply -f "$S3GC_RUN_DIR/.yaml" -kubectl -n logs -f job/ +kubectl -n '' logs -f 'job/' ``` | Phase | Change in `s3gc.env` | Required outcome | @@ -198,20 +225,29 @@ for a bounded trial delete. Clear it before the full run and verification. The use-phase query applies this limit separately to each sample, so it is not a global cap on the Job. -The entrypoint uses `--collectonly --keepdata` for collection, -`--usecollected --dry-run` for review and verification, and -`--usecollected --keepdata --non-interactive` for delete after the token and -topology-input gates. See the [CLI reference](../../docs/cli-reference.md) for -all parameters and the [manual shared-bucket workflow](../../docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters). +The entrypoint selects the CLI flags for each phase: + +| Phase | CLI flags | +| --- | --- | +| Collect | `--collectonly --keepdata` | +| Dry-run and verify | `--usecollected --dry-run` | +| Delete | `--usecollected --keepdata --non-interactive`, after the token and cluster-input gates | + +See the [CLI reference](../../docs/cli-reference.md) for parameter details. ### 5. Recover or verify If a delete Job fails, create a replacement delete Job with a new `JOB_NAME` -and the same `COLLECTTABLEPREFIX`. Do not re-collect unless the scope changed. -`backoffLimit: 0` prevents automatic retries. +and the same host, scope, and inventory settings. Inspect the failure and confirm +that the approval still applies first. Do not re-collect or drop the inventory +during recovery. A changed scope needs a new inventory, review, and approval. +`backoffLimit: 0` prevents automatic Job reruns. Individual S3 requests can still +retry within the running process. After a successful delete, run the verification dry-run. Do not declare the -cleanup complete until it reports zero candidates. +cleanup complete until it reports zero candidates with `USETOTAL` cleared. +That result covers the reviewed inventory and age window. It does not check +new objects absent from the inventory or prove the bucket is empty. ## Configuration reference @@ -219,6 +255,9 @@ cleanup complete until it reports zero candidates. values, unpinned images, invalid phases, invalid booleans, and unsafe age windows. Start with [example.env](example.env). +Renderer keys differ from CLI environment names. For example, `USEAGE_HOURS` +becomes `S3GC_USEAGE`, and `RUNLOG` becomes `S3GC_RUNLOG_FLAG`. + | Group | Required values | Notes | | --- | --- | --- | | Job | `JOB_NAME`, `NAMESPACE`, `IMAGE`, `IMAGE_PULL_SECRET`, `SERVICE_ACCOUNT`, `CREDENTIALS_SECRET` | `JOB_NAME` is a DNS label and becomes the durable run ID. Leave `IMAGE_PULL_SECRET` empty for the public image. | @@ -227,13 +266,13 @@ windows. Start with [example.env](example.env). | Limits | `SAMPLES`, `DELETE_BATCH_SIZE`, `USEAGE_HOURS`, `ACTIVE_DEADLINE_SECONDS`, `TTL_SECONDS_AFTER_FINISHED`, `MEMORY_REQUEST`, `MEMORY_LIMIT` | Keep `SAMPLES` unchanged after collect for partition pruning. | | Behavior | `PHASE`, `DELETE_CONFIRMATION`, `ORDER_BY_OBJPATH`, `VERBOSE` | The renderer requires the confirmation token for delete and development automation. | -`USETOTAL` is optional and limits each sample in a use phase. `RUNLOG` defaults to `true`; -set it to `false` only when stdout is an adequate operational record. +`USETOTAL` is optional and limits each sample in a use phase. `RUNLOG` defaults +to `true`. Set it to `false` only when stdout is an adequate operational record. ## Durable run history -Pod logs are for monitoring, not durable evidence. The kubelet rotates them, -and the Job TTL removes the pod. By default, `s3gc` writes run events to +The kubelet rotates pod logs, and the Job TTL removes the pod. By default, +`s3gc` also writes durable run events to `_log` in ClickHouse. Use the same replica-pinned host to inspect a run: @@ -255,12 +294,14 @@ the cleanup. with a fresh inventory and requires `DELETE_CONFIRMATION=DELETE_ORPHANS`, `CLUSTERNAME`, and `EXPECTED_REPLICAS`. -Use it only for isolated development fixtures. It bypasses the manual review -and approval gate, and it alone may use an age window below 24 hours. +Use it only for isolated development fixtures. It runs all three phases without +a pause for manual review. The renderer allows an age window below 24 hours +only for this phase, and the entrypoint passes the development override. +Never use it for customer or production data. ## Preserved safeguards - The manifest runs as a non-root user with a read-only root filesystem and no Linux capabilities. - Credentials come from `CREDENTIALS_SECRET` or workload identity, never the image or environment file. - The renderer requires a digest-pinned image and validates delete confirmation before the Job reaches the cluster. -- Delete requires the cluster/replica preflight, records checkpoints, and never retries automatically. +- A delete Job requires the cluster/replica preflight and records checkpoints. A failed Job is not rerun automatically. diff --git a/deploy/kubernetes/example.env b/deploy/kubernetes/example.env index 88c6208..2b91694 100644 --- a/deploy/kubernetes/example.env +++ b/deploy/kubernetes/example.env @@ -5,10 +5,10 @@ # Job identity and lifecycle JOB_NAME=s3gc-example-dry-run NAMESPACE=clickhouse -# Always digest-pinned; CI prints the exact line in its job summary. +# Always pin the image by digest. CI prints the exact line in its job summary. IMAGE=ghcr.io/altinity/s3gc@sha256:0000000000000000000000000000000000000000000000000000000000000000 -# The published image is PUBLIC, so leave this empty: the renderer then omits -# the imagePullSecrets block entirely. Set it only for a private mirror. +# Leave empty for the public image. The renderer omits imagePullSecrets. +# Set a secret name only for a private mirror. IMAGE_PULL_SECRET= PHASE=dry-run # Development automation only: set PHASE=dev-automation and @@ -22,9 +22,8 @@ CHHOST=clickhouse.example.svc.cluster.local CHPORT=8123 CLUSTERNAME=example-cluster EXPECTED_REPLICAS=2 -# COLLECTDATABASE names the dedicated auxiliary database. Without it, the -# auxiliary table lands in the ClickHouse user's current database, which may -# not be the one you granted. +# Use the dedicated auxiliary database. Without it, the inventory uses the +# ClickHouse user's current database, which may not have the required grants. COLLECTDATABASE=s3gc COLLECTTABLEPREFIX=s3gc_example_ @@ -36,8 +35,8 @@ S3PATH=clickhouse-data/ S3REGION=us-east-1 S3SECURE_FLAG=true S3DISKNAME=s3 -# static = explicit keys in the Secret; aws = boto3 chain / SSO profile; iam = IRSA/IMDS. -# Prefer iam in Kubernetes: it refreshes credentials and needs no boto3. +# static uses keys from the Secret. aws uses boto3 credentials or an SSO profile. +# iam uses workload identity. Prefer it for in-cluster AWS runs. S3AUTH=iam # Only for S3AUTH=aws. The session token, if any, belongs in the Secret. S3PROFILE= @@ -45,16 +44,13 @@ S3PROFILE= # Processing limits. Keep SAMPLES unchanged after collection. SAMPLES=4 DELETE_BATCH_SIZE=1000 -# Minimum 24, and the renderer refuses less for every phase except -# dev-automation. This window is the only protection against deleting a part -# between its upload to S3 and its registration in system.remote_data_paths. -# Raising it is fine and sometimes wise; lowering it is not. +# Use at least 24 hours for production. This window protects objects uploaded +# before ClickHouse registers their paths. Only dev-automation may use less. USEAGE_HOURS=24 -# Optional limit on how many collected objects each sample processes. Leave -# empty for a full run. Set it to bound the first delete against a new image or -# cluster; the whole run can process up to this limit per sample. +# Optional limit per sample. Use it for a trial delete with a new image or +# cluster. Leave empty for a full run and verification. USETOTAL= -# Leave false for Kubernetes Jobs; global ordering is unnecessary for deletion. +# Leave false for Kubernetes Jobs. Sorting candidates can be expensive. ORDER_BY_OBJPATH=false # Pod limits and observability. @@ -63,9 +59,7 @@ TTL_SECONDS_AFTER_FINISHED=604800 MEMORY_REQUEST=1Gi MEMORY_LIMIT=4Gi VERBOSE=true -# Record durable run history in _log in ClickHouse. -# Pod logs are rotated by the kubelet and deleted with the Job, so this table is -# the only evidence of what a cleanup did once TTL_SECONDS_AFTER_FINISHED -# expires. Needs the same CREATE TABLE grant as the auxiliary table; a missing -# grant degrades to stdout only instead of failing the run. +# Write durable run history to _log in ClickHouse. +# It survives pod-log rotation and Job deletion. It needs the same CREATE TABLE +# grant as the inventory. A logging failure warns and falls back to stdout. RUNLOG=true diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 3af38d8..7c665d1 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -70,6 +70,10 @@ run does not impose that requirement when the count is unset. | `--s3-read-timeout` | `S3GC_S3_READ_TIMEOUT` / `s3_read_timeout` | `120`. Integer S3 read timeout in seconds. | | `--s3-retries` | `S3GC_S3_RETRIES` / `s3_retries` | `3`. Integer HTTP retry count for transient S3 errors. | +HTTP retries happen within a running process. Batch deletion can also reconnect +and retry once after a transport failure. These retries are separate from +Kubernetes Job reruns, which the manifest disables. + Static mode does not automatically read the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` variables. Supply the corresponding `S3GC_*` values through a secret manager, or choose an appropriate AWS/IAM mode. Collection @@ -283,6 +287,18 @@ expected by s3gc. This example uses `S3GC_COLLECTDATABASE=s3gc`, `S3GC_COLLECTTABLEPREFIX=manual_cleanup_`, `S3GC_S3DISKNAME=s3`, and `S3GC_SAMPLES=4`. Use a unique Keeper path and a cluster with correctly configured `{shard}` and `{replica}` macros. Review the DDL for the installation. + +Check the database engine before choosing how to create the table: + +```sql +SELECT engine FROM system.databases WHERE name = 's3gc'; +``` + +The `CREATE TABLE` example below uses `ON CLUSTER` for an Atomic database. +If `s3gc` uses `ENGINE = Replicated`, omit `ON CLUSTER ''`. +That database distributes table DDL itself. See the +[Replicated database documentation](https://clickhouse.com/docs/engines/database-engines/replicated). + The table engine replicates data within a shard. It does not automatically share rows across independent shards or clusters. Table replication is separate from a Replicated database's DDL replication. See ClickHouse's @@ -310,17 +326,20 @@ clickhouse-client --host '' \ < "$S3GC_RUN_DIR/reviewed-candidates.csv" ``` -Verify the imported keys and metadata against the reviewed CSV, including count -and total bytes, and confirm replication has caught up where the table will be read. Replication is -asynchronous. Pinning all s3gc phases to the import replica avoids assuming an -immediate copy elsewhere. A partial import or an empty candidate table is not a -successful cleanup: resolve it before proceeding. The normal s3gc collector -creates a node-local table. It does not create this replicated table for you. +Compare the imported keys, metadata, count, and total bytes with the reviewed +CSV. Confirm replication has caught up where the table will be read. Replication +is asynchronous, so keep all s3gc phases pinned to the import replica. +Resolve any partial import before proceeding. An empty candidate table cannot +be passed to the use phase. If the review found no candidates, record that +result instead of starting delete. + +The normal s3gc collector creates a node-local table. You must create the +replicated table yourself for this workflow. ### 4. Preview, approve, delete, and verify -Configure the matching database, prefix, disk, bucket, prefix, execution cluster, -expected replicas, sample count, and age window. Refresh and compare references +Keep the same database, table prefix, disk, bucket, object prefix, execution +cluster, expected replicas, sample count, and age window. Refresh references from **all** consumers immediately before delete, and verify that candidate objects still have the reviewed metadata. Maintain the agreed period in which references cannot change and candidate keys cannot be overwritten or reused. @@ -339,10 +358,9 @@ clean up private staging artifacts according to your retention policy. ## Advanced: configuration files and precedence -Each parameter has a parser destination, shown below as its configuration key. -Use that key in a JSON or YAML file passed with `--cfg`, or set the listed -`S3GC_*` environment variable. For example, this non-secret JSON file selects a -replica and enables a preview: +The tables above list each parameter's configuration key. Use those keys in a +JSON or YAML file passed with `--cfg`. For example, this file selects a replica +and enables a preview: ```json { @@ -362,7 +380,7 @@ The parser applies other environment variables after that environment-selected file. Keep secrets outside config files and shell history. `--help` (`-h`) prints CLI help. `--print_config` prints the effective parsed -configuration and exits. **its output can contain credentials**, so do not +configuration and exits. **Its output can contain credentials**, so do not share or commit it. `--list-options` (`--listoptions`) prints environment-style options and redacts configured credentials. These inspection options do not contact ClickHouse or S3. `--cfg` has environment variable `S3GC_CFG`. The help diff --git a/docs/plans/cli-parameter-documentation.md b/docs/plans/cli-parameter-documentation.md index 04779e0..4c93dca 100644 --- a/docs/plans/cli-parameter-documentation.md +++ b/docs/plans/cli-parameter-documentation.md @@ -1,6 +1,10 @@ # Plan: complete CLI parameter documentation Status: implemented on `docs/cli-parameter-reference`. +This completed plan records the original work. Use the +[CLI reference](../cli-reference.md) and +[Kubernetes runbook](../../deploy/kubernetes/README.md) for current instructions. + Planning branch: `docs/cli-reference-plan`. Proposed implementation branch: `docs/cli-parameter-reference`. diff --git a/skills/altinity-clickhouse-s3gc/SKILL.md b/skills/altinity-clickhouse-s3gc/SKILL.md index eff6650..bf80991 100644 --- a/skills/altinity-clickhouse-s3gc/SKILL.md +++ b/skills/altinity-clickhouse-s3gc/SKILL.md @@ -5,79 +5,95 @@ description: Safely investigate, plan, review, and execute cleanup of orphaned o # ClickHouse S3 Garbage Collection -Use this skill for the `s3gc` workflow only: identifying objects in a configured object-store -bucket/prefix that are absent from ClickHouse `system.remote_data_paths`, reviewing candidates, -and removing approved orphaned objects. It is not a generic S3 cleanup procedure. +Use this skill to find, review, and remove approved orphaned objects with s3gc. +It compares an inventory with ClickHouse `system.remote_data_paths` for the +selected disk and cluster. Use the Kubernetes runbook for production. ## Non-negotiable rules -- Deletion is irreversible. For customer and production work, use the Kubernetes Job workflow: - **collect → dry-run → explicit human approval → delete → verify**. -- Scope the bucket and prefix as narrowly as possible. A wrong prefix can produce a misleading - clean result or expand deletion scope. -- Use the underlying object disk (for example `s3` or `gcs`) for `S3DISKNAME`, never its - `*_cache` wrapper. The cache does not represent a separate object-reference scope. -- Pin `CHHOST` to a per-replica ClickHouse Service, and use the same host for every phase. The - auxiliary inventory table is node-local; a load-balanced Service can send phases to different - replicas and invalidate the result. -- For clustered cleanup, require the actual cluster name and expected replica count. The delete - phase must pass `s3gc`'s cluster/replica preflight before any S3 call. -- Use a unique `COLLECTTABLEPREFIX` for each bucket/prefix cleanup. Preserve it after a partial - delete: confirmed deletions are checkpointed, and a replacement delete Job resumes from that - state without re-collecting. -- Keep `USEAGE_HOURS` at 24 or higher for production phases. This protects objects uploaded to S3 - before ClickHouse has registered their paths. -- An agent must not supply `DELETE_ORPHANS`, create a delete manifest with that token, or bypass - the reviewed dry-run gate. A human must review the immediate dry-run output and explicitly - authorize deletion. -- Never commit or record credentials, customer `.env` files, rendered manifests, bucket names, - cluster identifiers, or command output containing them. Keep run material outside this repo and - use Kubernetes Secrets or workload identity. +- Deletion is irreversible. Follow **collect → dry-run → explicit human approval + → delete → verify** for customer and production work. +- Limit the bucket and prefix to the intended scope. A wrong prefix can hide + orphans or include objects you must not delete. +- Set `S3DISKNAME` to the underlying object disk, such as `s3` or `gcs`. + Never use its `*_cache` wrapper. +- Pin `CHHOST` to one per-replica ClickHouse Service for every phase. The + default inventory is node-local. Keep the same pinned host even when using + a manually created replicated inventory. +- Require the actual cluster name and expected replica count. Delete must + pass the cluster/replica preflight before deleting any objects. +- Use a unique `COLLECTTABLEPREFIX` for each cleanup scope. Keep it after a + partial delete so a replacement Job can use the confirmed checkpoints. +- Keep `USEAGE_HOURS` at 24 or more for production. This protects objects + uploaded before ClickHouse registers their paths. +- An agent must not supply `DELETE_ORPHANS`, create a delete manifest with that + token, or bypass dry-run review. A human must review the immediate dry-run + output and explicitly authorize deletion. +- Never record credentials. Keep customer configuration, rendered manifests, + bucket names, cluster identifiers, and their output outside Git. Use a private + run directory and the authorized support record for run evidence. + Provide credentials through Kubernetes Secrets or workload identity. ## Workflow ### 1. Establish the cleanup scope -Before collecting, record the purpose, affected disk, endpoint, bucket, exact prefix, ClickHouse -cluster, expected replica count, namespace, and per-replica `CHHOST`. Confirm the intended S3 -authentication mode and the minimum ClickHouse/S3 permissions. Prefer workload identity (`iam`) -for an in-cluster AWS cleanup; use static credentials only from a runtime Secret. +Record the purpose, disk, endpoint, bucket, prefix, cluster, expected replicas, +namespace, and pinned host in the private run record. Confirm the authentication +mode and required ClickHouse and S3 permissions. Prefer `iam` for in-cluster AWS +work. Supply static credentials only through a runtime Secret. -Read [the Kubernetes operator guide](../../deploy/kubernetes/README.md) and start from -[`deploy/kubernetes/example.env`](../../deploy/kubernetes/example.env) copied to a private run -directory outside the repository. Use an immutable multi-architecture image digest, not a tag. -Select an existing namespace-local ServiceAccount configured for the cluster's intended workload -identity mode; the runner does not need Kubernetes API credentials. +Read the [Kubernetes runbook](../../deploy/kubernetes/README.md). Copy +[example.env](../../deploy/kubernetes/example.env) to the private run directory. +Use a multi-architecture image pinned by digest and an existing ServiceAccount +configured for the intended identity. The Job does not need Kubernetes API access. + +If independent clusters share the scope, follow the +[manual inventory workflow](../../docs/cli-reference.md#manual-inventory-for-a-bucket-shared-by-independent-clusters). +Check references from every cluster, replica, relevant disk, and other consumer. +Import only reviewed candidates. s3gc's configured-cluster check cannot see +independent clusters. Stop if the review cannot cover every consumer. ### 2. Collect and review -Render and run a `collect` Job, then a `dry-run` Job against the same `CHHOST` and collection -table. Collect does not delete; dry-run reports the candidate count and size. Review the exact -bucket and prefix too. Capture the rendered configuration location and the non-secret Job/log -evidence in the active support record without copying credentials or customer artifacts into Git. +Run `collect`, then `dry-run`, against the same host and inventory. Collection +does not delete objects. Dry-run reports the candidate count and size. Review +the bucket and prefix too. Store the configuration location and Job/log evidence +in the authorized support record without copying credentials or artifacts into Git. + +For a manually imported inventory, skip collect and start with dry-run. +`--use-collected` reads the existing table without listing S3 again. Keep the +all-consumer reference review current through deletion, as the manual workflow +requires. -Treat an empty or missing collected inventory as a failure to investigate, not proof that the -bucket is clean. Verify the prefix, disk name, and replica-pinned service first. For large buckets, -shard collection by prefix; interrupted collection has no checkpoint and must be rerun only for -the affected shard. +Investigate a missing or empty inventory before proceeding. Check the prefix, +disk, and pinned host. For large buckets, collect separate prefix shards. +Interrupted collection has no checkpoint. Rerun only the failed shard before +deletion, not while recovering a delete. ### 3. Require approval and delete -Present the dry-run candidate count, size, exact bucket/prefix scope, and any uncertainty to the -authorized customer/operator. Do not proceed without explicit approval. +Present the candidate count, size, exact scope, and any uncertainty to the +authorized operator. Do not proceed without explicit approval. -After the human supplies the delete confirmation in the private runtime configuration, render a -new delete Job using the same collection-table prefix. Monitor it to completion. A failed delete -Job has no automatic retry; start a replacement delete Job with the same prefix so the confirmed -deletion checkpoints are preserved. Do not re-collect unless the scope itself changed. Keep the -durable run log enabled unless stdout is the approved operational record. +After the human supplies the token in the private configuration, render a new +delete Job with the same host, scope, and inventory. Monitor it to completion. +Keep the durable run log enabled unless stdout is the approved record. + +A failed Job is not rerun automatically. Individual S3 requests can retry within +the process. Inspect the failure and confirm that approval still applies before +starting a replacement Job. Preserve the inventory and checkpoints. Do not +re-collect during recovery. A changed scope needs a new inventory, review, and +approval. ### 4. Verify and hand off -Run the final `dry-run` phase against the same host and inventory. It must report zero remaining -candidates before declaring the cleanup complete. Record the approved scope, image digest, -cluster/replica preflight result, candidate and deleted totals, verification result, and any -partial-failure recovery steps in the active ticket's handoff record. +Run a final dry-run against the same host and inventory with `USETOTAL` cleared. +It must report zero candidates before the cleanup is complete. That result +covers the reviewed inventory and age window, not newly uploaded objects. + +Record the approved scope, image digest, preflight result, candidate and deleted +totals, verification result, and recovery steps in the authorized handoff record. ## References @@ -86,4 +102,5 @@ partial-failure recovery steps in the active ticket's handoff record. - [s3gc README](../../README.md) — tool model, authentication modes, direct read-only usage, and local development checks. - [`deploy/kubernetes/example.env`](../../deploy/kubernetes/example.env) — non-secret renderer - template; copy it only to a private run directory. + template. Copy it only to a private run directory. +- [CLI reference](../../docs/cli-reference.md) — parameter details and manual inventory preparation. diff --git a/tests/dev_cluster_e2e_spec.md b/tests/dev_cluster_e2e_spec.md index 329635e..6338974 100644 --- a/tests/dev_cluster_e2e_spec.md +++ b/tests/dev_cluster_e2e_spec.md @@ -1,5 +1,10 @@ # Dev-cluster E2E spec +Historical acceptance criteria for the test now implemented in +[test_s3gc_dev_cluster.py](test_s3gc_dev_cluster.py). See its explicit opt-in +requirements before using an isolated development cluster. These criteria are +not instructions to run a live test as part of the default suite. + Add an opt-in test that validates s3gc's Replicated-database preflight and cross-replica inventory checkpoints against an isolated development ClickHouse cluster and S3 prefix. The test must never run in the default pytest suite. From b6534d11d5a251f71edc49184a2705a9ff10be90 Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Wed, 28 Oct 2026 12:47:12 +0100 Subject: [PATCH 3/3] skill improvement --- skills/altinity-clickhouse-s3gc/SKILL.md | 27 +++++++++++++++--------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/skills/altinity-clickhouse-s3gc/SKILL.md b/skills/altinity-clickhouse-s3gc/SKILL.md index bf80991..65f34da 100644 --- a/skills/altinity-clickhouse-s3gc/SKILL.md +++ b/skills/altinity-clickhouse-s3gc/SKILL.md @@ -20,8 +20,9 @@ selected disk and cluster. Use the Kubernetes runbook for production. - Pin `CHHOST` to one per-replica ClickHouse Service for every phase. The default inventory is node-local. Keep the same pinned host even when using a manually created replicated inventory. -- Require the actual cluster name and expected replica count. Delete must - pass the cluster/replica preflight before deleting any objects. +- For clustered cleanup, require the actual cluster name and expected replica + count. Delete must pass the cluster/replica preflight before deleting objects. + The Kubernetes delete workflow requires both settings. - Use a unique `COLLECTTABLEPREFIX` for each cleanup scope. Keep it after a partial delete so a replacement Job can use the confirmed checkpoints. - Keep `USEAGE_HOURS` at 24 or more for production. This protects objects @@ -56,15 +57,21 @@ independent clusters. Stop if the review cannot cover every consumer. ### 2. Collect and review -Run `collect`, then `dry-run`, against the same host and inventory. Collection -does not delete objects. Dry-run reports the candidate count and size. Review -the bucket and prefix too. Store the configuration location and Job/log evidence -in the authorized support record without copying credentials or artifacts into Git. +Choose the inventory path before running a Job: -For a manually imported inventory, skip collect and start with dry-run. -`--use-collected` reads the existing table without listing S3 again. Keep the -all-consumer reference review current through deletion, as the manual workflow -requires. +- For normal collection, run `collect`, then `dry-run`. +- For a manually imported inventory, skip `collect` and start with `dry-run`. + Never re-collect into the reviewed candidate table. + +Use the same pinned host and inventory for subsequent phases. +`--use-collected` reads the existing table without listing S3 again. For a manual +inventory, keep the all-consumer reference review current through deletion, as +the manual workflow requires. + +Collection does not delete objects. Dry-run reports the candidate count and +size. Review the bucket and prefix too. Store the configuration location and +only redacted, non-secret Job/log evidence in the authorized support record. +Keep credentials and customer artifacts out of Git. Investigate a missing or empty inventory before proceeding. Check the prefix, disk, and pinned host. For large buckets, collect separate prefix shards.