[Slach]

Summary

Replaces the way the operator-managed clickhouse-backup sidecar gets its settings.

Before: the storage Secret's keys came in as env vars through envFrom, and the credentials as secretKeyRef env vars. Kubelet reads env vars only when it starts a container, so a liveness probe compared the mounted Secret files with the sidecar's start-up environment and failed, so that the sidecar got restarted, whenever they differed.

After:

• The storage Secret (backup.storages[].secretRef) holds a key config.yml: a YAML fragment of the clickhouse-backup configuration (general:, s3:, gcs:, …).
• The operator builds a per-cluster Secret <chi>-<cluster>-backup-config from it (pkg/model/chi/backupconfig). The Secret is owned by the CHI and holds:
  • the user fragment;
  • the API and ClickHouse credentials from <chi>-auto-backup-secret;
  • default path/object_disk_path for every storage type;
  • general.disable_environment_override: false;
  • the storage-path-check settings under their env-var names.
• The Secret is mounted read-only at /etc/clickhouse-backup as a whole directory, not via subPath. clickhouse-backup server re-reads config.yml for every API operation (APIServer.ReloadConfig), so a change applies without a restart.
• The settings the Jobs rely on (API_LISTEN, API_SECURE, API_ALLOW_PARALLEL, RESTORE_SCHEMA_ON_CLUSTER, RESTORE_*_MAPPING, CLICKHOUSE_RESTART_COMMAND) and the per-host ClickHouse port/TLS stay env vars, which take precedence over the file.
• The operator drops these from the base container, because they would override or bypass the file:
  • the env vars API_USERNAME, API_PASSWORD, CLICKHOUSE_USERNAME, CLICKHOUSE_PASSWORD and CLICKHOUSE_BACKUP_CONFIG;
  • any mount at, below or above /etc/clickhouse-backup.
• The sidecar's own command waits until the config file exists, so the REST API is never served without credentials. A custom command makes the volume mandatory.
• The CHI reconcile creates and updates the config Secrets before StatefulSets. A 1-minute background sync keeps them in step with the storage Secrets between reconciles. It skips CHIs that are suspended, being deleted, reconcile in progress, or never completed. The operator still doesn't cache or watch user Secrets.
• The Job storage-path check now reads its settings from the generated Secret instead of the user's storage Secret.
• Removed: backup-sidecar-liveness.sh, the envFrom of the storage Secret, the Secret mounts under /var/run/secrets/clickhouse-backup, and the default-path logic in backup-sidecar.sh.
• New event BackupStorageConfigInvalid. BackupStorageSecretMissing now also fires when the Secret has no config.yml key.

⚠️ Breaking (API is EXPERIMENTAL): storage Secrets with env-style keys (REMOTE_STORAGE, S3_BUCKET, …) are no longer read and must be converted to config.yml. Docs, examples and e2e manifests are updated.

How the problems of the previous mechanism are addressed

1. A key removed from the storage Secret was never noticed. The probe only compared files that exist, so a deleted key such as S3_STORAGE_CLASS stayed in the sidecar's environment until something else restarted it. Now the config file is rebuilt from the whole Secret, and clickhouse-backup reloads it for every operation, so a removed key is gone from the next operation on.
2. A base container with its own livenessProbe disabled the mechanism entirely. With such a probe, Secret changes never reached the sidecar until the pod restarted. Applying a change no longer needs a probe or a restart, so a user-defined probe is irrelevant here.
3. The restart raced with the backup Jobs. busy() only saw the local sidecar's operations, so a sidecar on host B could be restarted while a Job was still working on host A and hit B mid-run. A restore is never resumed. Nothing restarts the sidecar any more: a running operation keeps the config it loaded, and the next operation picks up the new one.
4. Every Secret change looked like a crash. It produced Unhealthy / liveness-probe-failed events and bumped restartCount, which triggers container-restart alerts during routine credential rotation. Now there are no probe failures or restarts, only a Secret update.
5. Storage credentials sat in the process environment. They are no longer env vars of the sidecar, so they no longer show in /proc/<pid>/environ or in the container env. They live only in the mounted file, read-only, inside the sidecar container.

Known limitation, documented: clickhouse-backup checks REST API credentials against the config it has loaded, and only an authenticated operation reloads it. If <chi>-auto-backup-secret is deleted and recreated with new random passwords, the sidecars must be restarted by hand. Until then the Jobs get 401. The old mechanism restarted the sidecars in that case.

Testing

• New unit tests:
  • pkg/model/chi/backupconfig: merge, defaults, determinism, invalid input;
  • pkg/controller/chi: create, update only on change, invalid input leaves the Secret alone, registry, periodic sync and its skip conditions.
• Reworked the sidecar and path-check tests. The kubelet simulation now builds the config Secrets the way the operator does.
• go test ./pkg/... ./cmd/... passes in golang:1.27.2 (Linux). The script tests need /proc and GNU date, and are skipped or fail on macOS, as before.
• e2e test_010087 was updated for the new Secret format: it asserts the config volume, the generated Secret and the absence of credential env vars, and waits for the regenerated config before the shared-path backup. test_010087 (11m) and test_010088 (6m) pass on a local OrbStack Kubernetes against the operator image built from this branch. The full e2e suite passed in CI (run 40454429451, 3h 16m); test_010009, test_010011_5, test_010036, test_010086_1 and test_090099 needed a retry.

🤖 Generated with Claude Code