Skip to content

docs(operations): add encryption at rest guides for etcd and Keycloak - #742

Open
Andrei Kvapil (kvaps) wants to merge 3 commits into
mainfrom
docs/kms-encryption
Open

Andrei Kvapil (kvaps) wants to merge 3 commits into
mainfrom
docs/kms-encryption

Conversation

@kvaps

@kvaps Andrei Kvapil (kvaps) commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Do not merge until cozystack/cozystack#4818 is merged. The Keycloak page describes the chart behaviour from that PR: encryption is refused without a DEK set, and the static KEK is generated in a form the proxy accepts.

New section under Operations: Encryption at Rest. It goes to next/ only, because the etcd part needs Talos v1.14, which ships with the upcoming release.

Three pages:

  • Overview: what Cozystack stores where, what is encrypted by default and where the keys are.
  • etcd with KMS v2: management cluster etcd encrypted with Vault Transit through vault-kubernetes-kms running as a Talos static pod. It also encrypts Flux HelmReleases, because application values (passwords included) live in spec.values. Short notes for aggregated API servers (the Cozystack API has no storage of its own, so it is already covered) and for tenant clusters, where KMS is not possible yet and the page shows the local-key option through the existing controlPlane.apiServer values.
  • Keycloak: keycloak-kms-proxy with Vault Transit, including encrypting existing rows in a maintenance window.

The Keycloak page makes deksetSecretName mandatory. With the current chart default (empty) the proxy mints a new DEK on every start and keeps it only in memory, so after a proxy restart it cannot decrypt anything it wrote before (cmd/proxy/main.go, buildCipher). The fix is cozystack/cozystack#4818, which also makes the default static backend start at all.

How this was checked:

  • Every field, flag and command was compared with the source at pinned versions: Talos v1.14.1, Talm v0.35.0, vault-kubernetes-kms v1.4.0, keycloak-kms-proxy v0.2.3, Kubernetes 1.34, and the kubernetes and keycloak charts on main.
  • The tenant values and the Keycloak values render with helm template on main.
  • The same setup (KMS v2 on Talos 1.14 with a static pod, and the Keycloak proxy with a backfill done with Keycloak stopped) runs on a production installation.
  • The site builds and the internal links resolve.

Not run end to end on a clean cluster: the Talm node-file flow with $patch: delete for secretboxEncryptionSecret, and the local-key option for tenant clusters.

Add an Encryption at Rest section that explains what Cozystack stores
where and how it is protected by default, and a guide to encrypt the
management cluster etcd with KMS v2 and HashiCorp Vault Transit on
Talos v1.14. The guide also encrypts Flux HelmReleases, which carry
application values, explains what applies to aggregated API servers,
and shows the local-key option for tenant Kubernetes clusters, where
KMS is not supported yet.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Describe how to enable keycloak-kms-proxy on the platform Keycloak with
HashiCorp Vault Transit: Vault setup, creating the shared DEK set,
encrypting existing rows in a maintenance window, enabling the proxy
through the cozystack.keycloak package, verification and limits.

The guide makes deksetSecretName mandatory: without it the proxy mints
a new key on every start and cannot read data written before a restart.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
@netlify

netlify Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for cozystack ready!

Name Link
🔨 Latest commit 8da73b3
🔍 Latest deploy log https://app.netlify.com/projects/cozystack/deploys/6ac7c1e6f064b1000853d1a3
😎 Deploy Preview https://deploy-preview-742--cozystack.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0374e1b7-6811-4472-ab5c-e1087fbd552b
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The keycloak chart now refuses to enable encryption without
encryption.deksetSecretName, so describe that instead of a default
to avoid.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Andrei Kvapil (@kvaps) NOT LGTM. The pages themselves hold up against the code I checked; the Keycloak page has to wait for cozystack/cozystack#4818, which I just sent back with changes requested.

The Keycloak page says "The chart refuses to enable encryption without it". On cozystack main 52f542c4 that is false: deksetSecretName is optional there and the proxy falls back to an in-memory DEK. Since next/ is published from main, merging this first would document a guard that does not exist. #4818 may also still change in ways this page should mention, for example how the static KEK Secret survives disabling encryption. So please re-check the page against what #4818 merges with.

The PR description starts with "Do not merge until cozystack/cozystack#4818 is merged." This repo uses the description as the merge commit message, so that line would stay in main's history. Please drop it before merging.

What I checked on the Keycloak page and found correct: every encryption.* key and its default against packages/system/keycloak/values.yaml, the backfill flags (generate-dekset, encrypt-rows, -vault-mount, -dsn) against keycloak-kms-proxy v0.2.3, the encrypted columns against internal/config/fields.go, and the metrics port and kkp_decrypt_failures_total. The Vault policy is enough too, because the proxy only calls encrypt/ and decrypt/ on the key. The suspend step is safe as well: the Package reconciler keeps the HelmRelease's spec.suspend as it finds it. For the tenant part, --encryption-provider-config is not one of the flags the kubernetes chart drops or refuses, and secret volumes are allowed in extraVolumes.

One non-blocking note: the prerequisites say "Talos v1.14.1 runs etcd v3.7.1", but next already pins Talos v1.14.2, so the etcd version in that sentence should follow the pinned Talos.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants