feat(s3): support SSE-C customer-provided keys by schaurian · Pull Request #284 · cloudnative-pg/barman-cloud

schaurian · 2026-07-19T02:35:21Z

Add an optional sseCustomerKey field to S3Credentials. It references a secret with a base64-encoded 256-bit AES key. S3-compatible stores such as Hetzner Object Storage and some Ceph setups only offer SSE-C, so the existing bucket-managed encryption field cannot be used there.

The key is written to /controller/.sse-customer-keys/<secret>/<key> before the auth method is handled, so it works with every auth method, including inheritFromIAMRole. The path depends on the secret and key because one process can serve several object stores at once, for example a replica cluster archiving to its own store while restoring from the source one. A shared file would let a command pick up the wrong key. Key files are not removed when the field or the secret goes away. Changing the key makes existing objects unreadable, so a new key needs a new destinationPath or serverName.

The key is passed as --sse-customer-key file://<path> to every barman-cloud command except barman-cloud-check-wal-archive, which only lists objects and rejects the option. ValidateBackupConfiguration rejects sseCustomerKey together with data.encryption or wal.encryption, which barman-cloud refuses at run time.

The credentials types now list the secrets they reference through SecretNames(), with a test that fails when a new secret field is not listed. Consumers can use it to grant access to those secrets instead of keeping their own list.

Requires Barman 3.20.0 or later. The plugin side is cloudnative-pg/plugin-barman-cloud#1017.

Refs cloudnative-pg/plugin-barman-cloud#646

Surface the new `sseCustomerKey` field on `s3Credentials` through the ObjectStore CRD so users can enable Server-Side Encryption with Customer-provided keys (SSE-C). This is required by S3-compatible providers that only support SSE-C for encryption at rest, such as Hetzner Object Storage. The field flows through the embedded BarmanObjectStoreConfiguration from the barman-cloud library, so this change is limited to bumping the dependency, regenerating the CRD and the consolidated manifest, and documenting usage in the object stores guide. Depends on cloudnative-pg/barman-cloud#284 (temporarily pinned via a replace directive until that change is released). Closes cloudnative-pg#646 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Florian Schauer <florian@schauer.to>

fkammer · 2026-09-02T07:49:13Z

FYI: barman release is out: https://github.com/EnterpriseDB/barman/releases/tag/release%2F3.20.0

andrein · 2026-09-08T12:18:46Z

Tested at a300841 against a Ceph RGW object store that supports only SSE-C
(no SSE-S3 or SSE-KMS), with CloudNativePG 1.30.0 and the Barman Cloud plugin
from cloudnative-pg/plugin-barman-cloud#1017 with main merged in for
Barman 3.20.0.

WAL archiving continuous; objects under wals/ report Encryption: SSE-C
and return 400 when read without the key
base backup completed; Backup.status populated by barman-cloud-backup-list
reading the SSE-C backup.info
ObjectStore.status.serverRecoveryWindow populated
retention policy applied on the primary via barman-cloud-backup-delete
recovery into a new Cluster through externalClusters[].plugin reproduced
the source database: same row counts, table count and size

--sse-customer-key together with --encryption is rejected by Barman at run
time; schaurian#1 against this branch adds the corresponding check
to ValidateBackupConfiguration.

Surface the new `sseCustomerKey` field on `s3Credentials` through the ObjectStore CRD so users can enable Server-Side Encryption with Customer-provided keys (SSE-C). This is required by S3-compatible providers that only support SSE-C for encryption at rest, such as Hetzner Object Storage. The field flows through the embedded BarmanObjectStoreConfiguration from the barman-cloud library, so this change is limited to bumping the dependency, regenerating the CRD and the consolidated manifest, and documenting usage in the object stores guide. Depends on cloudnative-pg/barman-cloud#284 (temporarily pinned via a replace directive until that change is released). Closes cloudnative-pg#646 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Florian Schauer <florian@schauer.to>

schaurian · 2026-09-08T14:53:38Z

Thanks for the test run and the validation change, @andrein — merged schaurian#1 into this branch.

Updated the branch (14f89b6):

Rebased onto main (v0.6.0); the SSE-C commit and the validation commit are now separate commits on top of it.
Reworded the body of the validation commit to start with a capital letter: this repo's commitlint enforces body-case: sentence-case, and that was the one task ci failure on the previous head. Author and sign-off are unchanged.
The SSECustomerKey API comment used to say the field is "orthogonal" to encryption; it now states that the two cannot be combined, matching the validation. Description updated accordingly.

Full task ci (commitlint, spellcheck, golangci-lint v2.13.2, go test ./..., uncommitted-drift check) passes locally on this head.

For the record, Barman 3.20.0 with --sse-customer-key is released, and cloudnative-pg/plugin-barman-cloud#1017 is rebased on a main whose sidecar pins it.

neurodrone · 2026-09-25T14:05:50Z

👋 Is there a timeline on merging this? Looks like the use-cases backed by strong security+encryption requirements (and are not necessarily pointed at AWS) won't be able to use CNPG's backups until this merges.

cc @armru @mnencia

Point the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that keeps each SSE-C key in its own file. A replica cluster archives WAL to its own object store while restoring it from the source one in the same sidecar, and with a single shared file each operation could run with the key of the other object store. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Also points the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that serializes the writes of the SSE-C key files. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Point the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that stops passing --sse-customer-key to barman-cloud-check-wal-archive. That command rejects the option, and it runs before the first WAL file is archived, so an object store with an SSE-C key could never start archiving. The SSE-C e2e test caught it. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

armru · 2026-09-30T13:44:40Z

I pushed a few commits on top.

barman-cloud-check-wal-archive is the only barman-cloud command without --sse-customer-key, and it rejects the option. It runs before the first WAL file is archived, so an object store with an SSE-C key could never start archiving. The command only lists objects and does not need the key, so it no longer receives it.
The key was materialized in a single /controller/.sse-customer-key file shared by every object store served by the process. A replica cluster archives WAL to its own object store while restoring it from the source one in the same sidecar, so each operation could remove the key of the other, or have barman-cloud upload objects encrypted with the key of the other object store. Each key now lives in /controller/.sse-customer-keys/<secret>/<key>, and the command builder points at the same path.
The writes of these files are serialized: WriteFileAtomic names its temporary file after the current second, so two concurrent first writes of the same key could collide.

The same shared-file problem exists for the Google credentials, fixed separately in #309.

neurodrone · 2026-10-05T16:14:05Z

Great to see the progress here! Do we have an ETA on when this might go in? cc @armru @mnencia.

gustabowill · 2026-10-06T16:39:29Z

@schaurian @armru I also tested this PR locally and everything went fine. I would just like to ask if you guys see any security concern in not deleting the secret key files. As I understand, if sseCustomerKey is removed, points to another Secret, or the Secret is deleted, the old file remains there forever.

mxzinke · 2026-10-12T10:30:55Z

@schaurian @gustabowill Any chance to merge this PR this week? Happy to be helping test this.

A process can run barman-cloud commands for different object stores at the same time. For example, a replica cluster archives WAL to its own store and restores from the source one. All Google credentials went to one shared file, so a command could run with the identity of another store. A store using the GKE environment also emptied the file while other commands read it. Each secret key now has its own file, `/controller/.google-credentials/<secret>/<key>`, and `GOOGLE_APPLICATION_CREDENTIALS` points to it. The GKE environment no longer writes any file. Writes are serialized because `WriteFileAtomic` names its temporary file after the current second. Same problem as #284, for SSE-C keys. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

neurodrone · 2026-10-15T16:16:34Z

Any updates here? Happy to help with testing as well if it helps speed this up.

Add an `sseCustomerKey` field to the S3 credentials that references a secret holding a base64-encoded 256-bit AES key. When set, the key is materialized to a file and passed to every barman-cloud command through the `--sse-customer-key file://` option, enabling Server-Side Encryption with Customer-provided keys (SSE-C). This is required by S3-compatible providers that only support SSE-C for encryption at rest (e.g. Hetzner Object Storage). The option is injected in the shared `appendCloudProviderOptions` chokepoint, so it applies to all barman-cloud-* commands (backup, wal-archive, wal-restore, restore, backup-list, backup-delete, check-wal-archive), and is orthogonal to the existing bucket-managed `encryption` (SSE-S3/SSE-KMS) option. The key is materialized before the auth-method branching so it works with every authentication method, including inheritFromIAMRole. Requires a barman release that ships the `--sse-customer-key` option (EnterpriseDB/barman#973). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Florian Schauer <florian@schauer.to>

The barman-cloud commands refuse --sse-customer-key together with --encryption, so a configuration setting both fails on the first backup. Reject it at validation time instead, on the data and wal encryption fields. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Andrei Nistor <andrei@nistor.tech>

A single process can run barman-cloud commands for different object stores at the same time. A replica cluster, for instance, archives WAL to its own object store while restoring it from the source one. With one shared key file, each command overwrote or removed the key of the other: a missing file made the command fail, and a different key made barman-cloud upload objects encrypted with the key of another object store, which the configured key can no longer decrypt. Materialize each key in a file whose path depends on the referenced secret and key, and point the barman-cloud commands at that same path. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

The object stores sharing a key secret share its file, and a replica cluster can archive and restore WAL at the same time. WriteFileAtomic names its temporary file after the current second, so two concurrent writes of the same key could use the same temporary file: one of them could then fail its rename, or truncate the key file the other had just moved into place, making barman-cloud read an invalid key. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

The barman-cloud-check-wal-archive command is the only one without the --sse-customer-key option, and it rejects it. Since the check runs before the first WAL file is archived, an object store with an SSE-C key could never start archiving. The command only lists the objects in the bucket, so it does not need the key. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

The declaration of the mutex had split it from the function, so it documented the mutex instead. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

WriteFileAtomic uses a random temporary file name since machinery v0.6.1, so concurrent writes of the same key no longer collide. The mutex and its comment are obsolete. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

mnencia · 2026-10-16T07:56:38Z

@schaurian @armru I also tested this PR locally and everything went fine. I would just like to ask if you guys see any security concern in not deleting the secret key files. As I understand, if sseCustomerKey is removed, points to another Secret, or the Secret is deleted, the old file remains there forever.

@gustabowill
Not a new exposure class. The file is 0600 in a 0700 directory under /controller, which is pod-scoped scratch space, and it lives as long as the pod. The same process already holds the AWS credentials in its environment, and the Google credentials file is written the same way and never removed either. A revoked key stays on disk until the pod restarts, nothing more.

We don't prune on purpose: a replica cluster uses two object stores in one process, and removing a file that looks unreferenced can delete the key another command is about to read.

Surface the new `sseCustomerKey` field on `s3Credentials` through the ObjectStore CRD so users can enable Server-Side Encryption with Customer-provided keys (SSE-C). This is required by S3-compatible providers that only support SSE-C for encryption at rest, such as Hetzner Object Storage. The field flows through the embedded BarmanObjectStoreConfiguration from the barman-cloud library, so this change is limited to bumping the dependency, regenerating the CRD and the consolidated manifest, and documenting usage in the object stores guide. Depends on cloudnative-pg/barman-cloud#284 (temporarily pinned via a replace directive until that change is released). Closes cloudnative-pg#646 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Florian Schauer <florian@schauer.to>

Point the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that keeps each SSE-C key in its own file. A replica cluster archives WAL to its own object store while restoring it from the source one in the same sidecar, and with a single shared file each operation could run with the key of the other object store. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Point the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that stops passing --sse-customer-key to barman-cloud-check-wal-archive. That command rejects the option, and it runs before the first WAL file is archived, so an object store with an SSE-C key could never start archiving. The SSE-C e2e test caught it. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Make the key directory a variable, as done for the Google credentials, so that tests can write the files to a temporary directory. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

The description said every object is encrypted on upload and download, which is wrong for barman-cloud-check-wal-archive and reads as if downloads encrypt. Say what the key is for, and warn that changing it makes existing objects unreadable. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

The operator and the plugin keep their own list of the secrets an instance can read, so a new credential field is silently left out and fails at run time with a forbidden error. Own the list next to the types, and fail the tests when a field is not listed. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

The skip for UseDefaultAzureCredentials had no test, so dropping that operand from the condition went unnoticed. Also check that unset references are skipped when others are set. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

Point the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that stops passing --sse-customer-key to barman-cloud-check-wal-archive. That command rejects the option, and it runs before the first WAL file is archived, so an object store with an SSE-C key could never start archiving. The SSE-C e2e test caught it. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Rename reconcileAWSSSECustomerKey to writeAWSSSECustomerKey, as it only writes the file. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

Surface the new `sseCustomerKey` field on `s3Credentials` through the ObjectStore CRD so users can enable Server-Side Encryption with Customer-provided keys (SSE-C). This is required by S3-compatible providers that only support SSE-C for encryption at rest, such as Hetzner Object Storage. The field flows through the embedded BarmanObjectStoreConfiguration from the barman-cloud library, so this change is limited to bumping the dependency, regenerating the CRD and the consolidated manifest, and documenting usage in the object stores guide. Depends on cloudnative-pg/barman-cloud#284 (temporarily pinned via a replace directive until that change is released). Closes cloudnative-pg#646 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Florian Schauer <florian@schauer.to>

Point the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that keeps each SSE-C key in its own file. A replica cluster archives WAL to its own object store while restoring it from the source one in the same sidecar, and with a single shared file each operation could run with the key of the other object store. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Point the temporary barman-cloud replace at the head of cloudnative-pg/barman-cloud#284 that stops passing --sse-customer-key to barman-cloud-check-wal-archive. That command rejects the option, and it runs before the first WAL file is archived, so an object store with an SSE-C key could never start archiving. The SSE-C e2e test caught it. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

schaurian requested a review from a team July 19, 2026 02:35

schaurian mentioned this pull request Jul 19, 2026

feat(s3): support SSE-C customer-provided keys cloudnative-pg/plugin-barman-cloud#1017

Merged

schaurian force-pushed the feat/s3-sse-c branch from 76e12b2 to a300841 Compare August 9, 2026 13:26

schaurian requested a review from a team as a code owner August 9, 2026 13:26

schaurian closed this Sep 2, 2026

schaurian reopened this Sep 2, 2026

andrein mentioned this pull request Sep 8, 2026

feat: reject sseCustomerKey combined with SSE-S3 or SSE-KMS schaurian/barman-cloud#1

Merged

schaurian force-pushed the feat/s3-sse-c branch from d097344 to 14f89b6 Compare September 8, 2026 14:28

armru mentioned this pull request Sep 30, 2026

fix(gcs): use a separate credentials file per secret key #309

Merged

armru force-pushed the feat/s3-sse-c branch from ac9c5a9 to c6e4b16 Compare September 30, 2026 14:57

armru approved these changes Oct 1, 2026

View reviewed changes

armru force-pushed the feat/s3-sse-c branch 2 times, most recently from 7b0cad0 to 9ee8eac Compare October 1, 2026 08:40

mnencia force-pushed the feat/s3-sse-c branch from 9ee8eac to 7f0d7a6 Compare October 1, 2026 10:23

kevin-nca mentioned this pull request Oct 3, 2026

Encrypt backups before they reach Hetzner nca-apprentices/infra#38

Open

8 tasks

gustabowill reviewed Oct 6, 2026

View reviewed changes

Comment thread pkg/api/config.go Outdated

schaurian and others added 6 commits October 16, 2026 09:17

docs: keep the doc comment of reconcileAWSSSECustomerKey attached to it

a1c4489

The declaration of the mutex had split it from the function, so it documented the mutex instead. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

mnencia force-pushed the feat/s3-sse-c branch from 7f0d7a6 to a1c4489 Compare October 16, 2026 07:20

fix: drop the redundant SSE-C key write mutex

5e81555

WriteFileAtomic uses a random temporary file name since machinery v0.6.1, so concurrent writes of the same key no longer collide. The mutex and its comment are obsolete. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

mnencia changed the title ~~feat: add support for S3 SSE-C (customer-provided encryption keys)~~ feat(s3): support SSE-C customer-provided keys Oct 16, 2026

mnencia approved these changes Oct 16, 2026

View reviewed changes

mnencia force-pushed the feat/s3-sse-c branch from d452ffb to 7e0a00e Compare October 16, 2026 10:18

mnencia added 3 commits October 16, 2026 12:25

test(credentials): cover the SSE-C key file write

beb378c

Make the key directory a variable, as done for the Google credentials, so that tests can write the files to a temporary directory. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

mnencia force-pushed the feat/s3-sse-c branch from 7e0a00e to dd2190d Compare October 16, 2026 10:25

chore: trim SSE-C comments and tidy tests

4b220f3

Rename reconcileAWSSSECustomerKey to writeAWSSSECustomerKey, as it only writes the file. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

armru approved these changes Oct 16, 2026

View reviewed changes

mnencia merged commit ddd6050 into cloudnative-pg:main Oct 16, 2026
2 checks passed

cnpg-bot mentioned this pull request Oct 16, 2026

chore(main): release 0.7.0 #303

Merged

Navigation Menu

Repository navigation

feat(s3): support SSE-C customer-provided keys - #284

feat(s3): support SSE-C customer-provided keys#284
mnencia merged 12 commits into
cloudnative-pg:mainfrom
schaurian:feat/s3-sse-c

schaurian commented Jul 19, 2026 •

edited by mnencia

Loading

Uh oh!

fkammer commented Sep 2, 2026

Uh oh!

andrein commented Sep 8, 2026

Uh oh!

schaurian commented Sep 8, 2026

Uh oh!

neurodrone commented Sep 25, 2026

Uh oh!

armru commented Sep 30, 2026

Uh oh!

neurodrone commented Oct 5, 2026

Uh oh!

Uh oh!

gustabowill commented Oct 6, 2026

Uh oh!

mxzinke commented Oct 12, 2026

Uh oh!

neurodrone commented Oct 15, 2026

Uh oh!

mnencia commented Oct 16, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

schaurian commented Jul 19, 2026 • edited by mnencia Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fkammer commented Sep 2, 2026

Uh oh!

andrein commented Sep 8, 2026

Uh oh!

schaurian commented Sep 8, 2026

Uh oh!

neurodrone commented Sep 25, 2026

Uh oh!

armru commented Sep 30, 2026

Uh oh!

neurodrone commented Oct 5, 2026

Uh oh!

Uh oh!

gustabowill commented Oct 6, 2026

Uh oh!

mxzinke commented Oct 12, 2026

Uh oh!

neurodrone commented Oct 15, 2026

Uh oh!

mnencia commented Oct 16, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

schaurian commented Jul 19, 2026 •

edited by mnencia

Loading