Skip to content

Add s3.multipart_download_type range mode to avoid Ceph RGW 412 - #1635

Open
mvanhorn wants to merge 7 commits into
Altinity:masterfrom
mvanhorn:fix/1632-s3-multipart-download-range
Open

mvanhorn wants to merge 7 commits into
Altinity:masterfrom
mvanhorn:fix/1632-s3-multipart-download-range

Conversation

@mvanhorn

Copy link
Copy Markdown
Contributor

Why

With multipart download enabled, a backup download from Ceph RGW (including Hetzner Object Storage) fails for every part archive that was stored as a multipart object. The storage returns HTTP 412 PreconditionFailed and the download uses up its retries.

The multipart download sets concurrency and part size but leaves the transfer manager on its default, partNumber GETs. That mode takes the ETag from the first response and sends it as If-Match on later requests, while RGW answers a partNumber GET with that part's own ETag rather than the multipart object's ETag, so the precondition fails from the second part on. Byte-range GETs of the same object, with If-Match set to the object ETag, succeed.

What changed

s3.multipart_download_type, also read from S3_MULTIPART_DOWNLOAD_TYPE, defaults to part and validation accepts only part and range, case-insensitively, with a blank value keeping the part strategy. On the multipart download path the value is passed as GetObjectType: range selects byte-range GETs sized by the existing chunk calculation, and part keeps partNumber GETs. The README and changelog note that this runs only when allow_multipart_download is true and download_max_bytes_per_second is 0.

Tests

  • gofmt -l on the edited Go files printed nothing, so those files are already formatted.
  • go vet ./pkg/config/... ./pkg/storage/... finished with no reported issues.
  • go test ./pkg/config/... ./pkg/storage/... passed for the config, storage, bwlimit, and object_disk packages.
  • Ran make test in a Linux container (golang:1.24) for the clickhouse-backup S3Config change; it passed.

Fixes #1632

s3.multipart_download_type, also read from S3_MULTIPART_DOWNLOAD_TYPE,
defaults to part and validation accepts only part and range,
case-insensitively, with a blank value keeping the part strategy. On the
multipart download path the value is passed as GetObjectType: range
selects byte-range GETs sized by the existing chunk calculation, and
part keeps partNumber GETs. The README and changelog note that this runs
only when allow_multipart_download is true and
download_max_bytes_per_second is 0.

Fixes Altinity#1632
Slach and others added 2 commits October 22, 2026 14:33
partNumber GETs fail with HTTP 412 on Ceph RGW (Hetzner Object Storage),
so keeping `part` as default leaves every RGW user broken until they find
the knob. Byte ranges work on AWS S3 too and make PartSizeBytes effective.
Empty value now maps to range as well; `part` stays as explicit opt-out.
Add the new key to the testflows default_config snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An explicit empty value (yaml "" or S3_MULTIPART_DOWNLOAD_TYPE=) silently
picked a download strategy; require part or range instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Slach Slach changed the title Add s3.multipart_download_type range mode for Ceph RGW 412 Add s3.multipart_download_type range mode to avoid Ceph RGW 412 Oct 22, 2026
@Slach
Slach self-requested a review October 22, 2026 09:56
v2.8.2 is already tagged, the fix ships in the next release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mvanhorn

Copy link
Copy Markdown
Contributor Author

Thanks @Slach for the follow-ups and the approval. Making range the default and rejecting an empty value both make sense, since the 412 from Ceph RGW is otherwise easy to hit with the old default.

Slach and others added 2 commits October 28, 2026 09:30
Resolve the cli snapshot conflict by keeping both the new
multipart_download_type default and the timeout key from master.
@mvanhorn
mvanhorn force-pushed the fix/1632-s3-multipart-download-range branch from 9d6b3a1 to d2877c9 Compare October 28, 2026 11:51
The merge resolution placed the timeout key from master under general
instead of after http2_write_byte_timeout. The snapshot now matches
master plus the multipart_download_type line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3.allow_multipart_download fails with 412 PreconditionFailed on Ceph RGW: the transfer manager downloads by partNumber, please add a byte-range mode

2 participants