Skip to content

s3: implement s3.timeout (S3_TIMEOUT) as HTTP response-header timeout - #1646

Merged
Slach merged 1 commit into
masterfrom
s3_timeout_1645
Oct 28, 2026
Merged

Slach merged 1 commit into
masterfrom
s3_timeout_1645

Conversation

@Slach

@Slach Slach commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1645

Problem

The S3 backend had no per-request deadline. On Altinity.Cloud production create_remote --diff-from-remote hung 5+ days: workers blocked in net/http.(*persistConn).roundTrip inside S3.CopyObject → UploadPartCopy (10.7 GB object-disk blob), CPU ~0, TCP connection to S3 ESTABLISHED with Send-Q=0, keepalives answered — the server accepted the request and never sent response headers. The http2_* timeouts from #1490 don't help: AWS S3 speaks HTTP/1.1.

s3.timeout / S3_TIMEOUT never existed (unlike azblob/cos/ftp), so timeout: 3000 in real configs was silently ignored.

Changes

  • s3.timeout (S3_TIMEOUT) → http.Transport.ResponseHeaderTimeout in buildHTTPTransport(). Counts from "request fully written" to "response headers received", so long PUT/GET bodies are not truncated. The AWS SDK retryer treats the timeout as retryable (Timeout() == true), so a lost UploadPartCopy is re-sent on a fresh connection instead of failing the backup.
  • Default 30m (well above any healthy 5 GB UploadPartCopy), empty or 0 disables it.
  • Accepts a duration string (30m) or a bare integer number of seconds (3000 → 50m), so existing configs with timeout: 3000 keep starting instead of failing validation. Invalid values fail ValidateConfig.
  • Propagated to object-disk S3 connections (object_disk.go), which do the UploadPartCopy during object-disk restore.
  • ReadMe, testflows default-config snapshot.

Note: create.go runs object-disk CopyObject with context.WithoutCancel, so /backup/kill cannot unstick that path — the client-side timeout is the only defense there.

Trade-off: WithoutCancel was added in 9a99a15 because a canceled single-request CopyObject keeps copying server-side and can create dstKey after the abort cleanup already ran, leaving an orphan object. The response-header timeout gives up waiting the same way a cancel does, so a single-request CopyObject (object < 5 GB) which times out may still land dstKey later. Multipart UploadPartCopy is safe: parts are invisible until CompleteMultipartUpload. This needs the server to stay silent for more than 30m, i.e. exactly the hang this PR guards against, so a rare orphan object is accepted in exchange for not hanging forever.

Tests

  • TestBuildHTTPTransportResponseHeaderTimeout: httptest server that accepts the request and never answers. Without the fix the test hangs until go test -timeout; with it the request fails with timeout awaiting response headers after 1s. Also checks default 30m, 3000 → 3000s, empty → disabled.
  • TestValidateConfigHTTPTimeouts, TestParseDurationOrSeconds.
  • make test passes.

🤖 Generated with Claude Code

The S3 client had no per-request deadline. A request which the server
accepted but never answered (UploadPartCopy during server-side copy of
object disk data on AWS S3, HTTP/1.1) waited in persistConn.roundTrip
forever: create_remote hung 5+ days in rebaseBackupRegular while the TCP
connection stayed ESTABLISHED and answered keepalives. The http2_*
timeouts from #1490 don't apply because AWS S3 doesn't negotiate HTTP/2.

s3.timeout (S3_TIMEOUT) never existed although the same key works for
azblob/cos/ftp, so `timeout: 3000` in existing configs was silently
ignored. Now it sets ResponseHeaderTimeout (doesn't cut long bodies, the
SDK retries the timeout error), defaults to 30m, accepts a duration
string or a bare integer number of seconds so existing configs keep
starting, and is propagated to object disk S3 connections used by
restore.

Fixes #1645

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Slach
Slach merged commit bebb4cc into master Oct 28, 2026
56 of 60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3: implement s3.timeout (S3_TIMEOUT) as HTTP response-header timeout — UploadPartCopy can hang forever (no per-request timeout)

1 participant