Repository navigation
s3: implement s3.timeout (S3_TIMEOUT) as HTTP response-header timeout - #1646
Merged
Merged
Conversation
The S3 client had no per-request deadline. A request which the server accepted but never answered (UploadPartCopy during server-side copy of object disk data on AWS S3, HTTP/1.1) waited in persistConn.roundTrip forever: create_remote hung 5+ days in rebaseBackupRegular while the TCP connection stayed ESTABLISHED and answered keepalives. The http2_* timeouts from #1490 don't apply because AWS S3 doesn't negotiate HTTP/2. s3.timeout (S3_TIMEOUT) never existed although the same key works for azblob/cos/ftp, so `timeout: 3000` in existing configs was silently ignored. Now it sets ResponseHeaderTimeout (doesn't cut long bodies, the SDK retries the timeout error), defaults to 30m, accepts a duration string or a bare integer number of seconds so existing configs keep starting, and is propagated to object disk S3 connections used by restore. Fixes #1645 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1645
Problem
The S3 backend had no per-request deadline. On Altinity.Cloud production
create_remote --diff-from-remotehung 5+ days: workers blocked innet/http.(*persistConn).roundTripinsideS3.CopyObject→UploadPartCopy(10.7 GB object-disk blob), CPU ~0, TCP connection to S3ESTABLISHEDwithSend-Q=0, keepalives answered — the server accepted the request and never sent response headers. Thehttp2_*timeouts from #1490 don't help: AWS S3 speaks HTTP/1.1.s3.timeout/S3_TIMEOUTnever existed (unlike azblob/cos/ftp), sotimeout: 3000in real configs was silently ignored.Changes
s3.timeout(S3_TIMEOUT) →http.Transport.ResponseHeaderTimeoutinbuildHTTPTransport(). Counts from "request fully written" to "response headers received", so long PUT/GET bodies are not truncated. The AWS SDK retryer treats the timeout as retryable (Timeout() == true), so a lostUploadPartCopyis re-sent on a fresh connection instead of failing the backup.30m(well above any healthy 5 GBUploadPartCopy), empty or0disables it.30m) or a bare integer number of seconds (3000→ 50m), so existing configs withtimeout: 3000keep starting instead of failing validation. Invalid values failValidateConfig.object_disk.go), which do theUploadPartCopyduring object-disk restore.default-configsnapshot.Note:
create.goruns object-diskCopyObjectwithcontext.WithoutCancel, so/backup/killcannot unstick that path — the client-side timeout is the only defense there.Trade-off:
WithoutCancelwas added in 9a99a15 because a canceled single-requestCopyObjectkeeps copying server-side and can createdstKeyafter the abort cleanup already ran, leaving an orphan object. The response-header timeout gives up waiting the same way a cancel does, so a single-requestCopyObject(object < 5 GB) which times out may still landdstKeylater. MultipartUploadPartCopyis safe: parts are invisible untilCompleteMultipartUpload. This needs the server to stay silent for more than 30m, i.e. exactly the hang this PR guards against, so a rare orphan object is accepted in exchange for not hanging forever.Tests
TestBuildHTTPTransportResponseHeaderTimeout: httptest server that accepts the request and never answers. Without the fix the test hangs untilgo test -timeout; with it the request fails withtimeout awaiting response headersafter 1s. Also checks default30m,3000→ 3000s, empty → disabled.TestValidateConfigHTTPTimeouts,TestParseDurationOrSeconds.make testpasses.🤖 Generated with Claude Code