Skip to content

fix(rest): retry GET connection failures with bounded backoff - #1272

Merged
JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/rest-get-connect-retry
Nov 1, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/rest-get-connect-retry

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What

Retry REST GET connection failures (including DNS resolution failures) at most
twice, with jittered exponential backoff. Every attempt retains the existing
180-second connect/read timeouts; there is no new whole-request deadline.
Connection retries share the existing GET
attempt limit. The 429/503 policy and POST/DELETE paths are unchanged.

Why

The existing status retry loop returns immediately when reqwest fails before
receiving a response. A transient resolver failure can therefore abort native
catalog planning. This adds bounded tolerance, not a fix for persistent DNS
misconfiguration. No new dependency or configuration is introduced.

Tests

Added deterministic resolver tests for recovery, retry exhaustion, no mutation
retry, and one DNS failure followed by response headers delayed for six seconds.
Existing REST client tests are also included in the run.

  • cargo fmt --all --check and git diff --check: passed.
  • cargo test -p paimon --lib api::rest_client::tests --no-default-features --features storage-memory,storage-fs: 12 passed on head 7c44f424, including delayed response headers after DNS recovery.
  • No production DNS recovery or throughput claim.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review November 1, 2026 14:03

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 37bfa21 into apache:main Nov 1, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants