feat: batch retry attempt lookups

[alexluong]

Combine concurrent retry lookups into one log store query (opt-in), so a burst of retries costs ClickHouse a fraction of the CPU.

Stacked on #1154.

Problem

With the time-bounded lookup each retry costs ClickHouse ~14 ms of CPU, ~8-10 ms of which is fixed per-query overhead (planning, opening parts) rather than data. In the reproduction below (~8,400 retries), a 3 vCPU ClickHouse is still at its limit in 20 % of samples, and every retry holds a pool connection for its own query.

Idea

Retries that are due at the same time look up the same table with the same shape of query. A small coalescer in front of the lookup waits a few milliseconds for others, then fetches the latest attempt for all of them in one query. Each retry still gets its own result and is acked, backed off or dead-lettered on its own, exactly as before.

1. Log store: new LatestAttempts for a list of (tenant, event, destination, attempt time) keys, returning the latest attempt per key.
   • ClickHouse: the latest attempt_id per key from a grouped subquery, then the rows by attempt_id with an outer LIMIT and query_plan_max_limit_for_lazy_materialization, so payload columns are read only for the returned rows (other shapes measured 77-540 MiB per 100-key batch because they read payloads for whole granules; this one 2.6 MiB). The time bound is the oldest key's; each key's own bound is applied in Go. Duplicate rows (before merges) are deduplicated by key; if the result reaches the row limit, keys missing from it are looked up one by one. Servers that reject the setting (unknown, read-only or constrained) get the query without it.
   • Postgres: LATERAL over unnest(...) with each key's own bound: one index probe per key.
   • Memory driver: same semantics.
2. Coalescer in the retry path: flushes at RETRY_LOOKUP_BATCH_SIZE keys or RETRY_LOOKUP_BATCH_WAIT_MS after the first key, whichever comes first; splits a flush when the keys' attempt times span more than 15 min (one old task would widen the range for all). A batch of one uses the existing single lookup. Tasks without AttemptTime don't join batches. A key missing from the result is "no prior attempt", the same not-ready path as a single lookup.
3. Config: RETRY_LOOKUP_BATCH_SIZE (default 0 = off, max 100) and RETRY_LOOKUP_BATCH_WAIT_MS (default 50). One row each in the configuration reference.

Off by default: per-retry behavior is the same, but a failed batch query fails every retry in it (each goes through its normal backoff), and the efficient ClickHouse path needs ClickHouse 25.x.

Bench

Bounded lookups on the same 8 M-row table, ClickHouse at its 3 vCPU limit (50-200 concurrent queries); CPU per looked-up retry:

lookup	CPU per lookup	lookups/s
single, bounded	~14 ms	~200
batch of 10	~4 ms	~730
batch of 100	~1 ms	~2,800

Results

Local reproduction, 18,000 events in 60 s, destination failing for the first 20 s, batching off = the #1154 build, batching on = this branch with RETRY_LOOKUP_BATCH_SIZE=50:

	batching off	batch size 50, wait 50 ms
lookup queries	8,416	1,286 (for ~12,000 retries, ~9 per query)
ClickHouse CPU, mean over the run	73 %	29 %
ClickHouse CPU, peak	309 %	252 %
share of samples at the 3 vCPU limit	20 %	0 %
ClickHouse connection pool timeouts	0	0
events delivered after the destination recovered	18,000 / 18,000	18,000 / 18,000

The batched run did more retries (12,004 vs 8,416 failed deliveries): retries came back faster and more of them landed inside the 20 s failure window.

How the numbers were measured

Local reproduction: ClickHouse 25.8 limited to 3 vCPU with 8 M rows in the attempts table (30 days, ~90 KB average payload), 3 Outpost instances (all services in each, 2 CPUs, 2 GiB), Redis-compatible store for the retry queue, NATS as the internal queue. Retry schedule 1, 5, 300, 1800, ... seconds, LOG_BATCH_SIZE=50, LOG_BATCH_THRESHOLD_SECONDS=10, RETRY_VISIBILITY_TIMEOUT_SECONDS=30. Workload: 18,000 events (12 KB payload) in 60 s to one webhook destination that returns 5xx for the first 20 s, then 200 (about 6,000 failed first attempts, so about 6,000 retries due within ~20 s). Lookup cost from system.query_log; ClickHouse CPU from the container's cgroup. One run per build; three runs of the main baseline agreed within 3 % on pool timeouts and delivered events (lookup p50 duration 3.2-3.6 s).

Tests

• Log store driver suite (LatestAttempts, ClickHouse, Postgres, memory): latest attempt per key, newer-than-carried found, older excluded, absent keys missing, same event with different destinations not cross-matched.
• ClickHouse: five unmerged copies of one key's latest row plus an older second key with merges stopped: both keys returned (fails without the row-limit fallback).
• Coalescer (testing/synctest where time matters): flush by count and by time, split by span, error fan-out, per-caller cancel, batch of one, tasks without AttemptTime, disabled config, close.

Not in this PR

• The CI ClickHouse image (24.10) doesn't have query_plan_max_limit_for_lazy_materialization, so CI covers the fallback query; the measurements above are on 25.8.

🤖 Generated with Claude Code