fix: cap retry scheduler concurrency and jitter its backoff

[alexluong]

Keep the retry scheduler steady when many retries come due at once: bounded concurrency, jittered backoff, and limits that can be configured.

Problem

A burst of failed deliveries schedules thousands of retries that come due within seconds. Three things in the scheduler make that worse:

1. Monitor starts one goroutine per received message (go s.exec(...), no limit). In a local load test with ~6,000 retries due within 20 s, 3 instances reached ~3,400 goroutines in total, ~3,000 of them waiting for a ClickHouse connection (10 per instance). The client's pool wait is bounded by the dial timeout (10 s), so most lookups failed with acquire conn timeout without reaching ClickHouse.
2. A failed exec puts the task back with a visibility of 30 s, doubling per receive, with no jitter. Tasks that failed together come back together, so the burst repeats after each backoff step (30, 60, 120, 240 s).
3. After 5 receives (a constant) the task is dead-lettered, and a task that can't be parsed also takes all 5 receives to get there.

Change

1. Concurrency cap. RETRY_MAX_CONCURRENCY (default 100 per process, 0 = unbounded). The monitor takes a slot before it receives, so it never holds a message it can't run; the rest stay in the queue.
2. Jittered backoff. The exec-error backoff is multiplied by a random factor in [0.75, 1.25] (also at the 15 min maximum), so a group of tasks that failed together spreads out.
3. Malformed tasks. A task whose body can't be parsed (bad JSON or missing IDs) goes to the dead-letter queue on its first receive. Log store errors still retry as before.
4. Max receive count. RETRY_MAX_RECEIVE_COUNT (default 5, today's value; minimum 1).
5. ClickHouse pool. CLICKHOUSE_MAX_OPEN_CONNS (default 0 = the client's default of 10) and CLICKHOUSE_DIAL_TIMEOUT_SECONDS (default 10). clickhouse-go uses the dial timeout for the pool wait as well, so this is also the acquire timeout.

Each new option is one row in the configuration reference.

Behavior changes

• Retries now run at most 100 at a time per process by default (before: unbounded). This doesn't reduce throughput in practice: lookups already queue on the ClickHouse pool (10 connections), and a cap of 100 only binds before the pool does if the rest of a retry (the publish) takes ~10x longer than its lookup.
• Exec-error backoff is no longer exactly 30 s x 2^n; it varies by ±25 %.
• Unparseable retry tasks are dead-lettered on the first receive instead of the fifth.
• ClickHouse defaults are unchanged.

Results

Local reproduction, 18,000 events in 60 s, destination failing for the first 20 s, run on this branch (includes the receipt fix from the base PR; the retry lookup is still the unbounded one from main):

	main	this PR
ClickHouse connection pool timeouts	25,601	8,603
lookups that completed	1,343	6,108
events delivered after the destination recovered	13,242 / 18,000	18,000 / 18,000
time from the start of the burst until all events were delivered	not reached in 19 min	~15 min
ClickHouse CPU	300 % in waves at 0-150, 240-270, 480-520 s	216 % mean, at the 3 vCPU limit for ~10 min (no waves)

With the unbounded lookup ClickHouse still does ~0.3 s of CPU per retry, so it stays at its limit until the backlog drains, and the remaining pool timeouts come from 300 retries in flight (3 instances x 100) sharing 30 connections. The time-bounded lookup (#1152) removes that cost: the stack with #1152 and #1154 has no pool timeouts and delivers all events in ~5 min.

How the numbers were measured

Local reproduction: ClickHouse 25.8 limited to 3 vCPU with 8 M rows in the attempts table (30 days, ~90 KB average payload), 3 Outpost instances (all services in each, 2 CPUs, 2 GiB), Redis-compatible store for the retry queue, NATS as the internal queue. Retry schedule 1, 5, 300, 1800, ... seconds, LOG_BATCH_SIZE=50, LOG_BATCH_THRESHOLD_SECONDS=10, RETRY_VISIBILITY_TIMEOUT_SECONDS=30. Workload: 18,000 events (12 KB payload) in 60 s to one webhook destination that returns 5xx for the first 20 s, then 200 (about 6,000 failed first attempts, so about 6,000 retries due within ~20 s). Lookup cost from system.query_log; ClickHouse CPU from the container's cgroup. One run per build; three runs of the main baseline agreed within 3 % on pool timeouts and delivered events (lookup p50 duration 3.2-3.6 s).

Tests

• internal/scheduler/scheduler_load_test.go: a burst larger than the cap never runs more than the cap at once (fails with the cap disabled).
• internal/scheduler/backoff_test.go: jitter bounds, including at the maximum.
• internal/deliverymq/retry_dlq_test.go: malformed tasks dead-lettered on the first receive; log store errors still retried; max receive count honored.
• internal/config/config_retry_test.go and internal/clickhouse/clickhouse_test.go: parsing, defaults, validation and pool option wiring.

Not in this PR

• Treating "attempt not written yet" as not ready instead of a failure (fix: requeue retries whose attempt is not persisted yet without counting receives #1154).

🤖 Generated with Claude Code