fix: requeue retries whose attempt is not persisted yet without counting receives

[alexluong]

Retries wait at least one log flush, and a retry whose failed attempt still isn't in the log store is requeued without counting the receive toward the dead-letter limit.

The race (known, accepted)

Attempts reach the log store in batches, up to LOG_BATCH_THRESHOLD_SECONDS (default 10 s) after delivery, and nothing orders the retry after that write. A retry that runs before the write lands (a short schedule step, or a log write backlog) finds nothing:

delivery handler          retry scheduler            log store
----------------          ---------------            ---------
attempt fails
logmq.Publish(attempt) -------------------------->  batched
Schedule(retry, 1s) --->  receive retry
                          ListAttempt: not found
                                                     flush (up to
                                                     LOG_BATCH_THRESHOLD_SECONDS)

This is expected behavior and stays: the retry and the log write are separate pipelines, and there is no clear way to order them without coupling the two. This PR makes the retry wait for the write instead of treating the miss as a failure. The floor (4.) covers the usual flush; the requeue covers a backlog.

Before, a miss was an exec error: the task backed off 30 s, 60 s, ... and each miss counted toward RETRY_MAX_RECEIVE_COUNT, so a few seconds of flush delay cost at least 30 s of retry delay, and a long one dead-lettered the task.

Fix

retry finds no prior attempt	before	after
attempt within the grace window	exec error: backoff, counted	requeued, not counted; the wait grows with the attempt's age (flush threshold up to 5 min, jittered)
attempt older than the grace window	exec error: backoff, counted	same, plus an error log
task without AttemptTime (scheduled by v1.6.0 or earlier)	exec error: backoff, counted	same

Grace window: max(15 min, 3 × LOG_BATCH_THRESHOLD_SECONDS) since the failed attempt, how long a log write may lag before the retry stops waiting. A task whose attempt is never written still reaches the dead-letter queue.

1. rsmq: RequeueMessage, one Lua script: sets the visibility and decrements the receive count, only while the receipt still matches.
2. scheduler: scheduler.RetryLater(delay, err) from exec requeues the task through it.
3. deliverymq: the retry exec returns RetryLater for a missing attempt within the grace window.
4. Retry delays are at least LOG_BATCH_THRESHOLD_SECONDS, so the usual flush lands before the retry runs; the requeue covers lag beyond that.

Notes

• RETRY_SCHEDULE steps shorter than LOG_BATCH_THRESHOLD_SECONDS (default 10 s) now wait that long. The default exponential schedule (30 s base) is unaffected.

🤖 Generated with Claude Code