Skip to content

fix: keep retry tasks rescheduled during execution - #1151

Open
alexluong wants to merge 2 commits into
mainfrom
fix/scheduler-receipt-delete
Open

alexluong wants to merge 2 commits into
mainfrom
fix/scheduler-receipt-delete

Conversation

@alexluong

@alexluong alexluong commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes a race condition in the retry scheduler between deleting a finished task and rescheduling it under the same ID.

The race

A retry task's ID is fixed per event and destination (RetryID), so scheduling the next retry replaces the queued task. When the delivery fails before the retry scheduler deletes the task it just ran, the delete removes the next retry:

retry scheduler                            delivery handler
(scheduler/scheduler.go)                   (deliverymq/messagehandler.go)
------------------------                   ------------------------------
receive task RetryID
exec: deliverymq.Publish(task)  -------->  Handle(task)
                                           delivery fails
                                           retryScheduler.Schedule(task,
                                             WithTaskID(RetryID))
                                           -> replaces the queued task
DeleteMessage(RetryID)
-> deletes the task just scheduled

Fix

Each receive gets a receipt. Delete and backoff act only while the receipt is current; rescheduling the task clears it.

receive task       -> receipt 41
schedule retry 2   -> receipt cleared
delete(ID, 41)     -> mismatch, retry 2 kept
  1. rsmq: ReceiveMessage returns QueueMessage.Receipt; SendMessage with a custom ID clears it.
  2. rsmq: DeleteMessageWithReceipt and ChangeMessageVisibilityWithReceipt, one Lua script each.
  3. scheduler: uses them after exec, in the exec-error backoff and when moving a task to the dead-letter queue.

🤖 Generated with Claude Code

@alexluong
alexluong force-pushed the fix/scheduler-receipt-delete branch from 8b09f7d to cd7fce8 Compare October 19, 2026 19:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants