Repository navigation
fix(inbox): roll back a runner handler that returns handler_retry() and complete a skipped row - #63
Merged
Merged
Conversation
This was referenced Oct 16, 2026
Merged
…nd complete a skipped row A handler passed to InboxConsumerRunner that returned handler_retry() committed its writes and left the inbox row pending and locked, so every redelivery skipped it until release_stale_locks freed the row and the handler ran a second time on top of the first. The runner now treats a returned failure like a raise: the transaction rolls back and the commit follows the handler-exception rule of the ack strategy. handler_skipped() marks the row completed, so a redelivery is a duplicate instead of a row locked forever, and AT_LEAST_ONCE + ON_SUCCESS commits a skip but not a returned failure. Closes #59
AlexeyShalaev
force-pushed
the
fix/59-runner-handler-retry
branch
from
October 16, 2026 15:31
556637e to
0639162
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #59.
Builds on #62, whose commit is the first one here. Merge #62 first, then rebase this branch with
git rebase origin/master. Git drops the #62 commit as already applied and replays the second one with no conflicts; I tried that against a local squash of #62. Without the rebase, GitHub's merge reports conflicts indocs/agents.mdandconsume.py, because both commits touch the same lines. The base ismasterso CI runs on it.Problem
Suppose a handler passed to
InboxConsumerRunnerreturnshandler_retry(). The transaction commits, so the handler's writes are kept, and the inbox row stayspendingwith the lock thatmark_processingset. Every redelivery finds the row locked and skips the handler, and it doesn't commit underEXACTLY_ONCE_INBOX. That goes on untilrelease_stale_locksfrees the row, and then the handler runs a second time on top of the first. The issue's script produced two invoices for one message.handler_skipped()ends up in the same place: the offset is committed, but the row stayspendingand locked forever, so retention never removes it.Fix
handler_retry(...), or anyEventHandlerResultwithsuccess=False, processed=True. The runner raises a private exception inside theasync with, so the handler's writes and the inbox row roll back together. It then goes down the existing handler-error path:processed=False,inc_failedonce, and the commit rule for a handler exception under the configured strategy. UnderEXACTLY_ONCE_INBOXthe message is not committed, fix(inbox): rewind a message the runner does not commit instead of reading past it #62 rewinds it, and the redelivery starts from a clean slate.handler_skipped()marks the rowcompletedin the same transaction. The message was handled by a decision not to act on it. A redelivery is now a duplicate, and the row is unlocked, so retention can delete it.AT_LEAST_ONCE+ON_SUCCESScommits a success or a skip, and not a returned failure. Before, it committed wheneverprocessed=True, which includedhandler_retry(), and did not commit a skip. That was the opposite of what the name says, and with the skipped row now completed, leaving it uncommitted would only cost a redelivery.count_as_attemptandnext_retry_athave no effect in the runner, because it records no failure (rule 5). This is documented next to the handler-results table instead of being logged on every message.Rejected
Keeping the inbox row and recording the failure with
mark_failed, so that "attempts_made + 1" in the handler-results table holds for the runner too. That needs the handler's writes rolled back without the row, which means a savepoint the core layer cannot open, since it doesn't know there is a session. It would also add a second retry source, the batch processor, next to the broker's redelivery.Behaviour change, who notices
EXACTLY_ONCE_INBOX+handler_retry(): the handler's writes no longer survive, and the message is redelivered instead of stalling on a locked row. This is the fix itself.AT_LEAST_ONCE+ON_SUCCESS:handler_retry()is no longer committed (it is redelivered), andhandler_skipped()now is. Two existing unit tests pinned the old behaviour. I renamed them and updated their assertions.AT_LEAST_ONCE+ON_PERSIST:handler_retry()now behaves the way a raise already did. The transaction rolls back and the offset is committed ("after the transaction, success or failure"), so the message is gone. Before, it left a lockedpendingrow with the handler's writes committed, which a batch processor could reach after its lease expired. I kept one rule for "the handler failed" instead of special-casing a return value. The loss on a handler failure underON_PERSISTpredates this PR and applies to raises as well. If it's unwanted, it deserves its own issue.handler_skipped()under any strategy: the row is nowcompletedinstead ofpendingand locked.Tests
tests/integration/test_atomicity.pyruns against real Postgres, with a consumer that redelivers the uncommitted message the way Kafka does:handler_retry()on the first call: after the first delivery there is no business row, no inbox row, and nothing committed. The redelivery runs the handler once more, which gives one business row and acompleted, unlocked inbox row.handler_skipped(): after the first delivery the row iscompletedand unlocked. The redelivery is a duplicate, it is committed, and the handler is not called again.Fails before: with
consume.pyfrom #62's commit swapped back in, both integration tests fail ((False, True) != (False, False)for the retry,pending != completedfor the skip), and so do 12 unit tests.The issue's script against this branch (it reads the inbox row with
one_or_none(), since after a retry there is none):make checkis clean. The full suite gives 899 passed, 3 skipped. Unit coverage is 98.8%.