Skip to content

fix(inbox): roll back a runner handler that returns handler_retry() and complete a skipped row - #63

Merged
AlexeyShalaev merged 1 commit into
masterfrom
fix/59-runner-handler-retry
Oct 16, 2026
Merged

AlexeyShalaev merged 1 commit into
masterfrom
fix/59-runner-handler-retry

Conversation

@AlexeyShalaev

@AlexeyShalaev AlexeyShalaev commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Closes #59.

Builds on #62, whose commit is the first one here. Merge #62 first, then rebase this branch with git rebase origin/master. Git drops the #62 commit as already applied and replays the second one with no conflicts; I tried that against a local squash of #62. Without the rebase, GitHub's merge reports conflicts in docs/agents.md and consume.py, because both commits touch the same lines. The base is master so CI runs on it.

Problem

Suppose a handler passed to InboxConsumerRunner returns handler_retry(). The transaction commits, so the handler's writes are kept, and the inbox row stays pending with the lock that mark_processing set. Every redelivery finds the row locked and skips the handler, and it doesn't commit under EXACTLY_ONCE_INBOX. That goes on until release_stale_locks frees the row, and then the handler runs a second time on top of the first. The issue's script produced two invoices for one message. handler_skipped() ends up in the same place: the offset is committed, but the row stays pending and locked forever, so retention never removes it.

Fix

  • A returned failure is now handled exactly like a raise. That means handler_retry(...), or any EventHandlerResult with success=False, processed=True. The runner raises a private exception inside the async with, so the handler's writes and the inbox row roll back together. It then goes down the existing handler-error path: processed=False, inc_failed once, and the commit rule for a handler exception under the configured strategy. Under EXACTLY_ONCE_INBOX the message is not committed, fix(inbox): rewind a message the runner does not commit instead of reading past it #62 rewinds it, and the redelivery starts from a clean slate.
  • handler_skipped() marks the row completed in the same transaction. The message was handled by a decision not to act on it. A redelivery is now a duplicate, and the row is unlocked, so retention can delete it.
  • AT_LEAST_ONCE + ON_SUCCESS commits a success or a skip, and not a returned failure. Before, it committed whenever processed=True, which included handler_retry(), and did not commit a skip. That was the opposite of what the name says, and with the skipped row now completed, leaving it uncommitted would only cost a redelivery.
  • count_as_attempt and next_retry_at have no effect in the runner, because it records no failure (rule 5). This is documented next to the handler-results table instead of being logged on every message.

Rejected

Keeping the inbox row and recording the failure with mark_failed, so that "attempts_made + 1" in the handler-results table holds for the runner too. That needs the handler's writes rolled back without the row, which means a savepoint the core layer cannot open, since it doesn't know there is a session. It would also add a second retry source, the batch processor, next to the broker's redelivery.

Behaviour change, who notices

  • EXACTLY_ONCE_INBOX + handler_retry(): the handler's writes no longer survive, and the message is redelivered instead of stalling on a locked row. This is the fix itself.
  • AT_LEAST_ONCE + ON_SUCCESS: handler_retry() is no longer committed (it is redelivered), and handler_skipped() now is. Two existing unit tests pinned the old behaviour. I renamed them and updated their assertions.
  • AT_LEAST_ONCE + ON_PERSIST: handler_retry() now behaves the way a raise already did. The transaction rolls back and the offset is committed ("after the transaction, success or failure"), so the message is gone. Before, it left a locked pending row with the handler's writes committed, which a batch processor could reach after its lease expired. I kept one rule for "the handler failed" instead of special-casing a return value. The loss on a handler failure under ON_PERSIST predates this PR and applies to raises as well. If it's unwanted, it deserves its own issue.
  • handler_skipped() under any strategy: the row is now completed instead of pending and locked.

Tests

  • tests/integration/test_atomicity.py runs against real Postgres, with a consumer that redelivers the uncommitted message the way Kafka does:
    • handler_retry() on the first call: after the first delivery there is no business row, no inbox row, and nothing committed. The redelivery runs the handler once more, which gives one business row and a completed, unlocked inbox row.
    • handler_skipped(): after the first delivery the row is completed and unlocked. The redelivery is a duplicate, it is committed, and the handler is not called again.
  • Unit tests cover the commit decision for every strategy and result, the metrics, and the rewind.

Fails before: with consume.py from #62's commit swapped back in, both integration tests fail ((False, True) != (False, False) for the retry, pending != completed for the skip), and so do 12 unit tests.

The issue's script against this branch (it reads the inbox row with one_or_none(), since after a retry there is none):

delivery 1: committed=False processed=False handler_calls=1 invoices=0 inbox=None
delivery 2: committed=True processed=True handler_calls=2 invoices=1 inbox=(completed, None)
delivery 3: committed=True processed=False handler_calls=2 invoices=1 inbox=(completed, None)
release_stale_locks(1) -> 0
delivery 4: committed=True processed=False handler_calls=2 invoices=1 inbox=(completed, None)

make check is clean. The full suite gives 899 passed, 3 skipped. Unit coverage is 98.8%.

…nd complete a skipped row

A handler passed to InboxConsumerRunner that returned handler_retry() committed its writes and left the inbox row pending and locked, so every redelivery skipped it until release_stale_locks freed the row and the handler ran a second time on top of the first. The runner now treats a returned failure like a raise: the transaction rolls back and the commit follows the handler-exception rule of the ack strategy. handler_skipped() marks the row completed, so a redelivery is a duplicate instead of a row locked forever, and AT_LEAST_ONCE + ON_SUCCESS commits a skip but not a returned failure.

Closes #59
@AlexeyShalaev
AlexeyShalaev force-pushed the fix/59-runner-handler-retry branch from 556637e to 0639162 Compare October 16, 2026 15:31
@AlexeyShalaev
AlexeyShalaev merged commit e78d84c into master Oct 16, 2026
6 checks passed
@AlexeyShalaev
AlexeyShalaev deleted the fix/59-runner-handler-retry branch October 16, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A runner handler that returns handler_retry() commits its writes and leaves the inbox row locked, so the message is never committed

1 participant