Repository navigation
A reservation is not renewed while its action runs, so after the lease a waiting duplicate runs the action too #52
Description
Versions: idempotency-kit 0.4.1, redis 8.1.0, Redis 7, Python 3.13.
A reservation is never renewed while its action runs. Once in_flight_lease_seconds passes, the pending record expires even though its owner is alive, and the next caller with the same key runs the action again. In "wait" mode this happens to a duplicate that arrived while the first call was running: it waits, sees the marker disappear, reserves the key and runs its own action.
Reproduction
import asyncio
import logging
from redis.asyncio import Redis
from idempotency_kit import AsyncIdempotencyCoordinator, IdempotencyDomainService, JsonResultAdapter
from idempotency_kit.infra.storage.redis.aio import RedisAsyncIdempotencyRepository
async def main():
redis = Redis.from_url("redis://127.0.0.1:6379/0")
coordinator = AsyncIdempotencyCoordinator(
RedisAsyncIdempotencyRepository(redis), IdempotencyDomainService(),
in_flight="wait", in_flight_lease_seconds=1,
)
loop = asyncio.get_running_loop()
started = loop.time()
effects = []
async def charge(tag, hold):
effects.append((tag, round(loop.time() - started, 2)))
await asyncio.sleep(hold)
return {"charge_id": tag}
first = asyncio.create_task(coordinator.coordinate(
"payment.charge", "lease-demo", 3600, JsonResultAdapter(), charge, "first", 3,
))
await asyncio.sleep(0.1)
second = await coordinator.coordinate(
"payment.charge", "lease-demo", 3600, JsonResultAdapter(), charge, "second", 0,
)
print("second returned", second, "effects", effects)
await first
await redis.aclose()
logging.basicConfig(level=logging.CRITICAL)
asyncio.run(main())Actual
second returned {'charge_id': 'second'} effects [('first', 0.0), ('second', 1.05)]
Two charges for one key, with the first call still running.
Why
The pending record expires at the owner's start plus the lease. The waiter's deadline is its own arrival plus the lease (core/services/aio/coordinator.py:204), which is later, so it is still waiting when the marker disappears, and then it reserves the key "as a waiter does when the marker disappears" (comment at coordinator.py:237). The expiry exists to recover from crashed owners, but without renewal a slow live owner looks exactly like a dead one. The docs answer this by saying the lease has to outlive the action, which pushes callers towards long leases and slow crash recovery.
Proposal
- Renew the pending record while the action runs: a heartbeat every lease/3 that extends the expiry, stopped on completion, failure or cancellation. Expiry then means the owner is gone.
- Base the waiter's deadline on the record's expiry, not on the waiter's arrival.
- Until then, document that
"wait"(and any call after the lease) can end in running the action a second time.
Activity
AlexeyShalaev commentedon Oct 16, 2026
Reproduced on master:
second returned {'charge_id': 'second'} effects [('first', 0.0), ('second', 1.05)].Plan: while the action runs under our reservation, a background task rewrites the pending record every lease/3 with
expires_atpushed to now + lease (samereplace()the completion uses,created_atkept). It lives only between the reservation and the end of the action, wrapped around theawait action(...)so every exit goes through it: on completion, failure and cancellation the task is told to stop and awaited before the result is written or the reservation deleted. It is stopped, not cancelled, on purpose: a renewalSETcancelled mid-flight could land after the completed record and put a pending marker back over it, which after a lease is a second run. A renewal that fails is counted asstorage_renew_errorand logged, and the action carries on; two misses in a row are absorbed by the lease/3 spacing, a longer outage loses the reservation, which is today's behaviour. With that, expiry means the owner is gone (process died, or the event loop was blocked for a whole lease), and the lease stops having to outlive the action.The waiter's deadline I am leaving as it is: a waiter still waits at most one lease from its arrival and then gets
IdempotencyInProgressError. With renewal the marker no longer disappears under a live owner, so the waiter can no longer fall through into a second run, which is the bug here. Basing the deadline on the record's expiry would, with renewal, make the wait unbounded behind an action that hangs; a bounded wait plus a 409 the client retries is the safer default. In your script the second call now raisesIdempotencyInProgressErrorafter about a second and the provider sees one charge.Rejected: a conditional renewal (compare-and-set on the pending record via Lua, or Redis 8's
SET IFEQ). It would stop a renewal from overwriting a record that is no longer ours, but that only happens after the reservation was already lost to a long storage outage, when the duplicate has already happened; it needs an owner token on the record and a new repository method every custom repository would have to implement. Tests go against real Redis through testcontainers with a 1 s lease and a 3 s action, plus the stop ordering, cancellation and a failing renewal.AlexeyShalaev commentedon Oct 16, 2026
#56 is up. Your script against that branch:
One charge. The owner renews its pending record every lease/3 while the action runs, and the renewal is stopped and awaited, not cancelled, before the result is written or the key released, so a renewal still on its way cannot land on top of the completed record. Expiry now means the owner is gone; I checked that with an owner process killed mid-action (the key frees a lease after the kill) and one shut down by
asyncio.run()mid-action (the key is released, no task warnings). A failed renewal is counted asstorage_renew_error; a store down for a whole lease still loses the reservation, as before.As planned, the waiter's deadline is still one lease from its arrival, so in
"wait"mode a duplicate behind a long action gets a 409-shapedIdempotencyInProgressErrorafter a lease rather than waiting indefinitely. If you want waiters to sit through long actions, raise the lease; it no longer has to cover the action itself, only how long a dead worker may block retries and how long a duplicate should wait.Nothing to change on your side. The one new rule is that renewal runs on the event loop, so blocking work inside the action belongs in
asyncio.to_thread; the docs say so.