Idempotency & at-least-once processing
Core details
At-least-once delivery: duplicates happen—handlers must converge.
Idempotency key pattern: deterministic business key + store outcome; replays short-circuit identical effects.
Transactional outbox: write business row + outbox event in one DB transaction → relay publishes reliably without “dual write” fantasy.
Inbox / dedupe table on consumer side for broker redeliveries.
Sagas / process managers: multi-step workflows with explicit compensations—not “distributed transactions” denial realism honest.
Problem this solves: retries, broker redeliveries, client double-clicks, and worker crashes create duplicate attempts. Business effects still need to converge to one intended result.
Simple mental model: delivery may happen many times; the externally visible effect should happen once.
Understanding
Retries—including client, gateway, and broker—multiply attempts. Without idempotent effects you get double charges, duplicate tickets, fan-out storms. “Exactly-once” rhetoric in interviews should be reframed honestly: effective-once via dedupe + deterministic side-effect design.
Ordering: single partition per entity often trades throughput for simpler reasoning—state when you’d relax ordering.
The visual model below shows the practical “effective-once” path: the producer records business state and an outbox row atomically, the relay publishes at least once, and the consumer uses an inbox/dedupe record so retries converge instead of duplicating side effects.
Senior understanding
Webhook story: verify signature before acknowledging; persist idempotency outcome before returning 2xx; classify transient vs permanent downstream errors to avoid infinite poison replays without DLQ observability.
Discuss clock skew minimally only if relevant—focus on key design + transactional boundaries first.
Minimal schema shape
| Table | Key fields | Purpose |
|---|---|---|
idempotency_keys | key, request_hash, status, response_ref, expires_at | reject conflicting replays; return stored outcome |
outbox_events | event_id, aggregate_id, type, payload, published_at | reliable publish after business commit |
consumer_inbox | consumer, event_id, processed_at | dedupe broker redelivery per consumer |
Store the idempotency record and the business effect in the same transactional boundary when possible. If they are split, the design needs a recovery story for “effect happened but outcome record was not saved.”
Common mistakes
- Treating HTTP retry middleware as harmless without idempotency keys.
- Returning
2xxto a webhook before the durable dedupe/effect boundary. - Using random request IDs when a deterministic business key is required.
- Keeping dedupe rows forever without TTL, archival, or partitioning.
- Calling broker “exactly once” and forgetting the external side effect.
Interview answer structure
“I assume at-least-once delivery and make the handler converge. The producer writes business state and an outbox row in one transaction. The relay may publish more than once. The consumer checks an inbox/dedupe table before applying the effect. For webhooks, I verify the signature first, persist the idempotency outcome before acknowledging, and route poison messages to a DLQ with alerting.”
Follow-ups to expect:
- What key would you use for payment capture versus email sending?
- How do you handle same key with different payload?
- What happens if the worker crashes after the external API call?
- How long do dedupe records live?
Diagram
See also
Mark this page when you finish learning it.
Spotted something unclear or wrong on this page?