Idempotency: Keys, Dedupe Stores, Exactly-Once Illusion, Outbox

Definition

An operation is idempotent if executing it more than once has the same effect as executing it once (with respect to business invariants). Idempotency keys are client- or server-supplied stable identifiers attached to a mutating request so retries do not double-charge, double-ship, or double-book.

A dedupe store (Redis, DB unique constraint, append-only log with uniqueness) records seen keys or request hashes with a TTL long enough to cover retry windows.

Exactly-once processing is an illusion in distributed systems at scale—you compose at-least-once delivery + idempotent effects + dedupe to get effectively exactly-once behavior.

Transactional outbox — Write business row + outbox event in one DB transaction; a relay process publishes to a message bus—avoids "DB committed but message lost" or dual-write races.

Why it matters in interviews

Networks drop, delay, and duplicate. Every payment, inventory, and signup flow must answer: what if the client retries? what if the server processed but the response was lost? Idempotency keys and dedupe are the standard adult answer; outbox is the standard reliable publish pattern.

Tradeoffs

• Idempotency keys require client cooperation or deterministic server keys (dangerous if wrong).
• Dedupe store — Memory/DB cost, TTL vs infinite dedupe (legal/finance may need longer retention).
• Outbox — Extra component (poller/CDC), ordering still not free across partitions.
• Global uniqueness — UUID vs ULID vs natural keys; avoid hot monotonic keys on a single counter if sharding.

Concrete examples

1. Stripe-style payment — Client sends Idempotency-Key header; server stores request fingerprint and returns same response on replay—dedupe at the API edge.
2. Order service — orderId + operation=Cancel is naturally idempotent; Create needs client-generated UUID or reservation token to dedupe POST.
3. Event consumer — At-least-once Kafka consumer uses DB insert with unique (topic, partition, offset) or idempotent upsert so rebalance replays do not duplicate side effects.

How to say it in 30 seconds

"Retries are normal. I make effects idempotent with keys stored in a dedupe table or cache with a TTL. True exactly-once is rare; I pair at-least-once with idempotent writes. For DB + queue, I use the outbox pattern in one transaction and a publisher—no fragile double writes."

Common follow-up questions

• What TTL for idempotency keys? Cover client retry + clock skew; 24–72 hours common for APIs; longer for finance if required.
• What breaks idempotency? Non-deterministic side effects (send email without dedupe), time-dependent logic without versioning.
• How does outbox differ from sagas? Outbox is reliable event emission; saga is long-running transaction orchestration—often combined.

Cross-links (building blocks)

• Message queues, event sourcing, CQRS, and distributed transactions all intersect idempotency—see System design curriculum overview.

---

See also: System design curriculum overview