Ask an experienced distributed-systems engineer about “exactly-once delivery” and you’ll often get a knowing smile: at the network level, it doesn’t exist. Messages get lost, retried, and redelivered. A client double-clicks. A gateway times out and your service never finds out whether the call landed.
And yet, when the message is “move $500”, the result has to be exactly once. The money moves a single time, or the whole system loses trust. The trick is that exactly-once isn’t a property of the network — it’s a property you build, at the layer where it matters.
Idempotency is the foundation
Every state-changing operation in a payment system should carry a client-supplied idempotency key: a unique identifier for that specific intent. The rule is simple and absolute:
The same idempotency key must produce the same result, exactly once, no matter how many times it arrives.
The first request with a given key does the work and records the outcome. Every subsequent request with that key returns the recorded outcome — it does not redo the work. A retried “charge $500” returns the original receipt; it does not charge again.
This one discipline turns an unreliable, at-least-once world into an effectively exactly-once one, without requiring the impossible from the network.
Make the ledger the source of truth
Idempotency keys protect the entrance. The ledger protects the truth. We build on a few non-negotiables:
- Double-entry accounting. Every movement is a balanced pair of entries. The books must sum to zero. If they don’t, something is wrong — and you find out immediately, not at month-end.
- Append-only history. You never edit a posted entry. Corrections are new entries. The full history is immutable and auditable.
- Atomic balance updates. The debit, the credit, and the idempotency record commit in a single transaction. Either all of it happened, or none of it did.
Handle the in-between state honestly
The hardest cases live in the gaps: the request that timed out, the worker that died mid-transaction, the callback that never came. The answer is to make “unknown” a first-class state rather than a guess.
- Record an operation as pending before you attempt it, keyed by its idempotency key.
- Attempt the work. On success, transition to completed in the same transaction.
- If you crash in between, recovery re-reads the pending record and either completes it or marks it failed — deterministically, using the key to avoid doing it twice.
A process that can be safely re-run from any point is one you can recover without fear. That’s what lets you retry aggressively instead of hoping.
Reconcile, always
Even with all of the above, you reconcile — continuously, automatically, and against external sources of truth like your payment providers. Reconciliation isn’t an admission that the system is wrong; it’s the control that proves it’s right. When every balance ties out to the cent, every day, you’ve earned the one thing a money system can’t function without: the confidence that the numbers are real.
Exactly-once, in the end, isn’t magic. It’s idempotency keys, an honest ledger, recoverable state, and the humility to keep checking. Discipline, applied everywhere it counts.