Idempotency & Exactly-Once
What Ductor guarantees under retries and crashes — exactly-once state progress, at-least-once side effects, and how to collapse the gap.
"Exactly-once" is one of the most abused phrases in distributed systems. Ductor is precise about what it promises, because the difference between state progress and side effects is exactly where systems quietly go wrong.
The short version:
- State progress is exactly-once. A run's durable state advances at most once per logical step, no matter how many times a tick runs.
- Side-effect dispatch is at-least-once. A connector call or downstream publish may be delivered more than once across a crash.
- Idempotency records collapse the gap, so that at-least-once dispatch becomes effectively-once for consumers that participate.
Why the split exists
Ductor commits a run's state to Postgres and then dispatches the resulting work. These two acts cannot be a single atomic operation — one is a database transaction, the other is a network call to some other system. So there is always a window:
commit tick state ──► [crash here?] ──► dispatch side effectIf the process dies in that window, recovery re-derives what still needs doing from durable state and dispatches again. That's at-least-once by construction, and it's the honest guarantee for anything that leaves the database.
Ductor makes this the safe default by committing the intent first: progress is never lost (it's durable before dispatch), and a redelivery is always safe to reason about (the coordinator recomputes from committed state). See Commit, then dispatch.
Exactly-once state progress
State progress is exactly-once because of three reinforcing mechanisms:
- A single writer. Only the Coordinator mutates run state. There is no concurrent second writer to race against.
- Optimistic locking. Each tick commits with a conditional update on the
run's
record_version. If two ticks race, one commits and bumps the version; the other's conditional update matches zero rows and is rejected, then recomputed against fresh state. See Optimistic Locking. - Idempotent re-apply. A tick that folds in an attempt result checks the
attempt's stamp/token. Re-applying a result that was already applied is a
no-op — it produces zero new rows. A monotonic transition sequence
(
last_transition_seq) enforces that history only moves forward.
Together these mean: replay a tick, redeliver a wakeup, or reprocess an attempt as many times as you like — the run's durable state reaches the same place it would have reached exactly once.
At-least-once side effects
Everything that leaves Postgres — a connector action, an emitted event, a downstream publish — is dispatched after the state commit. If a crash lands between commit and dispatch, recovery redrives the dispatch. For an external system that isn't idempotent, that can mean a duplicate call.
This is not a bug to be fixed; it's the fundamental limit of talking to systems you don't transactionally control. Ductor's job is to make the duplicate safe and rare, and to give you the tools to collapse it.
Collapsing the gap: idempotency records
Ductor uses idempotency tables to turn at-least-once dispatch into effectively-once processing.
Routing ingress idempotency
The routing plane records every idempotent request in
eec_routing_idempotency (the eec_ prefix marks Ductor's durable Enterprise
Eventing Core tables), keyed by (tenant_scope, pool_id, idempotency_key).
The lifecycle is a small state machine:
- Reserve does an atomic
INSERT ... ON CONFLICT DO NOTHING, with a reclaim clause that re-claims rows that arefailed_before_commitor acompletedrow past its TTL. The first caller wins; duplicates see the row already exists. - Complete upserts the row to
completed, stamps the resulting decision ID, and sets a fresh validity TTL. - Expired
pendingrows are swept tofailed_before_commitso they can be reclaimed, and long-dead rows are purged by a retention worker.
You supply the key as the idempotency_key on the request. An empty key opts
out of dedup (always treated as a fresh request). Duplicate callers that arrive
while the first is still in flight are parked with a duplicate-waiter policy
(default: wait for the result) rather than racing.
The effect-intent outbox
For workflow side effects, Ductor uses an outbox: the intent to perform a
side effect is written to eec_effect_intent inside the state-commit
transaction, and a drainer marks each intent completed after it fires. A
post-commit notify plus a periodic poll drive the drain. The outbox guarantees
each intent is drained; the external effect is still at-least-once unless the
downstream is idempotent — which is why the idempotency key matters.
Make your downstream idempotent
Ductor guarantees exactly-once progress and gives you idempotency keys to deduplicate dispatch. For a truly exactly-once external effect, the receiving system must also honor an idempotency key (most payment and messaging APIs do). Pass one through and the at-least-once window closes.
Putting it together
| Layer | Guarantee | Mechanism |
|---|---|---|
| Run state | Exactly-once | Sole writer + record_version + stamp dedup |
| Decision commit | Exactly-once | eec_routing_idempotency reserve/complete |
| Side-effect dispatch | At-least-once | Post-commit dispatch, reconciler redrive |
| End-to-end effect | Effectively-once | Idempotency key honored by the downstream |
Where to go next
- Optimistic Locking — the concurrency primitive underneath exactly-once progress.
- Coordinator & Step Workers — the commit-then-dispatch model.
- Events & Event Sourcing — the durable, append-only side of the system.