# Idempotency & Exactly-Once (/docs/concepts/idempotency)



"Exactly-once" is one of the most abused phrases in distributed systems. Ductor
is precise about what it promises, because the difference between *state
progress* and *side effects* is exactly where systems quietly go wrong.

The short version:

* **State progress is exactly-once.** A run's durable state advances at most once
  per logical step, no matter how many times a tick runs.
* **Side-effect dispatch is at-least-once.** A connector call or downstream
  publish may be delivered more than once across a crash.
* **Idempotency records collapse the gap**, so that at-least-once dispatch
  becomes effectively-once for consumers that participate.

## Why the split exists [#why-the-split-exists]

Ductor commits a run's state to Postgres and *then* dispatches the resulting
work. These two acts cannot be a single atomic operation — one is a database
transaction, the other is a network call to some other system. So there is
always a window:

```text
   commit tick state   ──►   [crash here?]   ──►   dispatch side effect
```

If the process dies in that window, recovery re-derives what still needs doing
from durable state and dispatches again. That's *at-least-once* by construction,
and it's the honest guarantee for anything that leaves the database.

Ductor makes this the safe default by committing the *intent* first:
progress is never lost (it's durable before dispatch), and a redelivery is
always safe to reason about (the coordinator recomputes from committed state).
See [Commit, then dispatch](/docs/concepts/coordinator-workers#commit-then-dispatch).

## Exactly-once state progress [#exactly-once-state-progress]

State progress is exactly-once because of three reinforcing mechanisms:

1. **A single writer.** Only the Coordinator mutates run state. There is no
   concurrent second writer to race against.
2. **Optimistic locking.** Each tick commits with a conditional update on the
   run's `record_version`. If two ticks race, one commits and bumps the version;
   the other's conditional update matches zero rows and is rejected, then
   recomputed against fresh state. See
   [Optimistic Locking](/docs/concepts/optimistic-locking).
3. **Idempotent re-apply.** A tick that folds in an attempt result checks the
   attempt's stamp/token. Re-applying a result that was already applied is a
   no-op — it produces zero new rows. A monotonic transition sequence
   (`last_transition_seq`) enforces that history only moves forward.

Together these mean: replay a tick, redeliver a wakeup, or reprocess an attempt
as many times as you like — the run's durable state reaches the same place it
would have reached exactly once.

## At-least-once side effects [#at-least-once-side-effects]

Everything that leaves Postgres — a connector action, an emitted event, a
downstream publish — is dispatched *after* the state commit. If a crash lands
between commit and dispatch, recovery redrives the dispatch. For an external
system that isn't idempotent, that can mean a duplicate call.

This is not a bug to be fixed; it's the fundamental limit of talking to systems
you don't transactionally control. Ductor's job is to make the duplicate
*safe* and *rare*, and to give you the tools to collapse it.

## Collapsing the gap: idempotency records [#collapsing-the-gap-idempotency-records]

Ductor uses idempotency tables to turn at-least-once dispatch into
effectively-once processing.

### Routing ingress idempotency [#routing-ingress-idempotency]

The routing plane records every idempotent request in
`eec_routing_idempotency` (the `eec_` prefix marks Ductor's durable *Enterprise
Eventing Core* tables), keyed by `(tenant_scope, pool_id, idempotency_key)`.
The lifecycle is a small state machine:

```mermaid
stateDiagram-v2
    [*] --> pending: reserve
    pending --> completed
    pending --> failed_before_commit: reclaimable
```

* **Reserve** does an atomic `INSERT ... ON CONFLICT DO NOTHING`, with a
  reclaim clause that re-claims rows that are `failed_before_commit` or a
  `completed` row past its TTL. The first caller wins; duplicates see the row
  already exists.
* **Complete** upserts the row to `completed`, stamps the resulting decision ID,
  and sets a fresh validity TTL.
* Expired `pending` rows are swept to `failed_before_commit` so they can be
  reclaimed, and long-dead rows are purged by a retention worker.

You supply the key as the `idempotency_key` on the request. An empty key opts
out of dedup (always treated as a fresh request). Duplicate callers that arrive
while the first is still in flight are parked with a duplicate-waiter policy
(default: wait for the result) rather than racing.

### The effect-intent outbox [#the-effect-intent-outbox]

For workflow side effects, Ductor uses an **outbox**: the intent to perform a
side effect is written to `eec_effect_intent` inside the state-commit
transaction, and a drainer marks each intent `completed` after it fires. A
post-commit notify plus a periodic poll drive the drain. The outbox guarantees
each intent is *drained*; the external effect is still at-least-once unless the
downstream is idempotent — which is why the idempotency key matters.

<Callout type="warn" title="Make your downstream idempotent">
  Ductor guarantees exactly-once *progress* and gives you idempotency keys to
  deduplicate *dispatch*. For a truly exactly-once external effect, the receiving
  system must also honor an idempotency key (most payment and messaging APIs do).
  Pass one through and the at-least-once window closes.
</Callout>

## Putting it together [#putting-it-together]

| Layer                | Guarantee        | Mechanism                                    |
| -------------------- | ---------------- | -------------------------------------------- |
| Run state            | Exactly-once     | Sole writer + `record_version` + stamp dedup |
| Decision commit      | Exactly-once     | `eec_routing_idempotency` reserve/complete   |
| Side-effect dispatch | At-least-once    | Post-commit dispatch, reconciler redrive     |
| End-to-end effect    | Effectively-once | Idempotency key honored by the downstream    |

## Where to go next [#where-to-go-next]

* [Optimistic Locking](/docs/concepts/optimistic-locking) — the concurrency primitive underneath exactly-once progress.
* [Coordinator & Step Workers](/docs/concepts/coordinator-workers) — the commit-then-dispatch model.
* [Events & Event Sourcing](/docs/concepts/events) — the durable, append-only side of the system.