Durability Model: Coordinator & Workers
The architectural view of Ductor's central runtime pattern — serial state ownership, parallel execution, and the durable boundary between them.
This page assumes the mental model from Coordinator & Step Workers and goes into the internals: why the split is shaped this way, what the durability boundary buys, and how the pieces are wired.
One serial owner, many parallel executors
The design is a deliberate separation of two concerns that most workflow engines tangle together:
- Deciding what happens next is inherently sequential for a given run — it's
a state transition, and two transitions on the same state must be ordered. The
Coordinator owns this. It is the sole writer of
eec_workflow_run(theeec_prefix marks Ductor's durable Enterprise Eventing Core tables). - Doing the work is embarrassingly parallel — connector calls, rule evaluation, user logic. Step Workers own this, across the whole fleet, and they only ever append results.
By assigning "decide" to a single serial writer and "do" to many append-only workers, the corrupting-write class of bug is designed out. There is no lock to tune and no two-writer race to reason about, because there is only ever one writer of the thing that must be serialized.
The durability boundary
The heart of the model is a single ordering rule:
Everything to the left of the commit is a pure function of loaded state. Everything to the right is a best-effort, redrivable side effect. The commit is the durability boundary: state is durable before any work is dispatched.
This placement is what produces Ductor's two headline guarantees:
- Exactly-once state progress — the pure tick plus a version-checked commit means a run's durable state advances at most once per logical step, no matter how many times the tick runs. See Optimistic Locking.
- At-least-once side effects — dispatch happens after the commit, so a crash in between causes a safe redrive. See Idempotency & Exactly-Once.
Purity is the enabler
The tick computes over a clone of the state and emits a payload; it doesn't mutate anything or call out to the world. That purity is what makes a conflict retry trivial — just recompute against fresh state — and what keeps replay deterministic.
Wiring: how a run gets ticked
Something needs a run to advance — a worker finished an attempt, a timer fired, a signal arrived, or a new run was created.
A coordinator wakeup is enqueued. It travels either through SyncMatch (in-process, zero-latency) when a worker is co-located, or through the durable tiered queue on a dedicated system partition.
A coordinator worker goroutine dequeues the wakeup and runs one
Coordinator.Tick().
The tick loads state, computes a payload, commits under record_version, and
dispatches follow-on work post-commit.
On a version conflict, the tick retries in-process with backoff, then requeues through the tiered queue — bounded against livelock. See Optimistic Locking.
Because each run is ticked serially but runs are independent, the system scales horizontally: add pods and more runs make progress in parallel, while any single run stays strictly ordered.
Where the two roles write
| Role | Writes | Reads |
|---|---|---|
| Coordinator | eec_workflow_run (state, record_version), inserts eec_workflow_step_attempt rows on dispatch, transitions | attempt results, signals, timers, child runs |
| Step Worker | attempt results into eec_workflow_step_attempt | its dispatched task |
Note the asymmetry: workers never write run state, and the coordinator is the one inserting the attempt row when it dispatches — the worker fills in the result. A stale worker from a superseded attempt can't clobber a newer one, because the node's attempt generation is bumped on retry.
Failure modes it's designed against
- Double-write corruption → impossible: one writer, version-checked.
- Lost progress on crash → impossible: state is committed before dispatch.
- Stuck runs → bounded: a stuck-run watchdog and a conflict-livelock bound force-fail a run that can't make progress, visibly, rather than letting it hang.
- Unbounded history → bounded: ContinueAsNew and transition compaction cap run growth.
Where to go next
- Coordinator & Step Workers — the concept-level walkthrough.
- Data Flow — where run state, attempts, and counters physically live.
- Optimistic Locking — the commit's concurrency primitive.