# Durable Execution Workers (/docs/concepts/coordinator-workers)



Ductor's runtime follows a **coordinator-plus-worker** model. It exists to
answer one hard question — *how do you run a workflow across many machines
without two of them corrupting the same run state?* — with a structural answer
rather than careful locking everywhere.

This page is the **working mental model** — enough to reason about how a run
makes progress. The internals it deliberately glosses over — the exact locking
protocol, the table schema, and the crash-and-recovery windows — live in
[The Coordinator-Worker Model, in depth](/docs/architecture/coordinator-worker-model).

## Two roles [#two-roles]

### The Coordinator (serial, single writer) [#the-coordinator-serial-single-writer]

The Coordinator is the **only** component allowed to mutate a run's state row
(`eec_workflow_run`). It runs one **tick** per wakeup, and a tick is the atomic
unit of progress for a run.

Ductor's durable runtime tables share an `eec_` prefix — short for *Enterprise
Eventing Core*, its event, task, and workflow store (see the
[glossary](/docs/reference/glossary)).

### Step Workers (parallel, many) [#step-workers-parallel-many]

Step Workers execute the actual units of work — invoking connectors, evaluating
rules, running your step logic — in parallel across the fleet. When a worker
finishes, it writes its outcome as a row in `eec_workflow_step_attempt`. Workers
**never** touch run state; they only append attempts that the Coordinator later
reads and folds in.

<Callout title="Why split it this way?">
  Writers to shared state are where workflow engines quietly lose data. By
  making the Coordinator the sole writer and workers append-only, progress is
  always either committed by the Coordinator or visible as a recorded attempt —
  never silently dropped.
</Callout>

## Anatomy of a tick [#anatomy-of-a-tick]

A tick is deliberately structured as **load → compute (pure) → commit →
dispatch**. Keeping the middle *pure* — a function of loaded state with no side
effects — is what makes retries and replays safe.

```mermaid
sequenceDiagram
    participant W as Step Workers
    participant C as Coordinator
    participant P as Postgres
    W->>P: Append attempt results
    C->>P: Load run state and inputs
    C->>C: Compute payload on a clone (pure)
    C->>P: Commit if record_version matches
    C->>W: Dispatch runnable tasks (post-commit)
```

<Steps>
  <Step>
    **Load state.** Read the mutable run state (status, per-node states, run
    context, `record_version`) from the store. If the run is already terminal,
    the tick is a no-op and returns immediately.
  </Step>

  <Step>
    **Drain one control.** If an operator intent is pending (pause, resume,
    cancel, terminate, redrive, continue-as-new), handle it first — control is a
    first-class, coordinator-owned transition, not a side channel.
  </Step>

  <Step>
    **Load tick inputs.** Gather everything the compute step needs: new attempt
    results from workers, pending signals, fired timers, child-run snapshots,
    credentials, and heartbeats.
  </Step>

  <Step>
    **Compute the payload (pure).** Over a *clone* of the state — never the live
    row — run the tick phases in order:

    * **Advance nodes** — move ready nodes from pending to running, and apply the
      attempt results workers have reported.
    * **Drive fan-out** — fire scatter timers, and finalize scatter parents once
      their branches complete.
    * **Absorb external input** — drain signals and controls, and resolve expired
      timers and child waits.
    * **Apply and schedule** — apply effect patches, schedule newly runnable nodes,
      and detect completion or checkpoints.

    The output is a single `TickPayload` stamped with `ExpectedVersion =
    record_version` — no durable state has been mutated yet.
  </Step>

  <Step>
    **Commit atomically.** Persist the payload in one Postgres transaction: update
    the run row, insert new attempt rows, insert transitions, and mark applied
    signals — all conditional on the `record_version` still matching. If it moved,
    the commit is rejected and the tick is recomputed against fresh state (see
    [Optimistic Locking](/docs/concepts/optimistic-locking)).
  </Step>

  <Step>
    **Dispatch post-commit.** *Only after* the commit succeeds, dispatch the newly
    runnable step tasks, spawn sub-workflows, fire defers, release or reacquire
    concurrency leases, and enqueue any coordinator self-wakeups.
  </Step>
</Steps>

Because a single writer owns the transition and every commit is version-checked,
there is no read-modify-write race on run state — the invariant &#x2A;"only the
coordinator mutates `eec_workflow_run`"* is enforced, not hoped for.

## Commit, then dispatch [#commit-then-dispatch]

Dispatch happens **post-commit** on purpose. If Ductor dispatched a task and
*then* failed to persist that it did so, a crash could double-run work or lose
it. By committing the intent first and dispatching afterward, a redelivery is
always safe: the Coordinator re-derives what still needs doing from durable
state. This is exactly the boundary that makes state progress *exactly-once*
while side-effect dispatch is *at-least-once* — see
[Idempotency & Exactly-Once](/docs/concepts/idempotency).

## Attempts and stale workers [#attempts-and-stale-workers]

Workers write to `eec_workflow_step_attempt`; the Coordinator reads those rows
on its next tick and folds the results into run state. Two safeguards keep this
honest under retries:

* Each attempt carries a **stamp/token**. Re-applying a result that was already
  applied is a no-op — it produces no new rows.
* On retry, the node's attempt generation is bumped, so a **stale worker** from
  an older attempt cannot land its result on top of a newer one.

If a run is cancelled or terminated, any dangling `running` attempt rows are
closed out — nothing is left half-open.

## Beyond linear DAGs [#beyond-linear-dags]

The Coordinator detects and drives the richer patterns a real workflow needs.
Each is covered in depth in
[The DAG Workflow Model](/docs/concepts/dag-workflow-model):

* **ContinueAsNew** — close out an oversized run and start a fresh successor,
  carrying forward the state that matters. Can be triggered explicitly or
  automatically at a transition-count threshold.
* **Scatter / gather** — fan a step out into many parallel branches (inline or as
  child runs) and collect their results before proceeding.
* **Sub-workflows** — a step that runs another workflow to a terminal state or a
  named checkpoint.
* **Signals, waits, approvals, and event sets** — external events and timers
  that park a run and later wake it with new input.

## Wakeups [#wakeups]

Coordinators don't poll; they're driven by **wakeups** delivered through Redis —
the zero-latency in-process **SyncMatch** channel or the durable
[tiered queue](/docs/concepts/tiered-queue) — one tick per wakeup. This keeps
ticks cheap and lets many runs make progress concurrently while each individual
run stays strictly serial. The exact dequeue-and-requeue wiring lives in the
[architecture deep-dive](/docs/architecture/coordinator-worker-model#wiring-how-a-run-gets-ticked).

Because the tick is idempotent under `record_version`, wakeups are intentionally
*not* deduplicated: a redundant wakeup produces a harmless no-op tick, and that's
cheaper and safer than trying to suppress it. A *missed* wakeup would stall a
run; a *duplicate* one costs almost nothing.

## Where to go next [#where-to-go-next]

* [The DAG Workflow Model](/docs/concepts/dag-workflow-model) — the steps and patterns a tick drives.
* [Optimistic Locking](/docs/concepts/optimistic-locking) — the concurrency primitive that makes the commit safe.
* [The Coordinator-Worker Model, in depth](/docs/architecture/coordinator-worker-model) — the architectural view.