# Durable Runs & Workflows (/docs/management/workflows)



Workflow management has two distinct halves, and it's worth separating them up
front:

* **Definitions** — the *design* of a workflow: its graph of steps and edges.
  You edit a definition as a **draft**, then **publish** it as an **immutable,
  versioned** artifact.
* **Runs** — a *single execution* of a published definition. You **trigger** a
  run and then **operate** it: pause, resume, cancel, terminate, redrive, signal,
  or continue-as-new.

The [Coordinator](/docs/concepts/coordinator-workers) is the sole writer of run
state, so every run operation here is an **intent** the coordinator applies on
its next tick — never a direct state mutation. That's what makes the run control
surface safe to call concurrently.

<Callout title="Where it lives">
  Definitions: `WorkflowDefinitionService` under `/api/workflow-definitions`.
  Triggering: `WorkflowTriggerService`. Run control:
  `WorkflowRunControlService` under `/api/v2/workflow-runs`. Signals:
  `WorkflowSignalService`. Run reads: `WorkflowService` under `/api/workflows`.
  Run state persists to `eec_workflow_run`; per-step attempts to
  `eec_workflow_step_attempt` (the `eec_` prefix marks Ductor's durable
  *Enterprise Eventing Core* tables). Definition lifecycle logic is at
  `application/workflow/definition/`; run ops at `application/workflowops/`.
</Callout>

## Part 1 — Workflow definitions [#part-1--workflow-definitions]

### The lifecycle: draft → publish → archive [#the-lifecycle-draft--publish--archive]

A definition family (identified by a stable `family_slug`) moves through three
states:

| Status      | Meaning                                                    | Editable?                   |
| ----------- | ---------------------------------------------------------- | --------------------------- |
| `draft`     | Work-in-progress. The only editable state.                 | Yes, in place.              |
| `published` | An immutable, numbered version. What runs execute against. | No — publish a new version. |
| `archived`  | A retired published version.                               | No.                         |

The core rule: &#x2A;*you edit drafts, you run published versions.** Publishing
validates the draft and writes a brand-new immutable version row (version =
previous max + 1 for that family); the draft is untouched and remains available
for continued editing. Published versions are never mutated in place, so a run
pinned to version 3 always executes exactly the graph that was version 3.

```mermaid
flowchart TD
    Create["CreateWorkflowDefinition"] --> Draft["draft (vN in progress)"]
    Draft -->|"UpdateWorkflowDraft (optimistic, repeatable)"| Draft
    Draft -->|"PublishWorkflowDefinition"| Pub["published version (immutable)"]
    Pub -->|"fresh draft returned"| Draft
    Pub -->|"ArchiveWorkflowDefinition"| Arch["archived"]
```

### Create a draft [#create-a-draft]

`CreateWorkflowDefinition` — `POST /api/workflow-definitions`
(`workflow_definition:write`). You supply a `family_slug` and, optionally, the
initial graph.

```bash
curl -s -X POST https://api.ductor.io/api/workflow-definitions \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "family_slug": "lead-router",
    "title": "Lead Router",
    "description": "Enrich, score, and assign inbound leads",
    "tags": ["sales"]
  }'
```

The response is a `WorkflowDefinitionDetail` with `status: "draft"`, `version: 1`,
and a `draft_revision` you'll need for the next edit.

### Edit the draft [#edit-the-draft]

`UpdateWorkflowDraft` — `PATCH /api/workflow-definitions/{id}/draft`
(`workflow_definition:write`). Edits are expressed as **graph operations** and
guarded by the draft's revision for optimistic concurrency.

```bash
curl -s -X PATCH https://api.ductor.io/api/workflow-definitions/$DEF_ID/draft \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "expected_draft_revision": 3,
    "title": "Lead Router v2",
    "operations": [
      { "add_step": { "ref": "enrich", "type": "STEP_TYPE_ACTION", "action": "hubspot.enrich_contact" } }
    ]
  }'
```

* `expected_draft_revision` must equal the current revision. If another editor
  has changed the draft since you read it, the write fails (optimistic-lock
  conflict) and you re-read, reconcile, and retry.
* The response includes any non-fatal `warnings` alongside the updated detail.

A step (`StepDefinitionProto`) carries an `id`/`ref`, a `type`, and — for action
steps — an `action` plus `args`. Step types include `ACTION`, `WAIT`,
`APPROVAL`, `SCATTER`, `GATHER`, `SUBWORKFLOW`, `RACE`, and more; edges
(`EdgeDefinitionProto`) connect step refs and carry a `type` (`SUCCESS`, `ERROR`,
`TRUE`/`FALSE`, `APPROVED`/`REJECTED`, …) and an optional `condition`. Two
event-oriented step types are worth calling out:

* **`STEP_TYPE_EVENT_SET`** — parks a run until a *correlated set* of external
  events arrives (not just one). It waits on a `correlation_key` derived from the
  trigger and requirement expressions, then resumes once the set is satisfied.
* **`STEP_TYPE_DIGEST`** — collapses N upstream events that share a `digest_key`
  into a single downstream arrival, with optional per-arrival dedup. Runs merged
  into a digest surface the terminal `merged` run status.

The full graph vocabulary is described in [The DAG Workflow
Model](/docs/concepts/dag-workflow-model), and the event model behind
`event_set`/`digest` in [Events](/docs/concepts/events).

### Saga: compensation and finally [#saga-compensation-and-finally]

A step can declare a **compensation policy** (`domain/workflow/compensation_policy.go`)
so the workflow behaves like a saga — with inverse actions that undo committed
work and finalizers that always run:

| Block        | When it runs                                                                                                      |
| ------------ | ----------------------------------------------------------------------------------------------------------------- |
| `on_failure` | Inverse (rollback) actions, registered only after the owner step *succeeds*, invoked if the workflow later fails. |
| `on_cancel`  | Inverse actions invoked when the run is cancelled.                                                                |
| `finally`    | Terminal cleanup/audit actions — not rollback, always run at the end.                                             |

Compensation targets are connector actions (the first-class executable kind);
a failed compensation is recorded per `CompensationFailurePolicy`
(`manual_repair` / `fail_compensation` / `ignore_with_audit`) and never mutates
the already-terminal parent run.

### Scheduled workflows [#scheduled-workflows]

A definition can carry a **cron schedule** so runs fire on a timetable rather
than an external trigger. The schedule is a `ScheduleSpec`
(`domain/workflow/schedule`):

| Field            | Purpose                                                                |
| ---------------- | ---------------------------------------------------------------------- |
| `CronExpression` | The cron timetable.                                                    |
| `Timezone`       | IANA timezone the expression is evaluated in.                          |
| `CatchupWindow`  | How far back a recovering scheduler will backfill missed fires.        |
| `OverlapPolicy`  | What to do when the next fire lands while a prior run is still active. |

`OverlapPolicy&#x60; in v1 supports exactly two values: &#x2A;*`skip`*&#x2A; (drop the new
occurrence while a run is active) and &#x2A;*`allow_all`** (fire regardless). Anything
else is rejected (`ErrInvalidOverlapPolicy`).

### Publish [#publish]

`PublishWorkflowDefinition` — `POST /api/workflow-definitions/{id}/publish`
(`workflow_definition:write`). No body beyond the path `id` is needed.

```bash
curl -s -X POST https://api.ductor.io/api/workflow-definitions/$DEF_ID/publish \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT"
```

Publish runs **strict validation** — graph well-formedness, routing/reply/event
CEL checks, entitlement gates — before it writes the version. If validation
fails, you get a structured validation error and nothing is published. On
success the response carries both the newly `published` version and a fresh
`draft` to keep editing.

<Callout title="Published versions are immutable and pinned by version">
  Runs and sub-workflow references resolve a definition **by version**, and the
  published row is never mutated. That's the guarantee that lets a long-running
  or replayed run behave deterministically: the graph it started on can't change
  underneath it. To change behavior, publish a new version and trigger against
  it.
</Callout>

### Other definition operations [#other-definition-operations]

| Operation                          | RPC                           | HTTP                                                    | Action     |
| ---------------------------------- | ----------------------------- | ------------------------------------------------------- | ---------- |
| Get (full graph)                   | `GetWorkflowDefinition`       | `GET /api/workflow-definitions/{id}`                    | read       |
| List                               | `ListWorkflowDefinitions`     | `GET /api/workflow-definitions`                         | read       |
| Delete draft                       | `DeleteWorkflowDraft`         | `DELETE /api/workflow-definitions/{id}/draft`           | write      |
| Duplicate                          | `DuplicateWorkflowDefinition` | `POST /api/workflow-definitions/{id}/duplicate`         | write      |
| Archive                            | `ArchiveWorkflowDefinition`   | `POST /api/workflow-definitions/{id}/archive`           | write      |
| Env overrides (upsert/list/delete) | `…WorkflowEnvOverride`        | `/api/workflow-definitions/{workflow_id}/env-overrides` | write/read |

`ListWorkflowDefinitions` filters by `status`, free-text `q`, and `tags`.
`DuplicateWorkflowDefinition` clones the latest draft (or the published version
when no draft exists) into a fresh draft under a new `family_slug` — the way to
fork a workflow. Deleting a draft never affects published versions.

## Part 2 — Triggering a run [#part-2--triggering-a-run]

Start a run with `WorkflowTriggerService.TriggerWorkflow` — `POST
/api/v2/workflow-definitions/{definition_id}:trigger` (`workflow:write`). The
`definition_id` must be a **published** definition.

```bash
curl -s -X POST "https://api.ductor.io/api/v2/workflow-definitions/$DEF_ID:trigger" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "subject_type": "lead",
    "subject_id": "lead-123",
    "input": { "email": "prospect@example.com", "source": "webinar" },
    "wait_for_checkpoint": false,
    "idempotency_key": "lead-123-signup",
    "external_deployment_id": "prod-us-east-1-wave-1",
    "correlation_labels": [
      { "key": "conversation", "id": "conversation-42" },
      { "key": "support_ticket", "id": "ticket-9182" }
    ]
  }'
```

```json
{
  "run_id": "d1e2f3a4-...",
  "status": "running",
  "metadata": { "trace_id": "trace-...", "request_id": "req-...", "fetched_at": "2026-07-11T11:10:00Z" }
}
```

* `input` is the run's initial payload (arbitrary JSON).
* `idempotency_key` dedupes on `(tenant, definition_id, idempotency_key)` — retry
  the same trigger and you get the same run back, not a duplicate.
* `external_deployment_id` pins public admission to the server-declared live
  caller release when the production policy requires it. It is read only from
  the request body, never a generic header. See
  [Workflow release pinning](/docs/deployment/release-pinning).
* `correlation_labels` attaches stable business identities to this run and every
  descendant run. Use them to find all work for one conversation, ticket,
  import, order, or other domain journey without overloading the input payload.
* `wait_for_checkpoint: true` (with `checkpoint_timeout_seconds`) blocks the call
  until the workflow reaches its response checkpoint and returns
  `response_group_output` inline — useful for synchronous request/response
  workflows.

### Correlate a run tree [#correlate-a-run-tree]

System lineage (`root_run_id` and journey identity) tells Ductor how runs are
related internally. `correlation_labels` adds the bounded identities your
application already understands. Labels are indexed in workflow visibility and
propagate automatically through subworkflows and continue-as-new successors.
Set them once on the root trigger instead of reattaching them at every step.

| Contract           | Limit                                                        |
| ------------------ | ------------------------------------------------------------ |
| Labels per trigger | 8                                                            |
| Key                | 1–64 bytes; `^[a-z][a-z0-9_.-]*$`; unique within the request |
| ID                 | 1–128 bytes                                                  |

Ductor stores the canonical list under the reserved visibility attribute
`ductor.correlation_labels` and indexes each value as
`ductor.correlation.<key>`. Callers and workflow effects cannot write either
namespace directly; send `correlation_labels` on the trigger so the runtime can
validate and propagate them without allowing a step to spoof lineage.

Good keys describe a stable domain type (`conversation`, `support_ticket`,
`import`, `order`). Keep the id opaque and non-sensitive. Do not use labels for
secrets, prompt content, or arbitrary high-cardinality telemetry.

## Part 3 — Operating a run [#part-3--operating-a-run]

Run control is `WorkflowRunControlService` under `/api/v2/workflow-runs`. Every
control RPC dispatches a coordinator wakeup carrying the intent; the coordinator
applies the state change and writes the audit entry on its next tick. Handlers
never touch `eec_workflow_run` directly.

<Callout title="Use the /api/v2 control surface, not the legacy one">
  An older `WorkflowService` also exposes pause/resume/cancel/terminate under
  `/api/workflows/runs/{run_id}/…`. Those RPCs are **deprecated and have no
  production wiring** — they return without effect. Always drive run control
  through `WorkflowRunControlService` at `/api/v2/workflow-runs/{run_id}:…`.
</Callout>

### The core controls [#the-core-controls]

All require `workflow_run:write` and share a request shape: `request_id` (a
caller-supplied idempotency token) and `reason` (an audit note).

| Operation       | HTTP                                                | What it does                                         |
| --------------- | --------------------------------------------------- | ---------------------------------------------------- |
| Pause           | `POST /api/v2/workflow-runs/{run_id}:pause`         | Halts progress; run becomes `paused` (non-terminal). |
| Resume          | `POST /api/v2/workflow-runs/{run_id}:resume`        | Un-pauses a paused run.                              |
| Cancel          | `POST /api/v2/workflow-runs/{run_id}:cancel`        | Requests graceful cancellation.                      |
| Terminate       | `POST /api/v2/workflow-runs/{run_id}:terminate`     | Hard stop; terminal.                                 |
| Redrive         | `POST /api/v2/workflow-runs/{run_id}:redrive`       | Re-drives a stuck/failed run.                        |
| Continue-as-new | `POST /api/v2/workflow-runs/{run_id}:continueAsNew` | Ends this run and starts a fresh successor.          |

```bash
curl -s -X POST "https://api.ductor.io/api/v2/workflow-runs/$RUN_ID:pause" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{ "request_id": "op-8f2a-1", "reason": "manual hold pending review" }'
```

```json
{ "status": "accepted", "run_id": "d1e2f3a4-...",
  "metadata": { "trace_id": "trace-...", "request_id": "req-...", "fetched_at": "2026-07-11T11:12:00Z" } }
```

A `status` of `"accepted"` means the intent was durably enqueued for the
coordinator — including when a duplicate `request_id` is replayed idempotently.
It does **not** mean the state has already changed; poll the run (below) to
observe the applied transition.

`continueAsNew` additionally accepts `seed_run_context` (initial context for the
successor) and `discard_pending`, and its response carries the `new_run_id`.

### Node-level control [#node-level-control]

The same service exposes finer operator levers on individual steps of a run:
`PauseNode` / `ResumeNode` / `ResetNode`
(`/api/v2/workflow-runs/{run_id}/nodes/{step_ref}:…`), plus per-node retry-policy
and timeout updates. `GetRunSnapshot`
(`GET /api/v2/workflow-runs/{run_id}/snapshot`) and `ListRunAttempts`
(`/attempts`) give you the debug view — every node's state and every recorded
attempt.

### Reset / replay [#reset--replay]

`ResetRun` (`POST /api/v2/workflow-runs/{source_run_id}:reset`) forks a run from a
chosen transition sequence into a new run, optionally overriding step inputs and
preserving selected step outputs — the tool for surgical replay after a bad
deploy or data fix. `ListResetTargets` and `PreviewReset` let you see valid reset
points and dry-run the outcome first.

## Part 4 — Signals [#part-4--signals]

Some steps park waiting for an external decision or event. Signals resume them.
All signal RPCs are on `WorkflowSignalService`, take a `request_id` for
idempotency, and return `status: "accepted"`. Duplicate signals (same
`request_id`, or an already-resolved step) are no-ops.

| Signal              | HTTP                                                                      | Resumes                                                        |
| ------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- |
| Approve             | `POST /api/v2/workflow-runs/{run_id}/steps/{step_ref}:approve`            | An `APPROVAL` step → success edges.                            |
| Reject              | `POST /api/v2/workflow-runs/{run_id}/steps/{step_ref}:reject`             | An `APPROVAL` step → error edges (`approval_rejected`).        |
| Resolve interaction | `POST /api/v2/workflow-runs/{run_id}/steps/{step_ref}:resolveInteraction` | An interaction-wait step; merges payload into the step result. |
| Publish event       | `POST /api/v2/workflow-events:publish`                                    | Any run/step waiting on the event by correlation.              |

Approve/reject/resolve require `workflow_run:write`; publish-event requires
`workflow_events:publish`.

```bash
# Approve a human-approval step
curl -s -X POST "https://api.ductor.io/api/v2/workflow-runs/$RUN_ID/steps/manager-approval:approve" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{ "request_id": "sig-1", "approver_id": "alex@algib.com", "comment": "cleared", "payload": { "tier": "gold" } }'
```

Publishing an event is deliberately **decoupled** from run and step IDs — you
publish by `event_name` + `correlation_key`, and the engine resolves which
waiting runs match. `event_id` is the producer's idempotency token.

```bash
curl -s -X POST https://api.ductor.io/api/v2/workflow-events:publish \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{ "event_name": "order.paid", "correlation_key": "order-42", "event_id": "evt-abc", "payload": { "amount": 12900 } }'
```

The response reports `matched` — how many waiting pauses this event resolved. A
duplicate event or one with no waiters returns `matched: 0` (a safe no-op).

<Callout title="Dry-run a match before you publish" type="info">
  `ExplainWorkflowEventSetMatch` — `POST
    /api/v2/workflow-events:explainEventSetMatch` — evaluates a *sample* event
  against active `event_set` waits **without** appending receipts or emitting
  resume signals. It answers "would this event match, and if not, why?" —
  returning candidate-wait counts and reason codes only (never raw payloads,
  correlation keys, or dedupe keys). Reach for it when debugging why an event
  isn't resuming a run before you publish the real one.
</Callout>

## Part 5 — Inspecting runs [#part-5--inspecting-runs]

Run reads live on `WorkflowService` under `/api/workflows` (`workflow:read`).
These are the canonical observability endpoints:

| Read                                        | HTTP                                               |
| ------------------------------------------- | -------------------------------------------------- |
| List runs (filter by type/status/subject)   | `GET /api/workflows/runs`                          |
| Get a run (input, result, history, lineage) | `GET /api/workflows/runs/{run_id}`                 |
| Steps of a run                              | `GET /api/workflows/runs/{run_id}/steps`           |
| A single step                               | `GET /api/workflows/runs/{run_id}/steps/{step_id}` |
| Timeline / unified timeline                 | `GET /api/workflows/runs/{run_id}/timeline`        |
| Audit trail                                 | `GET /api/workflows/runs/{run_id}/audit`           |
| Attempt-level events                        | `GET /api/workflows/runs/{run_id}/events`          |

```bash
curl -s "https://api.ductor.io/api/workflows/runs?workflow_type=lead_router&status=running&active_only=true" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" -H "X-Tenant-ID: $DUCTOR_TENANT"
```

### Run status values [#run-status-values]

| Status       | Terminal? | Meaning                             |
| ------------ | --------- | ----------------------------------- |
| `pending`    | No        | Created, not yet started.           |
| `running`    | No        | Actively progressing.               |
| `paused`     | No        | Paused by a control signal.         |
| `completed`  | Yes       | Finished successfully.              |
| `failed`     | Yes       | Finished with an unrecovered error. |
| `cancelled`  | Yes       | Gracefully cancelled.               |
| `timed_out`  | Yes       | Exceeded its deadline.              |
| `terminated` | Yes       | Hard-stopped.                       |
| `merged`     | Yes       | Absorbed into a digest run.         |

## Where run state lives [#where-run-state-lives]

Run state is a single row in `eec_workflow_run` — status, current step,
`node_states` (per-step snapshots as JSONB), `run_context`, lineage
(`root_run_id`, `parent_run_id`, `nesting_depth`), and the optimistic-lock
counter `db_record_version`. Every worker attempt is a row in
`eec_workflow_step_attempt` (input, output, error, heartbeat), which is what
`ListRunAttempts` and the events endpoint surface.

The coordinator advances a run by reading pending attempts, computing the next
tick, and committing with `db_record_version = db_record_version + 1` guarded by
the version it read — a mismatch means another tick won the race and the
coordinator retries. This is the mechanism behind Ductor's [exactly-once
progress guarantee](/docs/concepts/idempotency); as an operator you observe it
as monotonic, never-lost run progress.

## Where to go next [#where-to-go-next]

<Cards>
  <Card title="The coordinator-worker model" href="/docs/concepts/coordinator-workers">
    Why every run op is an intent the coordinator applies.
  </Card>

  <Card title="The DAG workflow model" href="/docs/concepts/dag-workflow-model">
    Steps, edges, scatter/gather, sub-workflows, waits, ContinueAsNew.
  </Card>

  <Card title="Events" href="/docs/concepts/events">
    The correlation model behind event\_set waits and digest steps.
  </Card>

  <Card title="Workflow runtime" href="/docs/operations/workflow-runtime">
    How the coordinator and workers execute and advance runs.
  </Card>

  <Card title="SDKs" href="/docs/sdks">
    Define workflows in code instead of raw graph operations.
  </Card>

  <Card title="Connections" href="/docs/management/connections">
    The credentials that action steps dispatch through.
  </Card>
</Cards>