# Durable Route Runtime (/docs/operations/workflow-runtime)
The workflow runtime carries governance surfaces an operator needs to
understand: how much data run history is allowed to hold, how old history is
pruned, where expired records are archived, and the safety nets that keep stuck
work from wedging a coordinator. These bound cost and blast radius — they are
not app features.
## Journal payload budgets [#journal-payload-budgets]
Every durable journal write declares **how** its payload is stored, so run
history never becomes an unbounded JSON store and secrets never land in lineage.
A payload is classified into one of five classes:
| Class | Meaning |
| --------------------------- | -------------------------------------------------------------------- |
| `inline_redacted` (default) | Stored inline after secret/PII redaction |
| `inline_public` | Stored inline verbatim — only for payloads known free of secrets/PII |
| `external_artifact_ref` | Body offloaded to the archival backend; the row keeps a reference |
| `summary_only` | Only a bounded summary is kept |
| `discarded_by_policy` | Body dropped; only size + reason code retained |
The effective policy is a `JournalPayloadBudget` with `max_inline_bytes`,
`max_run_inline_bytes`, and an `overflow_action` of `externalize`, `summarize`,
`discard`, or `reject`. The classification is a **pure function** of the budget
and the (redact-before-measure) payload size, so replay and live runtime always
agree.
**Defaults are conservative:** 64 KiB per entry, 4 MiB per run, and
`externalize` on overflow — a payload is never silently stored as oversized raw
JSON. **Production hardening only tightens:** it drops the ceilings to 16 KiB
per entry / 1 MiB per run and downgrades `inline_public` to `inline_redacted`.
Authors cannot loosen a hardened budget.
Step outputs are operational state the coordinator reads back byte-for-byte,
so they are **never redacted, summarized, or discarded** — only stored inline
or externalized. The output inline ceiling is fail-closed: above it, the
output externalizes when a backend is available and otherwise the write
**fails**. Set your externalization threshold at or below the output inline
ceiling, or oversized outputs will be rejected rather than stored.
## Transition-history retention & compaction [#transition-history-retention--compaction]
A run accumulates transition-history rows. The compaction worker prunes the
low-value ones — but it is **off by default** and deliberately conservative.
Enable it under `workflow_runtime.transition_compaction`:
* It prunes only `debug_detail` and `discardable` rows, and only for runs
**closed at least `min_closed_age` ago** (default **7 days**).
* It **never** removes replay-carrying or delta-carrier rows, so replay stays
**byte-identical** after compaction.
* Every prune is atomic with a durable **compaction receipt** — the delete and
the receipt commit together.
This is why enabling it is safe: it can only remove rows that neither replay nor
audit depends on, and it always leaves an evidence trail.
## Dual-stream archival [#dual-stream-archival]
Expired routing decisions and workflow runs are archived per tenant against that
tenant's resolved **policy URIs**, in two independent streams — a durable
**history** stream and a **visibility** sidecar — each separately gated per
tenant. The application layer only decides which tenants to sweep and gates each
on the effective policy; the storage backend (`filestore`, `S3`, or `GCS`,
selected by URI scheme) is resolved by the infrastructure adapter, which builds
the batch writer, deduplicates, and deletes archived rows when the policy
allows.
## Reliability safety nets [#reliability-safety-nets]
The runtime ships several nets that keep bad runs from becoming outages, tuned
under `workflow_runtime.*`:
* **Continue-As-New (CAN)** — bounds history growth. Auto-triggers past
`continue_as_new_after_history` / `continue_as_new_after_wakeups`, with soft /
hard / terminate transition-count limits behind it. CAN is drain-by-default:
if signals, controls, or updates are still pending it defers rather than
stranding them (a DoS guard force-applies after a bounded defer count).
* **Stuck-run auto-fail** — a coordinator tick is bounded by
`coordinator_tick_timeout`; deadline-exceeded ticks are recorded through a
durable stuck-run counter so a wedged run cannot monopolize a worker.
* **Silent-strand recovery** — a reconciler scan catches payloads that queued
against a run mid-transition (the class of bug the CAN drain interlock
prevents forming in the first place).
* **Bulk cancellation** — terminate/cancel controls are exempt from the drain
interlock so operators can stop runs immediately.
### Run-concurrency lease ownership [#run-concurrency-lease-ownership]
Definitions that use run-mode concurrency acquire a Redis-backed lease before a
run starts. The lease token is persisted with the run and renewed before the
coordinator advances it. Renewal extends only an existing token—it never
recreates missing ownership. If the store is unavailable, the lease is missing,
or the next watchdog cannot be scheduled, the coordinator fails closed and does
not dispatch more work for that run.
Renewal runs at one third of `lease_ttl` (with a one-second floor), which leaves
multiple proof windows before expiry. A terminal run releases its slot after
the state commit. Optional early release gives a slot back while a run is parked
on an event, approval, or digest; the run must reacquire before resuming. A
leader-elected reclaimer removes expired or orphaned leases after crashes.
```yaml title="ductor.yaml"
workflow_runtime:
run_concurrency:
enabled: true
lease_ttl: 30m
resume_backoff: 30s
early_release:
enabled: false
reclaimer:
enabled: true
interval: 30s
key_pattern: "{wf-conc}:conc:wf:*"
```
Enabling the gate requires the shared Redis/Dragonfly flow client and aborts
startup if that dependency is missing. `lease_ttl` must be at least three
seconds. Keep early release off unless parked runs are a meaningful source of
slot pressure, and enable the reclaimer in production so crash-leaked slots are
bounded by TTL rather than operator intervention.
The runtime exposes the full lifecycle as Prometheus metrics:
| Metric suffix under `ductor_dag_` | Watch for |
| -------------------------------------------------------- | -------------------------------------------------------------- |
| `coordinator_run_concurrency_lease_acquired_total` | Admission throughput by workflow definition. |
| `coordinator_run_concurrency_lease_denied_total` | Sustained saturation or a limit set below demand. |
| `coordinator_run_concurrency_lease_released_total` | Terminal versus early releases. |
| `coordinator_run_concurrency_lease_release_failed_total` | Any nonzero rate; TTL becomes the only cleanup path. |
| `coordinator_run_concurrency_lease_renewal_total` | `lease_missing`, `store_error`, or `watchdog_schedule_failed`. |
| `coordinator_run_concurrency_lease_renewal_seconds` | Renewal-store latency. |
| `coordinator_run_concurrency_lease_remaining_seconds` | Unsafe remaining ownership after renewal. |
| `coordinator_run_concurrency_lease_persist_failed_total` | Resume-reacquire or early-release state drift. |
See [Observability](/docs/operations/observability#workflow-concurrency-alerts)
for alert guidance.
Most of these limits may be set to `0` for local/test compatibility.
`security_profile=enterprise` requires nonzero values for the tick timeout,
CAN thresholds, transition limits, and run-state caps — the runtime will not
start enterprise-hardened with the safety nets disabled.
## Related [#related]
The durable event substrate behind run history.
Authoring and operating workflows.
How work is queued and drained.