Durable Route Runtime

The workflow runtime carries governance surfaces an operator needs to understand: how much data run history is allowed to hold, how old history is pruned, where expired records are archived, and the safety nets that keep stuck work from wedging a coordinator. These bound cost and blast radius — they are not app features.

Journal payload budgets

Every durable journal write declares how its payload is stored, so run history never becomes an unbounded JSON store and secrets never land in lineage. A payload is classified into one of five classes:

The effective policy is a JournalPayloadBudget with max_inline_bytes, max_run_inline_bytes, and an overflow_action of externalize, summarize, discard, or reject. The classification is a pure function of the budget and the (redact-before-measure) payload size, so replay and live runtime always agree.

Defaults are conservative: 64 KiB per entry, 4 MiB per run, and externalize on overflow — a payload is never silently stored as oversized raw JSON. Production hardening only tightens: it drops the ceilings to 16 KiB per entry / 1 MiB per run and downgrades inline_public to inline_redacted. Authors cannot loosen a hardened budget.

Transition-history retention & compaction

A run accumulates transition-history rows. The compaction worker prunes the low-value ones — but it is off by default and deliberately conservative. Enable it under workflow_runtime.transition_compaction:

• It prunes only debug_detail and discardable rows, and only for runs closed at least min_closed_age ago (default 7 days).
• It never removes replay-carrying or delta-carrier rows, so replay stays byte-identical after compaction.
• Every prune is atomic with a durable compaction receipt — the delete and the receipt commit together.

This is why enabling it is safe: it can only remove rows that neither replay nor audit depends on, and it always leaves an evidence trail.

Dual-stream archival

Expired routing decisions and workflow runs are archived per tenant against that tenant's resolved policy URIs, in two independent streams — a durable history stream and a visibility sidecar — each separately gated per tenant. The application layer only decides which tenants to sweep and gates each on the effective policy; the storage backend (filestore, S3, or GCS, selected by URI scheme) is resolved by the infrastructure adapter, which builds the batch writer, deduplicates, and deletes archived rows when the policy allows.

Reliability safety nets

The runtime ships several nets that keep bad runs from becoming outages, tuned under workflow_runtime.*:

• Continue-As-New (CAN) — bounds history growth. Auto-triggers past continue_as_new_after_history / continue_as_new_after_wakeups, with soft / hard / terminate transition-count limits behind it. CAN is drain-by-default: if signals, controls, or updates are still pending it defers rather than stranding them (a DoS guard force-applies after a bounded defer count).
• Stuck-run auto-fail — a coordinator tick is bounded by coordinator_tick_timeout; deadline-exceeded ticks are recorded through a durable stuck-run counter so a wedged run cannot monopolize a worker.
• Silent-strand recovery — a reconciler scan catches payloads that queued against a run mid-transition (the class of bug the CAN drain interlock prevents forming in the first place).
• Bulk cancellation — terminate/cancel controls are exempt from the drain interlock so operators can stop runs immediately.

Run-concurrency lease ownership

Definitions that use run-mode concurrency acquire a Redis-backed lease before a run starts. The lease token is persisted with the run and renewed before the coordinator advances it. Renewal extends only an existing token—it never recreates missing ownership. If the store is unavailable, the lease is missing, or the next watchdog cannot be scheduled, the coordinator fails closed and does not dispatch more work for that run.

Renewal runs at one third of lease_ttl (with a one-second floor), which leaves multiple proof windows before expiry. A terminal run releases its slot after the state commit. Optional early release gives a slot back while a run is parked on an event, approval, or digest; the run must reacquire before resuming. A leader-elected reclaimer removes expired or orphaned leases after crashes.

workflow_runtime:
  run_concurrency:
    enabled: true
    lease_ttl: 30m
    resume_backoff: 30s
    early_release:
      enabled: false
    reclaimer:
      enabled: true
      interval: 30s
      key_pattern: "{wf-conc}:conc:wf:*"

Enabling the gate requires the shared Redis/Dragonfly flow client and aborts startup if that dependency is missing. lease_ttl must be at least three seconds. Keep early release off unless parked runs are a meaningful source of slot pressure, and enable the reclaimer in production so crash-leaked slots are bounded by TTL rather than operator intervention.

The runtime exposes the full lifecycle as Prometheus metrics:

See Observability for alert guidance.

Related