Operating the Clearing Layer
Running Ductor in production — the operational model, plus migrations, observability, security, keys, and day-2 runbooks.
Day-2 operations for a running Ductor deployment. These pages assume you've already deployed and focus on keeping it healthy, secure, observable, and recoverable. If you're wiring configuration, keep the configuration reference open alongside.
The operational model
Ductor ships as a single binary. Every process runs the same image; what a
process does is selected at boot by the --role flag on ductor serve
(default all — the monolith topology, byte-identical to a single-process
deployment):
| Role | Runs | Use for |
|---|---|---|
all | Everything (default) | Single-process and small deployments |
api | Public REST + Connect-RPC, MCP, admin transports | Stateless, horizontally-scaled API replicas |
worker | DAG coordinator + step workers + scheduling resolvers | The pods that own the workflow runtime |
executor | Internal ExecutorService only (task dispatch target) | Optional split executor plane |
Combine roles on one process — --role=worker,executor is common. Only the
worker role runs the coordinator: a
serial coordinator plus parallel workers, so correctness survives crashes and
horizontal scaling. Work is always either committed by the coordinator or
recorded as a worker attempt, never silently lost.
State splits cleanly by durability:
- PostgreSQL is the source of truth — workflow runs, step attempts, pools, rules, tenants, encrypted connections, idempotency records, audit. Losing it loses durable state; back it up.
- Redis/Dragonfly is ephemeral — cache, flow-control counters, the tiered queue, pub/sub. Rebuildable, but its loss is disruptive to in-flight work.
Because readiness re-checks Postgres and Redis on every call, pod rotation is
readiness-based: an unhealthy pod flips /ready to 503 and is pulled
from rotation without being killed. Keep liveness and readiness on different
endpoints — see Health checks.
Production checklist gates
Verify the final source revision with the full release, integration, and critical-flow gates
described in Release Verification. Configuration
validation (make validate-config) and generated documentation and API checks (make docs-check,
make openapi-check) are part of that evidence. Run migrations explicitly in production, and
update the relevant runbook whenever system behavior changes.
In this section
Release verification
Verify a fixed source revision, executed critical flows, and the identity of the built artifact.
Database migrations
goose-style numbered SQL, auto-migrate vs explicit runs, and the Helm hook.
Observability
The VictoriaMetrics stack, the K1 typed metric catalog, anomaly detection, and the tenant-scoped PromQL proxy.
Security & auth
OIDC, API keys, RBAC, entitlement enforcement, egress, and connector credential encryption.
Health checks & readiness
What /health, /livez, and /ready verify — and what to alert on.
Operational health verdict
One deterministic tenant/environment diagnosis shared by REST, MCP, CLI, and dashboard.
Workflow data streams
Durable progressive payloads, strict cursor handoff, producer credit, large-object recovery, retention, and residency.
Backup & restore
Tier-0 recovery assets, Postgres PITR, billing-ledger and failover runbooks.
Key management
Rotating the AEAD master key that protects connector credentials and payloads, plus BYOK/KEK boot unwrapping.
Dynamic config
Ductor's two dynamic-config systems — the tenant-scoped registry and the file-based feature-flag/A-B system — and how they differ.
Workflow runtime
Journal payload budgets, transition-history compaction, archival, and reliability safety nets.
Enterprise activation
Configure and verify the fail-closed security_profile=enterprise posture.
Promotions & rollout
Promote config and strategy artifacts through evidence-backed runtime gates.
Queue operations
Pause, resume, drain, apply admission control, and replay dead-lettered work.