# Operating the Clearing Layer (/docs/operations)



Day-2 operations for a running Ductor deployment. These pages assume you've
already [deployed](/docs/deployment) and focus on keeping it healthy, secure,
observable, and recoverable. If you're wiring configuration, keep the
[configuration reference](/docs/reference/configuration) open alongside.

## The operational model [#the-operational-model]

Ductor ships as a **single binary**. Every process runs the same image; what a
process does is selected at boot by the `--role` flag on `ductor serve`
(default `all` — the monolith topology, byte-identical to a single-process
deployment):

| Role       | Runs                                                   | Use for                                     |
| ---------- | ------------------------------------------------------ | ------------------------------------------- |
| `all`      | Everything (default)                                   | Single-process and small deployments        |
| `api`      | Public REST + Connect-RPC, MCP, admin transports       | Stateless, horizontally-scaled API replicas |
| `worker`   | DAG coordinator + step workers + scheduling resolvers  | The pods that own the workflow runtime      |
| `executor` | Internal `ExecutorService` only (task dispatch target) | Optional split executor plane               |

Combine roles on one process — `--role=worker,executor` is common. Only the
`worker` role runs the [coordinator](/docs/concepts/coordinator-workers): a
serial coordinator plus parallel workers, so correctness survives crashes and
horizontal scaling. Work is always either committed by the coordinator or
recorded as a worker attempt, never silently lost.

State splits cleanly by durability:

* **PostgreSQL is the source of truth** — workflow runs, step attempts, pools,
  rules, tenants, encrypted connections, idempotency records, audit. Losing it
  loses durable state; back it up.
* **Redis/Dragonfly is ephemeral** — cache, flow-control counters, the tiered
  queue, pub/sub. Rebuildable, but its loss is disruptive to in-flight work.

```mermaid
flowchart LR
    Binary["ductor binary<br/>role set by --role"] --> API["api"]
    Binary --> Worker["worker"]
    Binary --> Executor["executor"]
    API --> PG[("PostgreSQL<br/>source of truth")]
    Worker --> PG
    Executor --> PG
    API --> Redis[("Redis / Dragonfly<br/>ephemeral")]
    Worker --> Redis
    Executor --> Redis
```

Because readiness re-checks Postgres and Redis on every call, pod rotation is
**readiness-based**: an unhealthy pod flips `/ready` to `503` and is pulled
from rotation without being killed. Keep liveness and readiness on different
endpoints — see [Health checks](/docs/operations/health-checks).

<Callout title="Production checklist gates">
  Before promoting a change, the operator checklist expects three gates green:
  `make validate-config` (configuration validation), `make docs-check`, and
  `make openapi-check` (so operator docs and the generated API contract don't
  drift). Run migrations explicitly in production rather than relying on
  auto-migrate, and add or update a runbook in the same change set when a
  system behavior changes.
</Callout>

## In this section [#in-this-section]

<Cards>
  <Card title="Database migrations" href="/docs/operations/migrations">
    goose-style numbered SQL, auto-migrate vs explicit runs, and the Helm hook.
  </Card>

  <Card title="Observability" href="/docs/operations/observability">
    The VictoriaMetrics stack, the K1 typed metric catalog, anomaly detection,
    and the tenant-scoped PromQL proxy.
  </Card>

  <Card title="Security & auth" href="/docs/operations/security">
    OIDC, API keys, RBAC, entitlement enforcement, egress, and connector
    credential encryption.
  </Card>

  <Card title="Health checks & readiness" href="/docs/operations/health-checks">
    What `/health`, `/livez`, and `/ready` verify — and what to alert on.
  </Card>

  <Card title="Operational health verdict" href="/docs/operations/operational-health">
    One deterministic tenant/environment diagnosis shared by REST, MCP, CLI,
    and dashboard.
  </Card>

  <Card title="Workflow data streams" href="/docs/operations/workflow-data-streams">
    Durable progressive payloads, strict cursor handoff, producer credit,
    large-object recovery, retention, and residency.
  </Card>

  <Card title="Backup & restore" href="/docs/operations/backup-restore">
    Tier-0 recovery assets, Postgres PITR, billing-ledger and failover runbooks.
  </Card>

  <Card title="Key management" href="/docs/operations/key-management">
    Rotating the AEAD master key that protects connector credentials and
    payloads, plus BYOK/KEK boot unwrapping.
  </Card>

  <Card title="Dynamic config" href="/docs/operations/dynamic-config">
    Ductor's two dynamic-config systems — the tenant-scoped registry and the
    file-based feature-flag/A-B system — and how they differ.
  </Card>

  <Card title="Workflow runtime" href="/docs/operations/workflow-runtime">
    Journal payload budgets, transition-history compaction, archival, and
    reliability safety nets.
  </Card>

  <Card title="Enterprise activation" href="/docs/operations/enterprise-activation">
    Configure and verify the fail-closed `security_profile=enterprise` posture.
  </Card>

  <Card title="Promotions & rollout" href="/docs/operations/promotions-and-rollout">
    Promote config and strategy artifacts through evidence-backed runtime gates.
  </Card>

  <Card title="Queue operations" href="/docs/operations/queue-operations">
    Pause, resume, drain, apply admission control, and replay dead-lettered work.
  </Card>
</Cards>