Health checks & readiness

Ductor exposes liveness and readiness endpoints on the HTTP listener, plus a health CLI subcommand. The distinction between liveness and readiness matters: liveness answers "is the process alive?", readiness answers "can it serve traffic right now?".

The endpoints

All are served on the shared API listener (api.addr, default :50052 for the bare binary). The Docker image sets DUCTOR_API_ADDR=:8080, and the default Compose stack publishes that listener at http://localhost:8080. REST, Connect, and native gRPC share this listener; server.http_addr does not open a separate request listener.

Liveness — process is up

The liveness handler returns 200 unconditionally while the process runs. It does not check dependencies. A failed probe can also mean the address, network, or proxy is wrong; confirm the listener before treating it as a process failure. Use it for the container/Kubernetes liveness probe so a transient database blip doesn't cause a restart loop.

Readiness — safe to route traffic

The readiness handler returns 200 only when both are true:

1. Internal readiness is set — the app finished booting and wiring.
2. The dependency aggregate is not unhealthy — configured Postgres, Redis, and queue checks run with a one-second probe deadline. A degraded aggregate still permits traffic. Minimal embeddings without a dependency checker use the internal readiness flag alone.

With the dependency checker wired, readiness re-checks the aggregate on every call and returns 503 when it becomes unhealthy after boot. Use it for the readiness probe and load-balancer health so an unhealthy pod is pulled from rotation without being killed.

A 200 does not establish that a tenant's work is succeeding or that observations are fresh. Use the authenticated Clearing-Layer Health Verdict for that diagnosis. A successful empty observation can show an idle environment; an unavailable or stale source must remain unknown, even when /health and /ready return 200.

The CLI check

ductor health performs an HTTP GET /health against a running server and prints OK or the error. It's handy for scripts and the container healthcheck:

ductor health --api-addr :8080

Set the address to your actual API listener; use :50052 for an unmodified bare binary. This checks liveness only. To check readiness explicitly:

curl --fail-with-body --silent --show-error --max-time 5 http://localhost:8080/ready

The container image's built-in healthcheck does the equivalent (wget --spider http://localhost:8080/health).

Probe configuration

Docker Compose

Compose overrides the image healthcheck with a curl request to /health: a 10-second interval, 5-second timeout, 5 retries, and a 30-second start period. The production image alone uses a 10-second interval, 3-second timeout, 3 retries, and a 5-second start period. Both check liveness; configure a separate readiness probe when directing traffic to a deployment.

Kubernetes (Helm)

The chart wires all three probes to the right endpoints:

What to alert on

• Readiness failing across replicas — dependencies (Postgres/Redis) unreachable, or a bad rollout. This is your primary "is Ductor serving?" alert.
• Coordinator health — stuck runs and dead-lettered wakeups (workflow_dag.coordinator.stuck_run_threshold, workflow_dag.coordinator.wakeup_dead_letter_threshold) surface in the DAG metrics.
• Pending migrations — serve logs a warning at startup; ductor schema status exits non-zero. Gate deploys on it.

See Observability for wiring these signals into metrics and alerts.