Observability

Ductor emits the three signals — metrics, traces, and logs — and includes a VictoriaMetrics-based stack to receive them. This page covers what Ductor produces and how the bundled stack consumes it.

The pipeline at a glance

Bring the stack up locally with docker compose --profile observability up (or make up-full).

Metrics

Prometheus metrics are always exposed on the metrics listener (server.metrics_addr, default :9090) at /metrics. In the compose stack, vmagent scrapes ductor:9090 (job ductor, metrics_path: /metrics, 15s interval) and remote-writes to VictoriaMetrics.

What's covered:

• HTTP/gRPC request metrics (rates, latencies, status).
• Routing decision metrics.
• DAG-runtime metrics — coordinator ticks, wakeups, attempts, queue depth.

On Kubernetes, enable the Prometheus Operator ServiceMonitor:

serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s

Traces

Tracing is off by default. Enable it and point it at an OTLP collector — the compose defaults target VictoriaTraces:

export DUCTOR_TRACING_ENABLED=true
export DUCTOR_TRACING_ENDPOINT=victoriatraces:4317   # OTLP gRPC
export DUCTOR_TRACING_GRPC=true
export DUCTOR_TRACING_INSECURE=true
export DUCTOR_TRACING_SAMPLE_RATE=0.1                # 10% in production

VictoriaTraces listens for OTLP gRPC on :4317 and serves its query API on :10428. For HTTP export, leave tracing.grpc off and use the default HTTP endpoint (http://victoriatraces:10428/insert/opentelemetry/v1/traces). Spans cover the transport → application → infrastructure path, including routing pipeline stages and coordinator ticks. Full walkthrough: Enable tracing & metrics.

Logs

Ductor logs via log/slog in JSON by default (log.format=json). All structured log keys are canonical constants (in pkg/telemetry/logging/meta/), so field names are stable and low-cardinality — safe to index and alert on. In the compose stack, Vector reads container logs off the Docker socket, parses the structured JSON, and ships to VictoriaLogs (:9428), which retains 7 days.

Tune verbosity with log.level (debug/info/warn/error); enable log.add_source=true to include source file/line when debugging.

Alerting

vmalert (:8880) evaluates alerting rules against VictoriaMetrics and routes firing alerts to Alertmanager (:9093). Rules and Alertmanager config live under deploy/observability/ (vmalert.yml, alertmanager.yml), with a RUNBOOK.md alongside. Point Alertmanager at your paging/Slack integration.

Grafana

Grafana runs on :3000 (admin / admin by default; change GF_SECURITY_ADMIN_PASSWORD). Datasources and dashboards are provisioned from deploy/grafana/. Use it to explore metrics, traces, and logs together.

Analytics: the PromQL proxy

The embedded dashboard cannot query VictoriaMetrics directly — the browser has no VM auth, and VM has no tenancy. A tenant-scoped server-side proxy (application/analytics/promquery/) sits in the middle. Enable it under analytics.*:

analytics:
  enabled: false                 # 404s every analytics route when off
  victoria_metrics_url: "http://localhost:8428"
  allow_unscoped_metrics: false  # rejected under security_profile=enterprise

When enabled it mounts POST /api/v1/analytics/query (a PromQL query_range forwarder) and GET /api/v1/analytics/catalog (the K1 registry, read-only). The proxy is the injection defense: it parses the incoming PromQL and rejects anything that is not a recognized catalog metric name (plus a closed allowlist of PromQL functions), then injects the tenant label matcher where the metric actually carries a real tenant_id label. When analytics.enabled is off, all analytics routes 404 and the dashboard falls back to mock data.

Analytics: anomaly detection

The analytics anomaly module (modules/analytics/anomaly/) subscribes to quality signals and flags outliers. It runs a rolling per-metric buffer and supports four detection methods — zscore, iqr, rate_change, and pattern — with tunable thresholds (default z-score 3.0, IQR multiplier 1.5, rate-change drop 0.5). On a detected anomaly it takes one or more configured actions: alert, auto-pause, deprioritize, or log (the default action set is alert + log). Per-metric threshold overrides let you tighten or loosen individual signals without changing the global config.

What to watch

• Readiness — alert on /ready failing (see Health checks).
• Coordinator health — stuck-run and wakeup-dead-letter thresholds (workflow_dag.coordinator.stuck_run_threshold, wakeup_dead_letter_threshold).
• Queue depth and worker saturation — from the routing/queue metrics.
• Error ratio and latency — the reliability.health_signal.* degraded/ unhealthy thresholds mirror good alert boundaries.

Workflow concurrency alerts

Run-concurrency limits are safe only while lease ownership is being renewed and released reliably. Add these signals to the workflow-runtime dashboard:

• Page on any sustained rate of ductor_dag_coordinator_run_concurrency_lease_release_failed_total. The lease will eventually expire, but capacity is temporarily leaked.
• Alert on result="lease_missing", result="store_error", or result="watchdog_schedule_failed" in ductor_dag_coordinator_run_concurrency_lease_renewal_total. These outcomes stop run advancement by design.
• Keep the lower tail of ductor_dag_coordinator_run_concurrency_lease_remaining_seconds comfortably above the renewal interval. A falling margin means Redis latency or scheduler delay is consuming the safety window.
• Compare acquired, denied, and released rates by workflow definition. A sustained denied rate with normal renewal health is capacity pressure, not a lease failure.
• Alert on any sustained ductor_dag_coordinator_run_concurrency_lease_persist_failed_total. The bundled DuctorDAGLeasePersistFailed warning already covers this drift.

The workflow runtime guide documents renewal, early release, resume reacquisition, and orphan reclamation.