# Observability (/docs/operations/observability)



Ductor emits the three signals — metrics, traces, and logs — and includes a
VictoriaMetrics-based stack to receive them. This page covers what Ductor
produces and how the bundled stack consumes it.

## The pipeline at a glance [#the-pipeline-at-a-glance]

| Signal  | Ductor emits                       | Collected by                           | Stored in                         | Viewed in       |
| ------- | ---------------------------------- | -------------------------------------- | --------------------------------- | --------------- |
| Metrics | Prometheus on `:9090` `/metrics`   | vmagent (scrape)                       | VictoriaMetrics `:8428`           | Grafana `:3000` |
| Traces  | OTLP (HTTP or gRPC)                | — (direct export)                      | VictoriaTraces `:10428` / `:4317` | Grafana         |
| Logs    | structured JSON (`slog`) to stdout | Vector (docker logs)                   | VictoriaLogs `:9428`              | Grafana         |
| Alerts  | (from metrics)                     | vmalert `:8880` → Alertmanager `:9093` | —                                 | Alertmanager    |

```mermaid
flowchart LR
    Ductor["Ductor"] -->|"Prometheus :9090"| VMAgent["vmagent"] --> VM[("VictoriaMetrics")]
    Ductor -->|"OTLP"| VT[("VictoriaTraces")]
    Ductor -->|"JSON logs"| Vector["Vector"] --> VL[("VictoriaLogs")]
    VM --> Grafana["Grafana"]
    VT --> Grafana
    VL --> Grafana
    VM --> VMAlert["vmalert"] --> AM["Alertmanager"]
```

Bring the stack up locally with `docker compose --profile observability up` (or
`make up-full`).

## Metrics [#metrics]

Prometheus metrics are always exposed on the metrics listener
(`server.metrics_addr`, default `:9090`) at `/metrics`. In the compose stack,
**vmagent** scrapes `ductor:9090` (job `ductor`, `metrics_path: /metrics`, 15s
interval) and remote-writes to VictoriaMetrics.

What's covered:

* HTTP/gRPC request metrics (rates, latencies, status).
* Routing decision metrics.
* DAG-runtime metrics — coordinator ticks, wakeups, attempts, queue depth.

On Kubernetes, enable the Prometheus Operator `ServiceMonitor`:

```yaml
serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s
```

## Traces [#traces]

Tracing is **off by default**. Enable it and point it at an OTLP collector — the
compose defaults target VictoriaTraces:

```bash
export DUCTOR_TRACING_ENABLED=true
export DUCTOR_TRACING_ENDPOINT=victoriatraces:4317   # OTLP gRPC
export DUCTOR_TRACING_GRPC=true
export DUCTOR_TRACING_INSECURE=true
export DUCTOR_TRACING_SAMPLE_RATE=0.1                # 10% in production
```

VictoriaTraces listens for OTLP gRPC on `:4317` and serves its query API on
`:10428`. For HTTP export, leave `tracing.grpc` off and use the default HTTP
endpoint (`http://victoriatraces:10428/insert/opentelemetry/v1/traces`). Spans
cover the transport → application → infrastructure path, including routing
pipeline stages and coordinator ticks. Full walkthrough:
[Enable tracing & metrics](/docs/guides/enable-observability).

<Callout type="warn" title="Sample rate and insecure in production">
  Drop `tracing.sample_rate` well below `1.0` under production load, and set
  `tracing.insecure=false` with a TLS endpoint when traces cross a network
  boundary. Use `tracing.pressure.enabled` + `tracing.pressure.mode`
  (`drop`/`block`) to bound exporter backpressure.
</Callout>

## Logs [#logs]

Ductor logs via `log/slog` in JSON by default (`log.format=json`). All structured
log **keys** are canonical constants (in `pkg/telemetry/logging/meta/`), so field
names are stable and low-cardinality — safe to index and alert on. In the compose
stack, **Vector** reads container logs off the Docker socket, parses the
structured JSON, and ships to VictoriaLogs (`:9428`), which retains 7 days.

Tune verbosity with `log.level` (`debug`/`info`/`warn`/`error`); enable
`log.add_source=true` to include source file/line when debugging.

## Alerting [#alerting]

**vmalert** (`:8880`) evaluates alerting rules against VictoriaMetrics and routes
firing alerts to **Alertmanager** (`:9093`). Rules and Alertmanager config live
under `deploy/observability/` (`vmalert.yml`, `alertmanager.yml`), with a
`RUNBOOK.md` alongside. Point Alertmanager at your paging/Slack integration.

## Grafana [#grafana]

Grafana runs on `:3000` (`admin` / `admin` by default; change
`GF_SECURITY_ADMIN_PASSWORD`). Datasources and dashboards are provisioned from
`deploy/grafana/`. Use it to explore metrics, traces, and logs together.

## Analytics: the PromQL proxy [#analytics-the-promql-proxy]

The embedded dashboard **cannot query VictoriaMetrics directly** — the browser
has no VM auth, and VM has no tenancy. A tenant-scoped server-side proxy
(`application/analytics/promquery/`) sits in the middle. Enable it under
`analytics.*`:

```yaml
analytics:
  enabled: false                 # 404s every analytics route when off
  victoria_metrics_url: "http://localhost:8428"
  allow_unscoped_metrics: false  # rejected under security_profile=enterprise
```

When enabled it mounts `POST /api/v1/analytics/query` (a PromQL `query_range`
forwarder) and `GET /api/v1/analytics/catalog` (the K1 registry, read-only).
The proxy is the injection defense: it parses the incoming PromQL and **rejects
anything that is not a recognized catalog metric name** (plus a closed
allowlist of PromQL functions), then injects the tenant label matcher where the
metric actually carries a real `tenant_id` label. When `analytics.enabled` is
off, all analytics routes 404 and the dashboard falls back to mock data.

<Callout title="Check metric scope">
  Metrics without a `tenant_id` label return deployment-wide series. Treat a
  result as global unless its catalog entry explicitly declares tenant scope.
</Callout>

## Analytics: anomaly detection [#analytics-anomaly-detection]

The analytics anomaly module (`modules/analytics/anomaly/`) subscribes to
quality signals and flags outliers. It runs a rolling per-metric buffer and
supports four detection methods — `zscore`, `iqr`, `rate_change`, and
`pattern` — with tunable thresholds (default z-score `3.0`, IQR multiplier
`1.5`, rate-change drop `0.5`). On a detected anomaly it takes one or more
configured actions: `alert`, `auto-pause`, `deprioritize`, or `log` (the
default action set is `alert` + `log`). Per-metric threshold overrides let you
tighten or loosen individual signals without changing the global config.

## What to watch [#what-to-watch]

* **Readiness** — alert on `/ready` failing (see
  [Health checks](/docs/operations/health-checks)).
* **Coordinator health** — stuck-run and wakeup-dead-letter thresholds
  (`workflow_dag.coordinator.stuck_run_threshold`,
  `wakeup_dead_letter_threshold`).
* **Queue depth and worker saturation** — from the routing/queue metrics.
* **Error ratio and latency** — the `reliability.health_signal.*` degraded/
  unhealthy thresholds mirror good alert boundaries.

### Workflow concurrency alerts [#workflow-concurrency-alerts]

Run-concurrency limits are safe only while lease ownership is being renewed and
released reliably. Add these signals to the workflow-runtime dashboard:

* Page on any sustained rate of
  `ductor_dag_coordinator_run_concurrency_lease_release_failed_total`. The lease
  will eventually expire, but capacity is temporarily leaked.
* Alert on `result="lease_missing"`, `result="store_error"`, or
  `result="watchdog_schedule_failed"` in
  `ductor_dag_coordinator_run_concurrency_lease_renewal_total`. These outcomes
  stop run advancement by design.
* Keep the lower tail of
  `ductor_dag_coordinator_run_concurrency_lease_remaining_seconds` comfortably
  above the renewal interval. A falling margin means Redis latency or scheduler
  delay is consuming the safety window.
* Compare acquired, denied, and released rates by workflow definition. A
  sustained denied rate with normal renewal health is capacity pressure, not a
  lease failure.
* Alert on any sustained
  `ductor_dag_coordinator_run_concurrency_lease_persist_failed_total`. The
  bundled `DuctorDAGLeasePersistFailed` warning already covers this drift.

The [workflow runtime guide](/docs/operations/workflow-runtime#run-concurrency-lease-ownership)
documents renewal, early release, resume reacquisition, and orphan reclamation.