# Enable tracing & metrics (/docs/guides/enable-observability)



Ductor emits OpenTelemetry traces and Prometheus metrics out of the box. This
guide turns on trace export and confirms metrics are scraping. For the full
production observability topology, see
[Observability](/docs/operations/observability).

## Metrics — already on [#metrics--already-on]

Ductor always exposes Prometheus metrics on the metrics listener at `/metrics`:

```bash
curl -s http://localhost:9090/metrics | head
```

The address is `server.metrics_addr` (default `:9090`). To scrape it, point your
Prometheus-compatible agent at `ductor:9090` with `metrics_path: /metrics`. In
the repo's compose stack, **vmagent** does this for you (job `ductor`, 15s
interval) and remote-writes to VictoriaMetrics.

On Kubernetes, enable the bundled `ServiceMonitor` so Prometheus Operator
discovers the target:

```yaml
# Helm values
serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s
```

## Traces — turn on OTLP export [#traces--turn-on-otlp-export]

Tracing is **off by default** (`tracing.enabled=false`). Enable it and point it
at an OTLP collector. Two transports are supported — HTTP (default) and gRPC.

<Tabs items="[&#x22;gRPC&#x22;, &#x22;HTTP&#x22;]">
  <Tab value="gRPC">
    The compose observability profile runs VictoriaTraces with an OTLP gRPC listener
    on `:4317`. Match the compose defaults:

    ```bash
    export DUCTOR_TRACING_ENABLED=true
    export DUCTOR_TRACING_ENDPOINT=victoriatraces:4317
    export DUCTOR_TRACING_GRPC=true
    export DUCTOR_TRACING_INSECURE=true
    export DUCTOR_TRACING_SAMPLE_RATE=1.0     # sample everything in dev
    ```
  </Tab>

  <Tab value="HTTP">
    Leave `tracing.grpc` off (the default) to use the HTTP exporter. The default
    endpoint targets VictoriaTraces' HTTP OTLP path:

    ```bash
    export DUCTOR_TRACING_ENABLED=true
    # default endpoint when grpc=false:
    export DUCTOR_TRACING_ENDPOINT="http://victoriatraces:10428/insert/opentelemetry/v1/traces"
    export DUCTOR_TRACING_SAMPLE_RATE=0.1     # sample 10% in production
    ```
  </Tab>
</Tabs>

The equivalent `serve` flags exist too: `--tracing-enabled`,
`--tracing-endpoint`, `--tracing-grpc`, `--tracing-insecure`,
`--tracing-sample-rate`, `--tracing-backend`.

### Tracing config reference [#tracing-config-reference]

<TypeTable
  type="{
  &#x22;tracing.enabled&#x22;: { description: &#x22;Master switch.&#x22;, type: &#x22;DUCTOR_TRACING_ENABLED&#x22;, default: &#x22;false&#x22; },
  &#x22;tracing.endpoint&#x22;: { description: &#x22;OTLP target.&#x22;, type: &#x22;DUCTOR_TRACING_ENDPOINT&#x22;, default: &#x22;http://victoriatraces:10428/insert/opentelemetry/v1/traces&#x22; },
  &#x22;tracing.backend&#x22;: { description: &#x22;Backend hint.&#x22;, type: &#x22;DUCTOR_TRACING_BACKEND&#x22;, default: &#x22;victoriatraces&#x22; },
  &#x22;tracing.grpc&#x22;: { description: &#x22;gRPC exporter instead of HTTP.&#x22;, type: &#x22;DUCTOR_TRACING_GRPC&#x22;, default: &#x22;false&#x22; },
  &#x22;tracing.insecure&#x22;: { description: &#x22;Disable TLS on the exporter.&#x22;, type: &#x22;DUCTOR_TRACING_INSECURE&#x22;, default: &#x22;true&#x22; },
  &#x22;tracing.sample_rate&#x22;: { description: &#x22;Head sampling fraction.&#x22;, type: &#x22;DUCTOR_TRACING_SAMPLE_RATE&#x22;, default: &#x22;0.1&#x22; },
  &#x22;tracing.pressure.enabled&#x22;: { description: &#x22;Backpressure on the exporter.&#x22;, type: &#x22;DUCTOR_TRACING_PRESSURE_ENABLED&#x22;, default: &#x22;false&#x22; },
  &#x22;tracing.pressure.mode&#x22;: { description: &#x22;drop or block under pressure.&#x22;, type: &#x22;DUCTOR_TRACING_PRESSURE_MODE&#x22;, default: &#x22;drop&#x22; },
}"
/>

<Callout type="warn" title="insecure is dev-only">
  `tracing.insecure=true` disables TLS on the trace exporter — fine inside a
  trusted compose network, but set it to `false` and use a TLS endpoint when
  exporting across a network boundary.
</Callout>

## Verify traces are flowing [#verify-traces-are-flowing]

With the observability profile up, open Grafana at `http://localhost:3000`
(`admin` / `admin`) and query the VictoriaTraces datasource, or hit
VictoriaTraces directly on `:10428`. Drive a request through Ductor (create a
pool, run a workflow) and you should see spans appear.

## What you get [#what-you-get]

* **Metrics** — HTTP/gRPC request metrics, routing decision metrics, and
  DAG-runtime metrics (coordinator ticks, wakeups, attempts).
* **Traces** — spans across the transport → application → infrastructure path,
  including routing pipeline stages and coordinator ticks.
* **Logs** — structured `log/slog` output (JSON by default) that the Vector
  pipeline ships to VictoriaLogs in the compose stack.

## Next steps [#next-steps]

<Cards>
  <Card title="Observability" href="/docs/operations/observability">
    The full traces / metrics / logs / alerting topology.
  </Card>

  <Card title="Health checks & readiness" href="/docs/operations/health-checks">
    What `/health`, `/livez`, and `/ready` actually verify.
  </Card>
</Cards>