# Inference Proxy (/docs/ai/inference-proxy)
The inference proxy is a newer subsystem. Its config surface and routing behavior
may still shift; treat the specifics here as current-state, not a stability
guarantee.
This vertical is the clearest proof that Ductor's clearing engine is general, not
lead-specific. An LLM request is a unit of ; the providers
(OpenAI, Anthropic, Gemini, Bedrock, Ollama) are s; picking
one by live health and cost is a ; and the per-request cost
is metered like any other. The same machinery that clears a lead to a recipient
load-balances a completion to a provider — the cargo changes, the machine does not.
Every LLM call in Ductor — the [chat agent](/docs/ai/chat-agent), the
[`ai_action` and `ai_agent` workflow steps](/docs/ai/workflow-ai-steps), and
[preset agents](/docs/ai/agent-presets) — routes through one place: the AI inference
service. It is an **OpenAI-compatible, provider-agnostic router** that fronts the
`modules/verticals/ai-inference` proxy, so callers speak one chat-completions format
and the proxy decides which upstream provider actually serves the request. That
indirection is what lets a workflow ask for a model without hard-coding a vendor, and
lets an operator swap providers without touching a single step. Per-request cost
accounting in exact units ships today; it is the same metering that prices any
other worker's work.
Governed chat and workflow agents add a stricter layer: admission pins an explicit,
content-addressed provider/model decision and every proxy request must use it. The
proxy does not silently replace that decision. Any allowed transition needs a new
durable policy receipt.
## Operations [#operations]
The service (`application/ai.Service`) exposes:
| Operation | Shape | Purpose |
| ------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------ |
| `RouteInference` | unary, **non-streaming** | Run one chat completion through the proxy. |
| `RouteInferenceStream` | internal streaming callback | Normalize provider content, tool-call, usage, and terminal fragments for the durable runtime. |
| `ListProviders` / `GetProvider` | read | Enumerate configured providers. |
| `GetHealthStatuses` | read | Per-provider health derived from live stats. |
| `GetUsageStats` | read | Restart-safe request, token, cost, latency, and error rollups per provider+model and time range. |
`RouteInference` is a unary RPC that returns a single aggregated response. A
request with `stream: true` is rejected up front with a clear **400** rather than
a confusing buffered failure. Ductor-hosted agents use the separate internal
`RouteInferenceStream` path, which forwards fragments as they arrive and normalizes
provider-specific frames before the runtime commits them. Public unary callers
should omit `stream`.
Provider **health** is not a separate probe — it is computed from tracked request
outcomes. A provider is healthy when it is enabled and either has processed no
requests yet, or its observed error rate is **below 50%**. Cross that threshold and it
reports unhealthy.
## Providers [#providers]
Providers are configured by name under `ai_inference.providers.`. Five provider
types are recognized:
| Type | Notes |
| ----------- | ------------------------------- |
| `openai` | OpenAI chat-completions format. |
| `anthropic` | Anthropic messages. |
| `gemini` | Google Gemini. |
| `bedrock` | AWS Bedrock (uses `region`). |
| `ollama` | Local / self-hosted models. |
There is no separate "openrouter" type. An OpenRouter account — or any
OpenAI-compatible gateway — is configured as a provider of type `openai` with a
custom `base_url`. That single fact is how those credentials reach the proxy.
## Configuration [#configuration]
```yaml title="config (ai_inference.*)"
ai_inference:
enabled: true # default false
strategy: random_weighted # routing strategy (alias: weighted_random)
max_retries: 2 # attempts after the initial call; -1 = none
retry_backoff: 500ms # base, exponential per attempt
connect_timeout: 10s
response_timeout: 30s
stream_timeout: 10m
circuit_breaker:
enabled: true
consecutive_failures: 5
open_timeout: 30s
reset_interval: 60s
half_open_max_requests: 1
providers:
openrouter:
type: openai
base_url: https://openrouter.ai/api
api_key_env: OPENROUTER_API_KEY
models: [openai/gpt-4o, anthropic/claude-3-5-sonnet]
max_concurrent: 8
weight: 1.0
quality_score: 0.9
headers: {}
enabled: true
```
A separate `ai.routing.*` block bounds spend and model selection:
```yaml title="config (ai.routing.*)"
ai:
routing:
max_cost_per_decision_usd: 1.00 # reject a response whose measured cost exceeds this
monthly_budget_usd: 1000.00 # in-process aggregate spend guard
model_priority: [gpt-4o, gpt-4o-mini] # priority when a non-agent request omits a model
```
Each provider names an environment variable in `api_key_env`; the key is resolved
from that variable at runtime and is **never** stored in a config file — provider
config even redacts the key when serialized. This is how OpenRouter and every other
provider's credentials are passed at deploy time: set the env var the provider
points at, and inject it from a secret manager. A config file never carries a key.
At least one **enabled** provider must be configured when `ai_inference.enabled=true`,
or startup validation fails. Under `security_profile=enterprise`, enabling inference
additionally requires positive `ai.routing.max_cost_per_decision_usd`,
`monthly_budget_usd`, and a non-empty `model_priority`.
## Reliability [#reliability]
The proxy carries the reliability machinery you would otherwise hand-roll per caller:
```mermaid
flowchart LR
C["Caller
unary or stream"] --> S["Routing strategy
(weighted)"]
S --> CB["Circuit breaker
per provider"]
CB --> RT["Retries
(backoff)"]
RT --> U["Upstream provider"]
U --> M["Cost tracking +
usage metering"]
```
* **Routing strategy** picks among enabled providers (e.g. `random_weighted` by the
per-provider `weight`).
* **Per-provider circuit breakers** fail fast after `consecutive_failures`, stay open
for `open_timeout`, then probe half-open.
* **Retries** with exponential backoff (`max_retries`, `retry_backoff`).
* **Cost tracking** feeds `GetUsageStats` and the per-decision / monthly budget
guards. These in-process guards (`ai.routing.max_cost_per_decision_usd` /
`monthly_budget_usd`) are a **separate per-process mechanism** from the durable
tenant-scoped [budget policies](/docs/billing/usage-metering) — they bound a
single proxy's spend, not a tenant's windowed usage across the fleet.
* **Usage metering** records requests, tokens, estimated cost, provider-attempt
latency, and errors on the durable [usage plane](/docs/billing/usage-metering).
For unary calls, the proxy emits the canonical measured amount as
`X-Ductor-Cost-Micros` only after usage is recorded. The application layer validates
that receipt and carries it into the durable agent `CostMicros` counter, so an agent's
`MaxCostMicros` bound is based on Ductor's pricing table rather than inferred from
tokens later. If provider usage or durable accounting is unavailable, an admitted
tenant-budget reservation is settled conservatively at the reserved estimate and the
request fails closed.
When an agent definition declares an output schema, Ductor forwards an enforced
`json_schema` response format through OpenAI-compatible providers (including
OpenRouter) and translates it to Gemini's JSON response schema. The runtime still
validates the final payload itself; provider-side enforcement reduces invalid output
but never replaces Ductor's boundary check.
For a stream, usage is reconciled when the provider supplies its final usage frame or
the request terminates. Replaying already committed chat events does not traverse the
proxy and therefore does not reserve or bill a second provider request.
## Durable usage and attribution [#durable-usage-and-attribution]
`GET /api/ai/usage` now reads the durable usage store rather than process-local
counters. It defaults to the last 24 hours, accepts `from`, `to`,
`provider_name`, and `model` filters, and returns per-provider/model totals for
requests, prompt and completion tokens, estimated cost, average latency, and
errors. Because the source is persisted hourly rollups, results survive restarts
and aggregate across replicas.
Attach runtime identity through the inference request's `metadata` map:
```json
{
"model": "gpt-4o-mini",
"messages": [{ "role": "user", "content": "Summarize this lead" }],
"metadata": {
"agent_definition_id": "lead-analyst-v3",
"agent_session_id": "session-42",
"workflow_run_id": "run-8a12",
"workflow_step_id": "summarize-lead",
"experiment_id": "exp-7e0f",
"experiment_assignment_id": "a593d4f0b8c6f48202d78411eeb901da"
}
}
```
Ductor forwards these supported values as internal attribution headers and
records them as bounded usage dimensions and typed correlations. Other metadata
does not become durable usage attribution. Keep prompts, responses, and
credentials out of metadata; supported identity values are limited to 256 bytes
and reject control delimiters. Use the
`experiment_assignment_id` from the immutable
[experiment exposure](/docs/management/experiments#assign-a-variant-and-capture-exposure),
not a newly generated request id.
Each cost estimate records integer USD micros, the pricing source
`ductor_static_model_cost_table`, and the pricing-table version used. This
provenance makes historical estimates explainable when model prices change; it
does not claim exact parity with a provider invoice.
## Related [#related]
The full ai\_inference.\* and ai.routing.\* reference.
The durable chat host that consumes normalized inference fragments.
ai\_action and ai\_agent steps that route inference from inside a run.