Inference Proxy

This vertical is the clearest proof that Ductor's clearing engine is general, not lead-specific. An LLM request is a unit of Work; the providers (OpenAI, Anthropic, Gemini, Bedrock, Ollama) are Workers; picking one by live health and cost is a Route; and the per-request cost is metered like any other. The same machinery that clears a lead to a recipient load-balances a completion to a provider — the cargo changes, the machine does not.

Every LLM call in Ductor — the chat agent, the ai_action and ai_agent workflow steps, and preset agents — routes through one place: the AI inference service. It is an OpenAI-compatible, provider-agnostic router that fronts the modules/verticals/ai-inference proxy, so callers speak one chat-completions format and the proxy decides which upstream provider actually serves the request. That indirection is what lets a workflow ask for a model without hard-coding a vendor, and lets an operator swap providers without touching a single step. Per-request cost accounting in exact units ships today; it is the same metering that prices any other worker's work.

Governed chat and workflow agents add a stricter layer: admission pins an explicit, content-addressed provider/model decision and every proxy request must use it. The proxy does not silently replace that decision. Any allowed transition needs a new durable policy receipt.

Operations

The service (application/ai.Service) exposes:

Provider health is not a separate probe — it is computed from tracked request outcomes. A provider is healthy when it is enabled and either has processed no requests yet, or its observed error rate is below 50%. Cross that threshold and it reports unhealthy.

Providers

Providers are configured by name under ai_inference.providers.<name>. Five provider types are recognized:

Configuration

ai_inference:
  enabled: true                    # default false
  strategy: random_weighted        # routing strategy (alias: weighted_random)
  max_retries: 2                   # attempts after the initial call; -1 = none
  retry_backoff: 500ms             # base, exponential per attempt
  connect_timeout: 10s
  response_timeout: 30s
  stream_timeout: 10m
  circuit_breaker:
    enabled: true
    consecutive_failures: 5
    open_timeout: 30s
    reset_interval: 60s
    half_open_max_requests: 1
  providers:
    openrouter:
      type: openai
      base_url: https://openrouter.ai/api
      api_key_env: OPENROUTER_API_KEY
      models: [openai/gpt-4o, anthropic/claude-3-5-sonnet]
      max_concurrent: 8
      weight: 1.0
      quality_score: 0.9
      headers: {}
      enabled: true

A separate ai.routing.* block bounds spend and model selection:

ai:
  routing:
    max_cost_per_decision_usd: 1.00   # reject a response whose measured cost exceeds this
    monthly_budget_usd: 1000.00       # in-process aggregate spend guard
    model_priority: [gpt-4o, gpt-4o-mini]  # priority when a non-agent request omits a model

At least one enabled provider must be configured when ai_inference.enabled=true, or startup validation fails. Under security_profile=enterprise, enabling inference additionally requires positive ai.routing.max_cost_per_decision_usd, monthly_budget_usd, and a non-empty model_priority.

Reliability

The proxy carries the reliability machinery you would otherwise hand-roll per caller:

• Routing strategy picks among enabled providers (e.g. random_weighted by the per-provider weight).
• Per-provider circuit breakers fail fast after consecutive_failures, stay open for open_timeout, then probe half-open.
• Retries with exponential backoff (max_retries, retry_backoff).
• Cost tracking feeds GetUsageStats and the per-decision / monthly budget guards. These in-process guards (ai.routing.max_cost_per_decision_usd / monthly_budget_usd) are a separate per-process mechanism from the durable tenant-scoped budget policies — they bound a single proxy's spend, not a tenant's windowed usage across the fleet.
• Usage metering records requests, tokens, estimated cost, provider-attempt latency, and errors on the durable usage plane.

For unary calls, the proxy emits the canonical measured amount as X-Ductor-Cost-Micros only after usage is recorded. The application layer validates that receipt and carries it into the durable agent CostMicros counter, so an agent's MaxCostMicros bound is based on Ductor's pricing table rather than inferred from tokens later. If provider usage or durable accounting is unavailable, an admitted tenant-budget reservation is settled conservatively at the reserved estimate and the request fails closed.

When an agent definition declares an output schema, Ductor forwards an enforced json_schema response format through OpenAI-compatible providers (including OpenRouter) and translates it to Gemini's JSON response schema. The runtime still validates the final payload itself; provider-side enforcement reduces invalid output but never replaces Ductor's boundary check.

For a stream, usage is reconciled when the provider supplies its final usage frame or the request terminates. Replaying already committed chat events does not traverse the proxy and therefore does not reserve or bill a second provider request.

Durable usage and attribution

GET /api/ai/usage now reads the durable usage store rather than process-local counters. It defaults to the last 24 hours, accepts from, to, provider_name, and model filters, and returns per-provider/model totals for requests, prompt and completion tokens, estimated cost, average latency, and errors. Because the source is persisted hourly rollups, results survive restarts and aggregate across replicas.

Attach runtime identity through the inference request's metadata map:

{
  "model": "gpt-4o-mini",
  "messages": [{ "role": "user", "content": "Summarize this lead" }],
  "metadata": {
    "agent_definition_id": "lead-analyst-v3",
    "agent_session_id": "session-42",
    "workflow_run_id": "run-8a12",
    "workflow_step_id": "summarize-lead",
    "experiment_id": "exp-7e0f",
    "experiment_assignment_id": "a593d4f0b8c6f48202d78411eeb901da"
  }
}

Ductor forwards these supported values as internal attribution headers and records them as bounded usage dimensions and typed correlations. Other metadata does not become durable usage attribution. Keep prompts, responses, and credentials out of metadata; supported identity values are limited to 256 bytes and reject control delimiters. Use the experiment_assignment_id from the immutable experiment exposure, not a newly generated request id.

Each cost estimate records integer USD micros, the pricing source ductor_static_model_cost_table, and the pricing-table version used. This provenance makes historical estimates explainable when model prices change; it does not claim exact parity with a provider invoice.

Related