Routing & Work Experiments

Experiments let you compare live behavior without losing the connection between the variant Ductor selected and the outcome that arrived later. The experiment service owns the configuration and lifecycle, assigns subjects consistently, persists an immutable exposure receipt for workflow and agent executions, and accepts delayed scores against that exact receipt.

Use an experiment when a variant is allowed to affect live behavior for a controlled slice of traffic. Use a shadow campaign when you need counterfactual evidence with no authoritative side effects.

Every operation is tenant-scoped. Reads require experiment:read; creates, assignments, lifecycle changes, scores, and deletion require experiment:write.

Supported experiment surfaces

subject_scope tells Ductor what the variants control. subject_ref anchors the experiment to the exact resource being changed.

Routing experiments also require pool_id. If subject_scope is omitted, Ductor treats the experiment as routing_strategy and uses pool_id as the subject reference.

Lifecycle

An experiment is created as draft. Start it to admit assignments, pause it to temporarily return traffic to the default behavior, and either complete or stop it when data collection ends.

• draft — configuration state before activation; no live assignments.
• running — the only state that accepts experiment traffic.
• paused — temporarily inactive and restartable.
• completed — concluded with results retained for analysis.
• stopped — manually terminated with results retained.

Starting a routing-strategy experiment validates every named strategy and its options against the live strategy registry. Strategy variants must use deterministic splitting so assignments remain reproducible across retries, replay, and process restarts.

Create an experiment

The following experiment compares two routing strategies for half of a pool's traffic. Variant weights are relative: equal weights produce an even split inside the admitted 50% traffic slice.

curl -s -X POST "https://api.ductor.io/api/experiments" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lead assignment quality",
    "description": "Compare the current strategy with the candidate on qualified conversion",
    "pool_id": "550e8400-e29b-41d4-a716-446655440000",
    "subject_scope": "routing_strategy",
    "subject_ref": "550e8400-e29b-41d4-a716-446655440000",
    "traffic_split_method": "hash",
    "traffic_percentage": 50,
    "variants": [
      {
        "id": "control",
        "name": "Current strategy",
        "strategy": "swrr",
        "weight": 1,
        "is_control": true
      },
      {
        "id": "candidate",
        "name": "Learning strategy",
        "strategy": "epsilon_greedy",
        "strategy_options": { "epsilon": 0.1 },
        "weight": 1
      }
    ],
    "metrics": [
      {
        "name": "qualified_conversion",
        "type": "gauge",
        "minimum_detectable_effect": 0.05
      }
    ]
  }'

Creation rejects configurations with fewer than two variants, no control variant, duplicate variant ids, no positive variant weight, an invalid traffic percentage, or an unavailable routing strategy. Omitted variant ids are generated; omitted variant names default to their ids.

Start the experiment only after the draft matches the hypothesis you intend to measure:

curl -s -X POST \
  "https://api.ductor.io/api/experiments/$EXPERIMENT_ID/start" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT"

Assign a variant and capture exposure

POST /api/experiments/{experiment_id}/assign always requires a stable entity_id. Ductor uses that identity for sticky bucketing, so retries for the same entity return the same arm.

For a workflow or agent experiment, also send at least one execution identity: workflow_run_id, workflow_step_id, or agent_session_id. Ductor then returns an exposure alongside the variant.

curl -s -X POST \
  "https://api.ductor.io/api/experiments/$EXPERIMENT_ID/assign" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "entity_id": "account-42",
    "workflow_run_id": "run-8a12",
    "workflow_step_id": "summarize-lead"
  }'

{
  "variant": {
    "id": "candidate",
    "name": "Concise prompt",
    "weight": 1,
    "is_control": false
  },
  "exposure": {
    "id": "a593d4f0b8c6f48202d78411eeb901da",
    "experiment_id": "exp-7e0f",
    "variant_id": "candidate",
    "entity_id": "account-42",
    "subject_scope": "workflow_step",
    "subject_ref": "summarize-lead",
    "workflow_run_id": "run-8a12",
    "workflow_step_id": "summarize-lead",
    "assigned_at": "2026-08-23T20:14:12Z"
  }
}

The exposure is immutable and tenant-bound. Its identity is derived from the experiment plus the run, step, and agent-session coordinates. Replaying the same execution returns the original receipt even if the experiment configuration changed later. That makes the receipt the durable attribution boundary—not the experiment's current configuration.

Record a delayed score

Real outcomes often arrive after the run that produced them: a conversion, human rating, refund, quality review, latency observation, or model-judge score. Attach that value to its exposure with POST /api/experiments/{experiment_id}/scores.

curl -s -X POST \
  "https://api.ductor.io/api/experiments/$EXPERIMENT_ID/scores" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "exposure_id": "a593d4f0b8c6f48202d78411eeb901da",
    "metric_name": "qualified_conversion",
    "value": 1,
    "idempotency_key": "conversion:order-9182"
  }'

The metric name must be declared on the experiment, the value must be finite, and the exposure must belong to the experiment and authenticated tenant. The idempotency key is part of the observation identity. Retrying the same exposure, metric, and key returns the original observation instead of counting the outcome twice.

Use an idempotency key that names the real-world fact, such as conversion:<order_id>, rating:<review_id>, or judge:<evaluation_id>. Do not use a random key on every retry.

Read results

Results remain readable while an experiment is running and after it is paused, completed, or stopped:

curl -s \
  "https://api.ductor.io/api/experiments/$EXPERIMENT_ID/results" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT"

The response groups observations by variant and returns sample_count plus a metric_means map. Compare only metrics with enough representative samples; an observed mean is evidence, not an automatic promotion decision. Keep business, fairness, and safety guardrails separate from the primary objective.

API surface

Production checklist

• Write the hypothesis, primary metric, guardrails, and stopping rule before starting.
• Use stable entity ids and deterministic splitting for any strategy variant.
• Keep one control variant and use relative weights that match the intended allocation.
• Persist exposure ids with workflow, agent-session, or decision lineage.
• Use deterministic idempotency keys for delayed outcomes.
• Separate score collection from promotion approval; inspect sample quality and guardrails before adopting a winner.
• Pause first when results or instrumentation look wrong. Complete only when the planned analysis is finished; stop when the run is being terminated.

Related