Provider-Aware Prompt Caching

Stable tenant-bound cache prefixes, native provider hints, normalized cache usage, and cache-aware cost accounting.

Cache invariantReuse the prefix. Never loosen the tenant, lineage, or prompt contract.

Identity: Tenant + immutable definition + exact projected prefix
Semantics: Cache hit and miss produce the same model request
Accounting: Read, write, input, output, and fixed costs settle separately

Provider prompt caching can reduce latency and input cost without changing the model-visible prompt. Ductor enables it only when an immutable definition proves a stable, tenant-scoped prefix. The feature is off by default.

Definition policy

{
  "ref": { "id": "support-agent", "version": 7 },
  "provider": "openai",
  "model": "gpt-4.1",
  "model_policy": {
    "prompt_cache": {
      "enabled": true,
      "eligible_segments": ["system", "tools"],
      "min_prefix_tokens": 1024,
      "ttl_seconds": 300,
      "fail_open": true
    }
  }
}

TTL must be 300, 3600, or 86400 seconds. Ductor currently requires fail_open: true: an adapter without cache support sends the unchanged request without a cache hint. Invalid policy, mutable definition identity, or missing tenant identity fails before inference.

The cache identity covers tenant, definition and version, provider/model, exact projected leading system messages, tool definitions, skill and processor pins, projection version, and compaction lineage. Any meaningful change produces a new identity.

Provider behavior

Provider	Ductor behavior
OpenAI	Sends `prompt_cache_key`; 24-hour policy also sends `prompt_cache_retention: "24h"`
Anthropic	Emits request-level ephemeral cache control for 5-minute and 1-hour policies
Gemini	Uses provider-managed implicit caching and reads cached-content token usage
Bedrock	Ignores unsupported hints while preserving the original request

Cache hit and miss must produce identical messages, tools, structured output, and replay behavior.

Usage, pricing, and budgets

Ductor normalizes total input tokens plus separate cache-read and cache-write tokens. Model pricing may declare cache_read_price_per_token and cache_write_price_per_token. If absent, the normal input rate is charged conservatively; an explicit zero is honored.

Budget settlement includes uncached input, cache reads, cache creation, output, and fixed request cost. Durable rollups include ai.tokens.cache_read, ai.tokens.cache_write, ai.cache_read.cost.usd, and ai.cache_write.cost.usd.

Caching is a data-governance decision

Ductor stores only hashes and accounting metadata, but provider-side caches still process prompt content. Confirm retention, region, zero-data-retention, and contractual controls before enabling caching for a definition.

Safe rollout

Pin cache read/write prices and approve provider data handling.
Enable one immutable definition version with a conservative minimum prefix.
Compare normalized usage and cost with provider invoices.
Prove hit/miss semantic equivalence and expected invalidation.
Promote or disable with a new immutable definition version.

Inference Proxy

Provider routing, usage, cost, health, and circuit breaking.

Durable Runtime

Deterministic projection and compaction lineage.