Synthetic Canaries

A synthetic canary checks a connector action using a controlled request and explicit assertions. Manual and scheduled probes enter the governed action executor on the synthetic_canary execution surface. They use the configured connection, policy, setup-readiness, admission, and execution-ledger gates.

Use canaries alongside certification: certification records evidence for promotion, while canaries provide repeated observations of a tenant and environment's connector path.

Create and verify a probe

1. Choose a registered action and a connection in the intended tenant and environment. Use an action whose declared semantics allow a safe probe.
2. Create the definition with enabled: false and at least one assertion.
3. Preview it to check configuration and safety eligibility.
4. Run it manually, then inspect the attempt, assertions, and execution record.
5. Enable the definition and the scheduler only after the manual result is acceptable. Read the returned schedule to confirm the effective defaults.

The API derives the tenant from the authenticated caller. Authorization uses the connector_canary resource with separate read, write, and run actions.

Create and update requests wrap the mutable fields in definition. For example, this request body creates a disabled read-only probe with a ten-second budget. Replace the provider, action, connection, and arguments with values from your deployment; your_provider and get_status below are illustrative names.

{
  "definition": {
    "name": "Status probe",
    "target": {
      "kind": "action",
      "provider_key": "your_provider",
      "action_key": "get_status",
      "connection_id": "00000000-0000-0000-0000-000000000001",
      "args": {}
    },
    "schedule": {
      "interval_seconds": 300,
      "jitter_seconds": 15,
      "timeout_seconds": 10,
      "max_attempts": 1,
      "budget_millis": 10000
    },
    "assertions": [{ "kind": "expect_success" }],
    "evidence_policy": {
      "retention_mode": "redacted_summary",
      "max_payload_bytes": 8192
    },
    "enabled": false
  }
}

Set definition.environment_id when targeting an environment. Supply either target.connection_id or target.connection_route_key, never both. A manual run is allowed while the definition is disabled; enabled controls scheduled eligibility.

Safety and assertions

Eligibility comes from the action's canonical semantics. The authored safety policy cannot widen those semantics.

Use expect_success for an availability probe. Other supported assertions are expect_failure, latency_ceiling_ms, result_size_ceiling_bytes, json_path_exists, json_path_equals, and response_hash_equals. Combine response-shape or latency assertions with the expected success or failure assertion so the verdict describes what you intended to test.

expect_failure can verify a provider's error behavior after dispatch. It cannot turn an admission denial, a failure to dispatch, an execution-completion failure, or an exhausted execution budget into a passing canary.

Cadence and execution budget

Zero or omitted values receive defaults where shown below. Negative authored values are rejected; they are not silently converted into defaults. Create and update validate the definition before saving it, so an invalid update leaves the stored definition unchanged.

When timeout is omitted, connector.synthetic_canaries.default_timeout supplies the default, capped to the interval. Fractional-second defaults retain their millisecond precision through budget_millis. An authored smaller budget stays smaller. The effective deadline is the earliest of the timeout, execution budget, and inherited parent deadline. Keep max_attempts: 1; larger accepted values do not enable automatic retries.

Recurring jitter adds a deterministic delay based on the tenant, canary, and nominal due time. Initial probes are immediately due. Preview and manual runs do not move the scheduled due time.

Scheduler configuration

The background scheduler is disabled by default through connector.synthetic_canaries.enabled. The API can remain available while the scheduler is disabled, provided connector storage and the action executor are wired.

The scheduler's interval is its polling cadence; each definition's schedule.interval_seconds controls that probe's cadence. Enabling the worker requires positive interval, batch_size, max_concurrency, and lease_ttl settings under connector.synthetic_canaries. Configuration changes require a restart; see the configuration reference.

Scheduled work is claimed through a durable owner lease, renewed during the probe. Losing the lease cancels execution. The worker stops renewal before releasing the claim and records the next due time. Cancellation also leaves a bounded completion window for persistence and release; it does not grant extra provider execution time.

Read the result before acting

Inspect attempt.outcome, failure_category, assertion results, and execution_id together. A provider response alone does not prove the governed execution completed successfully.

Evidence retention defaults to redacted_summary; hash_only and none retain less. Evidence stores bounded summaries, hashes, and sizes rather than raw provider response bodies. An invalid evidence limit is rejected, not clamped. Runtime validation also rejects invalid stored policies before provider dispatch.

The PostgreSQL store persists an attempt, its evidence, and its health rollup in one transaction. Readiness projection is a separate step. Compare the rollup's timestamp and the readiness observation when investigating disagreement; a saved attempt alone does not prove readiness projection succeeded.

Boundaries to account for

The current runner supports action targets. Reserved sync and proxy target kinds are rejected. Canary-specific quota_bucket, fairness_class, and max_in_flight fields do not establish independent runner quotas or concurrency limits. The scheduler's worker concurrency and shared execution admission still apply, but a schedule lease does not serialize manual and scheduled invocations of the same canary.

Admission uses the greater of the authored canary cost and the provider action's cost. These units participate in admission decisions; they are not a monetary charge or an atomic quota debit. Reusing a receipt requires a matching request hash and an admitted, unreleased receipt whose expiry, when present, is still in the future. That expiry check does not provide crash recovery for active direct executions. Do not assume a crashed process automatically releases its reserved capacity. See execution admission.

Use the attempt and execution IDs for correlation. Do not depend on admission_receipt_id in the canary attempt being populated.

Where to go next

• Certification & Testing — evidence used before promotion.
• Policies, Quotas & Audit — the shared execution gates.
• Release Verification — proving the build and critical flows under test.
• Operational Health — interpret readiness alongside other runtime evidence.