# Synthetic Canaries (/docs/connectors/synthetic-canaries) A synthetic canary checks a connector action using a controlled request and explicit assertions. Manual and scheduled probes enter the governed action executor on the `synthetic_canary` execution surface. They use the configured connection, policy, setup-readiness, admission, and execution-ledger gates. Use canaries alongside [certification](/docs/connectors/certification-and-testing): certification records evidence for promotion, while canaries provide repeated observations of a tenant and environment's connector path. ## Create and verify a probe [#create-and-verify-a-probe] 1. Choose a registered action and a connection in the intended tenant and environment. Use an action whose declared semantics allow a safe probe. 2. Create the definition with `enabled: false` and at least one assertion. 3. Preview it to check configuration and safety eligibility. 4. Run it manually, then inspect the attempt, assertions, and execution record. 5. Enable the definition and the scheduler only after the manual result is acceptable. Read the returned schedule to confirm the effective defaults. The API derives the tenant from the authenticated caller. Authorization uses the `connector_canary` resource with separate `read`, `write`, and `run` actions. | Operation | HTTP endpoint | Permission | | ---------------------- | ------------------------------------------------ | ---------- | | List definitions | `GET /api/connector-canaries` | `read` | | Read a definition | `GET /api/connector-canaries/{id}` | `read` | | Create a definition | `POST /api/connector-canaries` | `write` | | Replace mutable fields | `PUT /api/connector-canaries/{id}` | `write` | | Delete a definition | `DELETE /api/connector-canaries/{id}` | `write` | | Preview eligibility | `POST /api/connector-canaries/{id}/preview` | `read` | | Run one probe | `POST /api/connector-canaries/{id}/run` | `run` | | Read a health rollup | `GET /api/connector-canaries/{canary_id}/result` | `read` | | List health rollups | `GET /api/connector-canaries/results` | `read` | | Explain status | `GET /api/connector-canaries/{id}/explain` | `read` | Create and update requests wrap the mutable fields in `definition`. For example, this request body creates a disabled read-only probe with a ten-second budget. Replace the provider, action, connection, and arguments with values from your deployment; `your_provider` and `get_status` below are illustrative names. ```json title="POST /api/connector-canaries" { "definition": { "name": "Status probe", "target": { "kind": "action", "provider_key": "your_provider", "action_key": "get_status", "connection_id": "00000000-0000-0000-0000-000000000001", "args": {} }, "schedule": { "interval_seconds": 300, "jitter_seconds": 15, "timeout_seconds": 10, "max_attempts": 1, "budget_millis": 10000 }, "assertions": [{ "kind": "expect_success" }], "evidence_policy": { "retention_mode": "redacted_summary", "max_payload_bytes": 8192 }, "enabled": false } } ``` Set `definition.environment_id` when targeting an environment. Supply either `target.connection_id` or `target.connection_route_key`, never both. A manual run is allowed while the definition is disabled; `enabled` controls scheduled eligibility. `preview_ok` is a skipped preview outcome. It confirms schedule and evidence validation, catalog lookup, safety eligibility, and budget resolution. Preview does not resolve the connection, exercise all execution gates, call the provider, or evaluate response assertions. Use a manual run to verify those parts of the path. ## Safety and assertions [#safety-and-assertions] Eligibility comes from the action's canonical semantics. The authored safety policy cannot widen those semantics. | Effective safety class | Requirement | | ----------------------- | -------------------------------------------------------------------------------------- | | `read_only` | Eligible by default | | `idempotent_write` | Explicit `safety_policy.allow_idempotent_write: true` and a bound `sandbox_object_ref` | | `destructive_forbidden` | Rejected, including unknown or irreversible effects | Use `expect_success` for an availability probe. Other supported assertions are `expect_failure`, `latency_ceiling_ms`, `result_size_ceiling_bytes`, `json_path_exists`, `json_path_equals`, and `response_hash_equals`. Combine response-shape or latency assertions with the expected success or failure assertion so the verdict describes what you intended to test. `expect_failure` can verify a provider's error behavior after dispatch. It cannot turn an admission denial, a failure to dispatch, an execution-completion failure, or an exhausted execution budget into a passing canary. ## Cadence and execution budget [#cadence-and-execution-budget] Zero or omitted values receive defaults where shown below. Negative authored values are rejected; they are not silently converted into defaults. Create and update validate the definition before saving it, so an invalid update leaves the stored definition unchanged. | Field | Default | Constraint or behavior | | ----------------------------------- | --------------------------------- | ------------------------------------------------------------------------------ | | `schedule.interval_seconds` | `300` | Between `30` and `86400` | | `schedule.jitter_seconds` | `0` | Non-negative and less than the interval; zero disables jitter | | `schedule.timeout_seconds` | Configured default, normally `30` | Positive and no greater than the interval | | `schedule.budget_millis` | Derived from the timeout | Positive; limits the total probe execution time | | `schedule.max_attempts` | `1` | Validated between `1` and `10`; the runner currently performs one attempt | | `admission.cost_units` | `1` | Positive after normalization; cannot lower the provider's declared action cost | | `admission.priority` | `0` | Between `-100` and `100` | | `evidence_policy.max_payload_bytes` | `8192` | Between `1` and `8192` after normalization | When timeout is omitted, `connector.synthetic_canaries.default_timeout` supplies the default, capped to the interval. Fractional-second defaults retain their millisecond precision through `budget_millis`. An authored smaller budget stays smaller. The effective deadline is the earliest of the timeout, execution budget, and inherited parent deadline. Keep `max_attempts: 1`; larger accepted values do not enable automatic retries. Recurring jitter adds a deterministic delay based on the tenant, canary, and nominal due time. Initial probes are immediately due. Preview and manual runs do not move the scheduled due time. ### Scheduler configuration [#scheduler-configuration] The background scheduler is disabled by default through `connector.synthetic_canaries.enabled`. The API can remain available while the scheduler is disabled, provided connector storage and the action executor are wired. The scheduler's `interval` is its polling cadence; each definition's `schedule.interval_seconds` controls that probe's cadence. Enabling the worker requires positive `interval`, `batch_size`, `max_concurrency`, and `lease_ttl` settings under `connector.synthetic_canaries`. Configuration changes require a restart; see the [configuration reference](/docs/reference/configuration). Scheduled work is claimed through a durable owner lease, renewed during the probe. Losing the lease cancels execution. The worker stops renewal before releasing the claim and records the next due time. Cancellation also leaves a bounded completion window for persistence and release; it does not grant extra provider execution time. ## Read the result before acting [#read-the-result-before-acting] Inspect `attempt.outcome`, `failure_category`, assertion results, and `execution_id` together. A provider response alone does not prove the governed execution completed successfully. | Signal | Meaning and response | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `pass` | The execution satisfied the authored assertions within its budget. Check that the assertions cover the intended behavior. | | `execution_not_dispatched` | Execution failed before provider dispatch. Inspect connection, policy, readiness, and admission evidence. | | `execution_completion_failed` | The provider path ran, but completion failed. Inspect execution-ledger, admission-release, and other completion errors before retrying a write. | | `schedule_invalid` or `evidence_policy_invalid` | The stored configuration fails validation. Repair the definition; no provider call was made. | | `action_not_registered` | The target is absent from the action catalog. Check the deployed provider/action keys. | | `preview_ok` | Configuration eligibility passed. This is not a provider-health observation. | Evidence retention defaults to `redacted_summary`; `hash_only` and `none` retain less. Evidence stores bounded summaries, hashes, and sizes rather than raw provider response bodies. An invalid evidence limit is rejected, not clamped. Runtime validation also rejects invalid stored policies before provider dispatch. The PostgreSQL store persists an attempt, its evidence, and its health rollup in one transaction. Readiness projection is a separate step. Compare the rollup's timestamp and the readiness observation when investigating disagreement; a saved attempt alone does not prove readiness projection succeeded. ## Boundaries to account for [#boundaries-to-account-for] The current runner supports `action` targets. Reserved `sync` and `proxy` target kinds are rejected. Canary-specific `quota_bucket`, `fairness_class`, and `max_in_flight` fields do not establish independent runner quotas or concurrency limits. The scheduler's worker concurrency and shared execution admission still apply, but a schedule lease does not serialize manual and scheduled invocations of the same canary. Admission uses the greater of the authored canary cost and the provider action's cost. These units participate in admission decisions; they are not a monetary charge or an atomic quota debit. Reusing a receipt requires a matching request hash and an admitted, unreleased receipt whose expiry, when present, is still in the future. That expiry check does not provide crash recovery for active direct executions. Do not assume a crashed process automatically releases its reserved capacity. See [execution admission](/docs/connectors/policies-quotas-audit#receipt-reuse-and-canary-cost). Use the attempt and execution IDs for correlation. Do not depend on `admission_receipt_id` in the canary attempt being populated. ## Where to go next [#where-to-go-next] * [Certification & Testing](/docs/connectors/certification-and-testing) — evidence used before promotion. * [Policies, Quotas & Audit](/docs/connectors/policies-quotas-audit) — the shared execution gates. * [Release Verification](/docs/operations/release-verification) — proving the build and critical flows under test. * [Operational Health](/docs/operations/operational-health) — interpret readiness alongside other runtime evidence.