Certification & Testing

Proving connector quality — the certification workbench, synthetic canaries, provider API drift intelligence, and the simulator/fixture dev tooling.

An action in the inventory is only as trustworthy as the evidence that it actually works — and trust is what lets a Route or an agent dispatch to it unattended. Ductor has a layered set of quality surfaces: a workbench that turns tests into promotable scorecards, canaries that keep probing production paths, drift intelligence that watches for provider-side API changes, and credential-free dev tooling to exercise connectors without a live upstream.

Evidence here, the declaration next door

Everything on this page — the workbench, scorecards, canaries, and drift intelligence — ships today and produces real evidence of quality. That evidence is now recorded as a declaration on the assurance axis: a certified claim carries its contract-test result, conformance run, canary verdict, author, and an expiry, and both expiry and revocation downgrade an action to community at read time. Assurance ships opt-in via connector_assurance.enabled.

The certification workbench

The certification workbench (docs/connectors/certification-workbench.md) turns coverage and conformance tests into durable, redacted scorecards that routing, marketplace installs, promotion, and headless tools evaluate before production traffic reaches a capability. Status is ordered from no evidence to strongest provider-backed evidence:

(plus the out-of-band states blocked, stale, and waived). Production deployment activation in the enterprise profile requires recorded_replay_pass by default for connector actions unless a more specific requirement is supplied. Normal CI only runs mocked and recorded-replay suites; sandbox and live suites require explicit operator opt-in and provider allowlists.

Evidence is redacted or it is rejected

Scorecard evidence stores labels, bounded metadata, hashes, and sizes — never raw provider payloads. Fixtures are scanned for forbidden content (Authorization, Bearer , xoxb-, ya29., refresh_token, client_secret, access_token); PII must be minimized before a payload becomes a fixture. Prefer synthetic values, deterministic ids, and schema-shaped records over copied customer data.

Marketplace manifests can declare runtime.certification_requirements, and enterprise installs plus deployment promotion evaluate those against tenant-scoped scorecards and fail closed when a requirement is missing, stale, blocked, or below threshold. Waivers exist for bounded operator exceptions — they require an actor, reason, and expiry, and are meant to be replaced with fresh evidence, not extended.

Synthetic canaries

Certification proves a capability at promotion; a synthetic canary (Plan 105, docs/connector-system/synthetic-canaries.md) keeps answering, on a schedule, is this deployed tenant/env/provider/action path healthy right now? It reuses the canonical ExecuteActionWithRequest boundary on a distinct synthetic_canary execution surface, so it inherits every production gate — installation, registry lock, connector policy, setup readiness, execution admission/fairness, budget, circuit breaker, and the execution ledger. Canaries never bypass a gate.

Eligibility is derived from canonical action semantics and is fail-closed:

read_only — eligible by default.
idempotent_write — eligible only with an explicit governed opt-in (safety_policy.allow_idempotent_write = true) and a bound sandbox object.
destructive_forbidden — any destructive, irreversible, or unknown-effect action is never eligible; no policy can escalate it. An ineligible target is surfaced as misconfigured and never reaches the provider.

The scheduler is disabled by default (connector.synthetic_canaries.enabled) and guarded by a durable schedule lease, so many workers can run it without double-firing. Evidence defaults to redacted_summary (secret-redacted, size-bounded body); modes none and hash_only store even less. Canary rollups feed availability readiness, and can be adapted into rollout evidence packs, provider-drift findings, and certification-staleness signals — but the canary runtime never pushes into those systems directly.

Provider API drift intelligence

Provider API drift intelligence (Plan 090, docs/connectors/provider-api-drift-intelligence.md) compares Ductor connector evidence against pinned or governed provider metadata. It is an evidence and governance plane only: drift runs produce snapshots, findings, advisories, impact previews, and promotion-block decisions, but they never edit live connector definitions, workflows, routing bundles, mappings, permissions, or credentials.

Deterministic local sources (the Ductor action corpus, bootstrap manifests, operator-pinned integration-template metadata, local OpenAPI files) are safe for normal CI. Live docs/changelog/OpenAPI probes are disabled by default and, when enabled, require tenant/environment scope, an egress policy ref, rate limits, timeouts, and credential-free requests. Findings classify endpoint, method, path-template, schema, scope, auth-mode, API-version, rate-limit, deprecation, and sunset changes; promotion preflight blocks unacknowledged breaking or sunset findings.

Dev and test tooling

Two credential-free harnesses let provider authors exercise connectors without a live upstream:

Connector simulator (infrastructure/connector/simulator) is the provider — a credential-free, deterministic in-memory Store implementing the real connector operation contracts, generating sync, event, and failure behavior from check-in-able scenarios. Its FailureMode vocabulary (rate_limit, timeout, provider_error, validation_error, auth_expired, stale_cursor, malformed_payload) maps onto the real execution-error constructors, so simulated failures classify exactly like real ones.
Fixture harness (infrastructure/connector/fixtureharness) validates a provider's HTTP behavior against a declarative mock server. Scenario builders cover Fixed, Conditional, Switch, Sequence, Paginated, RateLimited, Malformed, Timeout, and ProviderError; comparators assert only the request shape, result subset, error envelope, or effect-ledger receipt that matters.

The simulator is never routable in production

The simulator is recognized by domain/connector.IsSimulatorProviderKey, and both ExecuteActionWithRequest and ConnectorRouter.Select reject it with ErrSimulatorProviderNotRoutable unless the caller explicitly sets AllowSimulator. It is wired only into tests, certification, conformance, and sandbox harnesses — production routing can never select it by accident. A simulator pass ranks 15 on the certification ladder (above metadata_only, below mocked_pass), so it can never graduate a connector on its own.

Both harnesses bridge into the certification workbench (a fixture pass lands as mocked_pass), and connectortest separately validates provider registration metadata. The external component corpus intake is a read-only reference set — Ductor does not import, execute, alias, or translate third-party source automatically; those candidates stay blocked_by_policy.

Where to go next

Marketplace & Deployments — how certification requirements gate installs and promotion.
Building a Provider — authoring the actions these harnesses exercise.
Policies, Quotas & Audit — the production gates a canary inherits.