Certification & Testing

Proving connector quality — the certification workbench, synthetic canaries, provider API drift intelligence, and the simulator/fixture dev tooling.

An action in the inventory is only as trustworthy as the evidence that it actually works — and trust is what lets a Route or an agent dispatch to it unattended. Ductor has a layered set of quality surfaces: a workbench that turns tests into promotable scorecards, canaries that keep probing production paths, drift intelligence that watches for provider-side API changes, and credential-free dev tooling to exercise connectors without a live upstream.

Evidence here, the declaration next door

Workbench runs, scorecards, canary attempts, and drift findings provide different kinds of evidence. Check their scope, freshness, and outcome before using them to authorize production traffic. Evidence is recorded on the assurance axis: a certified claim carries its contract-test result, conformance run, canary verdict, author, and an expiry, and both expiry and revocation downgrade an action to community at read time. Assurance ships opt-in via connector_assurance.enabled.

The certification workbench

The certification workbench (docs/connectors/certification-workbench.md) turns coverage and conformance tests into durable, redacted scorecards that routing, marketplace installs, promotion, and headless tools evaluate before production traffic reaches a capability. Status is ordered from no evidence to strongest provider-backed evidence:

(plus the out-of-band states blocked, stale, and waived). Production deployment activation in the enterprise profile requires recorded_replay_pass by default for connector actions unless a more specific requirement is supplied. Normal CI only runs mocked and recorded-replay suites; sandbox and live suites require explicit operator opt-in and provider allowlists.

Evidence is redacted or it is rejected

Scorecard evidence stores labels, bounded metadata, hashes, and sizes — never raw provider payloads. Fixtures are scanned for forbidden content (Authorization, Bearer , xoxb-, ya29., refresh_token, client_secret, access_token); PII must be minimized before a payload becomes a fixture. Prefer synthetic values, deterministic ids, and schema-shaped records over copied customer data.

Marketplace manifests can declare runtime.certification_requirements, and enterprise installs plus deployment promotion evaluate those against tenant-scoped scorecards and fail closed when a requirement is missing, stale, blocked, or below threshold. Waivers exist for bounded operator exceptions — they require an actor, reason, and expiry, and are meant to be replaced with fresh evidence, not extended.

Synthetic canaries

Certification records evidence at promotion; a synthetic canary repeatedly checks a connector action in a tenant and environment. Manual and scheduled runs use the canonical ExecuteActionWithRequest boundary on the synthetic_canary execution surface, including the configured policy, setup-readiness, admission, budget, and execution-ledger gates. Preview checks configuration and eligibility; it does not call the provider or prove those execution gates will pass.

Eligibility is derived from canonical action semantics and is fail-closed:

read_only — eligible by default.
idempotent_write — eligible only with an explicit governed opt-in (safety_policy.allow_idempotent_write = true) and a bound sandbox object.
destructive_forbidden — any destructive, irreversible, or unknown-effect action is never eligible; no policy can escalate it. An ineligible target is surfaced as misconfigured and never reaches the provider.

The scheduler is disabled by default (connector.synthetic_canaries.enabled). Scheduled probes use renewable durable leases, while manual probes run on demand. Evidence defaults to redacted_summary; modes none and hash_only retain less. Attempts, evidence, and health rollups are persisted together, with availability-readiness projection performed separately.

See Synthetic Canaries for the API, schedule and evidence limits, failure interpretation, and the boundaries of retry and concurrency controls. A passing preview or a provider response alone is insufficient evidence of a healthy governed execution.

Provider API drift intelligence

Provider API drift intelligence (Plan 090, docs/connectors/provider-api-drift-intelligence.md) compares Ductor connector evidence against pinned or governed provider metadata. It is an evidence and governance plane only: drift runs produce snapshots, findings, advisories, impact previews, and promotion-block decisions, but they never edit live connector definitions, workflows, routing bundles, mappings, permissions, or credentials.

Deterministic local sources (the Ductor action corpus, bootstrap manifests, operator-pinned integration-template metadata, local OpenAPI files) are safe for normal CI. Live docs/changelog/OpenAPI probes are disabled by default and, when enabled, require tenant/environment scope, an egress policy ref, rate limits, timeouts, and credential-free requests. Findings classify endpoint, method, path-template, schema, scope, auth-mode, API-version, rate-limit, deprecation, and sunset changes; promotion preflight blocks unacknowledged breaking or sunset findings.

Dev and test tooling

Two credential-free harnesses let provider authors exercise connectors without a live upstream:

Connector simulator (infrastructure/connector/simulator) is the provider — a credential-free, deterministic in-memory Store implementing the real connector operation contracts, generating sync, event, and failure behavior from check-in-able scenarios. Its FailureMode vocabulary (rate_limit, timeout, provider_error, validation_error, auth_expired, stale_cursor, malformed_payload) maps onto the real execution-error constructors, so simulated failures classify exactly like real ones.
Fixture harness (infrastructure/connector/fixtureharness) validates a provider's HTTP behavior against a declarative mock server. Scenario builders cover Fixed, Conditional, Switch, Sequence, Paginated, RateLimited, Malformed, Timeout, and ProviderError; comparators assert only the request shape, result subset, error envelope, or effect-ledger receipt that matters.

The simulator is never routable in production

The simulator is recognized by domain/connector.IsSimulatorProviderKey, and both ExecuteActionWithRequest and ConnectorRouter.Select reject it with ErrSimulatorProviderNotRoutable unless the caller explicitly sets AllowSimulator. It is wired only into tests, certification, conformance, and sandbox harnesses — production routing can never select it by accident. A simulator pass ranks 15 on the certification ladder (above metadata_only, below mocked_pass), so it can never graduate a connector on its own.

Both harnesses bridge into the certification workbench (a fixture pass lands as mocked_pass), and connectortest separately validates provider registration metadata. The external component corpus intake is a read-only reference set — Ductor does not import, execute, alias, or translate third-party source automatically; those candidates stay blocked_by_policy.

Where to go next

Marketplace & Deployments — how certification requirements gate installs and promotion.
Building a Provider — authoring the actions these harnesses exercise.
Policies, Quotas & Audit — the production gates a canary inherits.