# Release Verification (/docs/operations/release-verification) A release needs evidence for the exact source revision and artifact being promoted. Record what ran, whether it passed, and which environment it exercised. A package-level pass can coexist with skipped tests, and a focused regression run covers less than the full release gate. Run the commands on this page from the **Ductor Go repository**, not the frontend workspace. Use the repository-managed cache through Make targets or `scripts/dev-cache.sh`; all local worktrees share that bounded cache. ## Choose the right gate [#choose-the-right-gate] | Command | Coverage | | -------------------------------- | ----------------------------------------------------------------------------------------------------- | | `make ci` | Fast development gate for changed packages | | `make ci-release` | Full static analysis, generated/contract checks, unit-level race and coverage tests, and binary build | | `make test-integration-storage` | Real PostgreSQL storage and migration integration tests | | `make test-integration-workflow` | Workflow integration tests | | `make test-e2e-critical` | Required routing, durable-event, replay, restart, and journey flows | | `make ci-release-full` | `ci-release` plus storage, workflow, and critical end-to-end gates | `make validate-config`, `make docs-check`, and `make openapi-check` verify configuration and documentation/API consistency. They do not replace the test, lint, integration, or build gates. Focused tests are useful during development; run the full gate again after the final source change before assigning release evidence to that revision. ## Verify with isolated dependencies [#verify-with-isolated-dependencies] Start from a clean checkout of the intended revision, with the repository's required toolchain installed and Docker running. The hermetic wrapper creates isolated TimescaleDB/PostgreSQL and Dragonfly containers, applies the schema baseline, supplies test connection settings, and removes its containers on exit. ```bash git status --short git rev-parse HEAD make validate-config ./scripts/dev-cache.sh run -- ./scripts/run-hermetic-tests.sh make ci-release-full make verify-build-identity ./bin/ductor version --output=json ``` Preserve the full command output, exit status, test and coverage artifacts, and the reported dependency/toolchain versions with the release record. A failed prerequisite or unavailable database is a failed verification run. Resolve the cause and rerun the affected gate; do not substitute a skipped suite or a lint exclusion for passing evidence. `make verify-build-identity` builds a disposable binary and verifies that its embedded commit equals the full source revision. Separately inspect the `bin/ductor` artifact built by the release gate: its version JSON exposes `version`, `commit`, `build_time`, and `dirty`. The commit must match the verified revision and `dirty` must be false. Record the digest of the image or binary actually promoted as well; a matching commit does not identify an artifact on its own. ## Critical tests must execute [#critical-tests-must-execute] `make test-e2e-critical` requires these ten test cases to run and pass: | Test package | Required case | What it exercises | | ------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `tests/e2e` | `TestE2ESuite/TestDecisionRecording` | Persisted routing decisions | | `tests/e2e` | `TestE2ESuite/TestCatchallDelayE2E` | Delayed catchall behavior | | `tests/e2e` | `TestE2ESuite/TestCatchallImmediateE2E` | Immediate catchall behavior | | `tests/e2e` | `TestE2ESuite/TestCatchallWithPrimaryAvailableE2E` | Primary routing when a catchall is present | | `tests/e2e` | `TestE2ESuite/TestRouteEmitsDurableEvents` | Persisted routing request and assignment events with tenant isolation | | `tests/e2e` | `TestE2ESuite/TestEventSourcedReplay` | Event-backed replay without changing the live decision or its history | | `tests/e2e/restartmatrix` | `TestRestartAfterRoutingLifecycleCommitDrainsEffectIntentsExactlyOnce` | Recovery of committed routing effects after restart | | `tests/e2e/restartmatrix` | `TestJourneyE2E_MultiPool_CrashRestart_ExactlyOnceAcrossHops` | Multi-pool journey recovery across a crash | | `tests/e2e/restartmatrix` | `TestJourneyE2E_BudgetBreachEmitsTypedEffectIntentNotSpawn` | Typed budget-breach effects without spawning a child | | `tests/e2e/restartmatrix` | `TestRoutingEnrichStep_RealPGConnectorLedgerReplayAndFilterCELVisibility` | Connector enrichment, durable ledger replay, and filter visibility | The test runner consumes Go test events and matches the exact package and test name. It rejects a required case that is missing, skipped, failed, or unfinished, even when the underlying Go command exits successfully. A passing parent suite does not stand in for a required child test. Each successful requirement produces a `REQUIRED PASS` line with the number of completed runs. For a focused investigation, the same runner accepts repeatable `--require-test` options before `--`. The package is the full import path and the required name is exact; the Go `-run` expression after `--` selects what to run. ```bash ./scripts/dev-cache.sh run -- ./scripts/run-hermetic-tests.sh \ go run ./tools/testwatch \ --require-test 'github.com/ductor-io/ductor/tests/e2e::TestE2ESuite/TestRouteEmitsDurableEvents' \ -- -race -count=1 -timeout=10m ./tests/e2e \ -run '^TestE2ESuite$/^TestRouteEmitsDurableEvents$' ``` Use the Make target for the complete critical set. If a case is renamed, update both the selection expression and the exact requirement in the same change. The requirement list is part of the release contract. ## Interpret durable-event and replay evidence [#interpret-durable-event-and-replay-evidence] The routing checks use PostgreSQL-backed event persistence and read the stored history. They wait for routing's asynchronous completion work before asserting that request and assignment events exist and belong to the expected tenant. Observing only an in-memory callback does not establish durable delivery. Replay checks load the recorded decision or routable, retrieve its event history, and run the simulation through an isolated routing graph. The simulation may write its own audit record, while the original decision and live routing event history remain unchanged. This verifies that flow's isolation; it does not establish that every historical dependency can be reconstructed at any timestamp. See [Events & Event Sourcing](/docs/concepts/events#durable-routing-history-and-replay). ## Complete operational acceptance [#complete-operational-acceptance] Keep source verification and deployment observations together in the release record: * The full source revision, clean build identity, artifact digest, commands, exit statuses, and test artifacts. * The target environment's configuration and explicit [migration result](/docs/operations/migrations). * [Readiness](/docs/operations/health-checks), [operational health](/docs/operations/operational-health), and applicable [canary results](/docs/connectors/synthetic-canaries) after rollout. * The [backup and restore](/docs/operations/backup-restore) evidence and [promotion or rollback procedure](/docs/operations/promotions-and-rollout) appropriate to the change. The local release suite proves the code paths and dependencies it exercised. Production credentials, external provider behavior, deployment topology, and recovery procedures require evidence from the intended environment. Keep failures and unsupported assumptions visible in the release decision until they are resolved and verified.