Testing
Sandswap is tested in three layers:
| Layer | Directory | Needs |
|---|---|---|
| 1. Unit tests | next to the sources (*_test.go, #[cfg(test)]) | nothing — no KVM, no root |
| 2. E2E integration | e2e/ | root + KVM for full-stack scripts; the etcd-layer scripts need only etcd |
| 3. Deploy verification | deploy/ | a deployed system |
The layers build on each other: unit suites gate every build, the e2e battery gates every change, and the deploy drills gate real rollouts.
Every script in layers 2 and 3 is built on one shared framework and is verified by one matrix runner — see Script framework. Both matrices — deployment fixtures included — are driven by one repository-root command, run-round.sh.
1. Unit tests
Unit tests live next to the sources and run per module:
-
Go —
*_test.gofiles throughoutgateway/,sandlot/,boxout/,dns/(the sandns sources),sandctl/, andcommon/. Run per module:cd gateway && go test ./... # likewise for sandlot/, boxout/, dns/, sandctl/, common/The resilience-relevant gateway subsets (lease, pool, handler) are also run in strict mode (with the race detector, parsed pass-count floors) by
e2e/resilience.sh— see the E2E catalog. -
warden (Rust) — inline
#[cfg(test)]modules in the warden sources; run withcargo testinsidewarden/. The CI suite is unprivileged-safe; the chroot/mount/seccomp tests are#[ignore]-gated and run only as root (cargo test -- --ignored). The warden build runs this suite first (WARDEN_SKIP_TESTS=1opts out for scratch builds), so a broken Rust suite fails the build;e2e/warden-unit.shruns the same suites as a suite-visible step in the e2e runner. -
sandvmm (Rust) — unit tests across the
vmm/crates (the cloud-hypervisor fork); run withcargo testin thevmm/workspace.
Unit tests deliberately stop at module boundaries — cross-process behavior (gateway → sandlot → sandvmm → warden), real KVM, real etcd, and failure injection are the e2e layer’s job.
2. E2E integration tests
Every script under e2e/ is self-contained: it cleans the environment, deploys its own binaries and configs, runs, and tears everything down — no script depends on another having run first, and any script can be run independently. Properties enforced via the shared e2e/common.sh helper (55 tc_* helpers):
- Strict exit-code semantics — exit 0 only if everything passed, 1 on any failure. Missing KVM, missing build artifacts, or a non-root invocation of a root script fail loudly — a broken environment shows up as FAILED, never as a silent skip that CI would read as green.
- Freshness gating — each script verifies artifact freshness (
tc_check_freshness): staleoutput/artifacts are rebuilt automatically; withAUTO_REBUILD=0the script aborts instead, so you can never accidentally test old binaries. The matrix runner establishes freshness ONCE for the whole pass before any verdict is recorded; rows then skip their per-row gate so no row can re-stamp the shared artifacts mid-round. - Serial only — full-stack scripts share ports (3001/3100/9998), the deploy directory, the vhost-vsock CID space, and the etcd ports; two concurrent runs are undefined.
run-matrix.shexists partly to make that rule executable. - Scoped process assertions — identify processes by recorded PID (
tc_start_*$!), by this run’s own VM records (tc_run_vm_pids, re-adopt-aware), or by subtree (tc_descendant_pids). Neverpgrep/psname matches: they cannot distinguish this run’s processes from a leaked previous-run process or an unrelated same-name service. - External dependencies skip loudly — no LLM key → the LLM scripts SKIP with a visible line; no docker daemon → the Keycloak script SKIPs. A silent green no-op would be a false pass.
The serial runner:
sudo bash run-round.sh # whole round: e2e then deploy (daemonized)
sudo bash e2e/run-matrix.sh # one tree: e2e (LLM rows need an API key)
sudo bash e2e/run-matrix.sh --fast # skip benchmarks and heavy rows
sudo bash e2e/run-matrix.sh --only e2e/smoke.sh
sudo bash e2e/run-matrix.sh --list
It runs each row in turn (bounded by a per-row timeout), sweeps leftover processes / nftables rules / TAP devices / etcd between them, records a verdict per script in a resumable checkpoint (reuse is keyed on the script’s closure and the build-artifact identity), prints a summary table, and exits non-zero on any FAIL or NOT-RUN row. Rows are classified in the runner’s SPEC; LLM/docker gating is a runner rule and --fast skips the bench rows. The drills that verify an operator-DEPLOYED stack instead of deploying their own are not suite scripts — they live in deploy/baremetal/verify-*.sh (layer 3 above). The battery rows all run under an isolated battery root (SANDSWAP_E2E_ROOT, default /var/lib/sandswap-e2e) — a row’s environment sweep can never touch a production deployment under /var/lib/sandswap.
Prerequisites for full-stack scripts: root (deploy directory, host networking, vsockguard, process cleanup), /dev/kvm and /dev/vhost-vsock (usually modprobe vhost_vsock), and a built output/ — see Building. Defaults can be overridden by setting variables before the script runs, e.g. POOL_SIZE=2 sudo bash e2e/smoke.sh.
The full per-script catalog — what each of the 64 suite scripts validates, how to run it, and its prerequisites — is the E2E catalog.
3. Deploy verification
Deploying and verifying on real hosts is its own section — see Deployment. The shape there mirrors this one: each deploy/ script has a matching verification step —
- Bare metal:
deploy/baremetal/deploy-*.shinstall the systemd stack per topology; the deployed-stack verifiers areverify-binpack.sh(DNS bin-pack concentration),verify-failover.sh(sandlot cross-gateway failover),verify-eino.sh(a real multi-agent workload),verify-cert-rotation.sh(whole-PKI rotation with fail-closed re-convergence),verify-etcd-fence.sh(the etcd-partition self-fence, live), andverify-negative-path.sh(each broken plane answers a named retryable refusal, never a silent success or a bare 500);sandswap-ops.sh(operations CLI) androtate-volume-seed.sh(seed-rotation chain) also operate a deployed stack, andsecurity/verify-host-security.shsweeps every host invariant (also the pre-deploy checklist in the Deployment section). - Kubernetes:
deploy/k8s/deploy-k8s-*.sh(embedded / boxout / egress-enabler modes) +verify-k8s.sh(auto-detects the deployed mode and runs the full drill). - Operational drills against a live deployment — certificate rotation, volume-seed rotation, rollback — live with their operations pages rather than here.
Those scripts verify deployment-specific concerns (systemd units, DNS bin-packing, k8s DaemonSets) that the self-contained e2e layer deliberately does not touch.
4. The full verification round
Both trees — the e2e battery and the deploy battery with its deployment fixtures — are driven by one command from the repository root:
sudo bash run-round.sh # e2e tree, then deploy tree (background)
sudo bash run-round.sh --fast # e2e benchmark rows skipped
The round deploys and verifies everything the batteries cover: each e2e row stands up its own stack, the deploy tree installs the real systemd topologies, runs the deployed-stack verifiers against them, and leaves the host clean whether a row passes or fails. It must run as root (KVM + host preparation).
Properties the round guarantees:
- Self-daemonizing — the driver re-execs itself under
nohup, so a closed terminal (or a timeout around the foreground shell) cannot kill the round mid-row.ROUND_FOREGROUND=1keeps it attached for interactive debugging. - Disk logs — one driver log plus one log per tree under
/var/tmp/sandswap-round/<UTC stamp>/(never/tmp: a tmpfs reboot must not clear it). The launcher prints the exact paths. - Resumable — the per-tree checkpoints persist under
/var/tmp/r3-matrix/. Rerunning the same command after an interruption or a reboot reuses every row whose closure and build-artifact identity are unchanged and re-executes FAIL and NOT-RUN rows; no row is ever silently re-greened. - One exit code — each tree’s matrix gate (any FAIL or NOT-RUN row) decides its status, and the round fails if either tree fails. The driver log ends with a triage summary: the FAIL rows with their reasons.
- Environment gates, not false greens — rows needing an LLM key or a docker daemon SKIP with the recorded reason when the dependency is absent. To exercise the real-agent rows, pass the key through sudo:
sudo env DEEPSEEK_API_KEY=sk-... bash run-round.sh(plainsudostrips the environment and the rows recordneeds-llm-keySKIPs).
Variables honored: ROUND_TREES (default e2e deploy), ROUND_FOREGROUND, ROUND_LOG_DIR, plus the tree runners’ SANDSWAP_E2E_ROOT and MATRIX_STATE_DIR. Row classification, gates, and per-script behavior: E2E catalog and Script framework.