Reproducing the Benchmarks
The numbers in the four reports — bare-metal latency, nested-virt latency, bare-metal microVM reuse capacity and nested-virt microVM reuse capacity — come from self-contained scripts in e2e/. Each one deploys its own throwaway stack (binaries, certs, volumes, logs — everything under $DEPLOY_DIR, default /var/lib/sandswap), runs, and removes all of it on exit, so no prior deployment is needed.
There are two ways to get a runnable setup:
| Path 1 — developer | Path 2 — prebuilt release | |
|---|---|---|
| Audience | contributors with a full toolchain | anyone with a KVM host |
| Source of the binaries | built locally by build.sh all | shipped in the release tarball |
| Toolchain needed | Go, Rust, kernel build deps (~30 min first build) | none |
| Scripts | e2e/ in the clone | bundled in the release tarball (same tag as the binaries) |
Both paths run the same scripts against the same on-disk layout — a directory holding output/ — so the commands after setup are identical.
1. Prerequisites (both paths)
- x86_64 bare-metal host (nested-virt works but is ~3× slower — see the nested-virt report for what to expect)
- KVM:
/dev/kvmand/dev/vhost-vsockreadable (modprobe vhost_vsockif the latter is missing), running as root curl,python3,openssl(mTLS cert generation)iproute2(bridge/TAP setup),util-linux(setpriv),kmod(modprobe);nftablesfor guest NAT and forward isolation — installed automatically by the scripts via the system package manager (apt/dnf/yum/ zypper/pacman) when missing; without it the run is refused (fail-closed)- Only for the
distributedtopology: etcd is vendored in the release and downloaded automatically in a dev tree — nothing to install - Only for
mcpmode: an LLM API key (DEEPSEEK_API_KEY/OPENAI_API_KEY/GOOGLE_API_KEY). The defaulthttpmode needs none.
2. Path 1 — developer: clone, build, benchmark
git clone <repo-url> # clone this repository
cd sandswap
bash build/install-deps.sh # one-shot dependency installer (apt + dnf/yum)
bash build/build.sh all # Go + Rust + kernel + initramfs (~30 min first build)
The build ends with a freshness check; every e2e script re-runs it (tc_check_freshness) and auto-rebuilds stale artifacts, so the binaries you benchmark always match the sources you cloned.
Then run the benchmarks — see 4 below.
3. Path 2 — use the release package
Releases ship a repo-shaped tree with prebuilt static binaries, the guest kernel, initramfs, templates, vendored etcd — and the benchmark scripts bundled at the same tag as the binaries, so there is nothing else to download and no version-skew risk between scripts and binaries.
Resolve the latest tag from the releases/latest redirect, then curl the assets:
TAG=$(curl -fsSL -o /dev/null -w '%{url_effective}' \
https://github.com/ChaoyiHuang/sandswap/releases/latest | awk -F/ '{print $NF}')
curl -fLO "https://github.com/ChaoyiHuang/sandswap/releases/download/${TAG}/SHA256SUMS"
curl -fLO "https://github.com/ChaoyiHuang/sandswap/releases/download/${TAG}/sandswap-${TAG}-linux-amd64.tar.gz"
sha256sum -c --ignore-missing SHA256SUMS # verify the tarball
tar xzf "sandswap-${TAG}-linux-amd64.tar.gz" && cd "sandswap-${TAG}-linux-amd64"
Or pin an explicit version instead of following latest:
TAG=<release-tag>
curl -fLO "https://github.com/ChaoyiHuang/sandswap/releases/download/${TAG}/sandswap-${TAG}-linux-amd64.tar.gz"
Extracting gives you output/ (prebuilt artifacts) and e2e/ (the benchmark scripts) side by side. Scripts inside a package detect the .release-stamp marker and skip the source-freshness gate — there are no sources to compare against.
Note: match the tag — a release tarball is self-consistent (scripts and binaries from the same tag); prefer it over mixing scripts from main with release binaries, as both can drift between tags.
4. Latency benchmarks
Each script takes measurements from production surfaces (the gateway’s own per-stage metrics, the X-Sandbox-Lease header, /stats unmount_stats) — no log scraping, no extra instrumentation.
# AI agent latency (http mode, single-node topology — the bare-metal report's
# main tables): 1 warmup + 100 warm execs + 100 cold mounts (50 rounds)
sudo bash e2e/latency-benchmark-aiagent.sh http single
# distributed topology (etcd + exec/human gateways)
sudo bash e2e/latency-benchmark-aiagent.sh http distributed
# mcp mode — additionally times the LLM and MCP tool round trips
DEEPSEEK_API_KEY=sk-... sudo bash e2e/latency-benchmark-aiagent.sh mcp single
DEEPSEEK_API_KEY=sk-... sudo bash e2e/latency-benchmark-aiagent.sh mcp distributed
# per-sample detail (the report appendix tables)
BENCH_VERBOSE=1 sudo bash e2e/latency-benchmark-aiagent.sh http single
# Unmount full-path latency (baseline + 16 MiB cache + 16 MiB dirty workloads)
sudo bash e2e/latency-benchmark-unmount.sh all
# single scenario, or a cache-size curve via the workload file size
sudo bash e2e/latency-benchmark-unmount.sh cache16m
FILE_MB=128 sudo bash e2e/latency-benchmark-unmount.sh cache16m
5. Load benchmarks (microVM reuse capacity)
Two steps: first measure the baseline (one microVM, no background load), then run the offered-agent levels with its four P95s.
# step 1: the baseline (one microVM, no background load): gateway E2E and
# client P95, warm and cold
sudo bash e2e/latency-benchmark-aiagent.sh http distributed
# step 2: the realistic AI agent model. Offered-agent levels, one flat volume
# per simulated agent; the benchmark reserves whole cores for the host stack
# (auto-sized, or named per role with PIN_STACK_LAYOUT) and gives the rest to
# the microVM pool. Pass the baseline's four P95s so the latency tiers are
# measured.
PIN_MODE=1 PIN_RESERVE_CORES=auto PIN_SLICE_CPUS=2 POOL_SIZE=<remaining cores> \
LOADTEST_AGENT_LEVELS="<agent levels>" LOADTEST_LATENCY_TIERS="1.0 2.0" \
SKIP_ESTIMATE_BENCH=1 \
WARM_GW_P95_MS=<warm-gw-p95> COLD_GW_P95_MS=<cold-gw-p95> \
WARM_CLIENT_P95_MS=<warm-client-p95> COLD_CLIENT_P95_MS=<cold-client-p95> \
sudo -E bash e2e/latency-benchmark-load.sh
# optional capability tests:
# ideal — ideal paced-lane test: one lane per microVM, no between-turn think
# gap (the collision-free ceiling)
# shaped — traffic-shaped test: a full agent population queued per lane, one
# in-flight request per VM (the realistic version of that ceiling)
POOL_SIZE=2 LOADTEST_WORKLOAD=ideal LOADTEST_LANES=2 \
LOADTEST_TURN_PERIODS_MS="14 16 18 20 22 24" \
SKIP_ESTIMATE_BENCH=1 \
WARM_GW_P95_MS=<...> COLD_GW_P95_MS=<...> \
WARM_CLIENT_P95_MS=<...> COLD_CLIENT_P95_MS=<...> \
sudo -E bash e2e/latency-benchmark-load.sh
POOL_SIZE=2 LOADTEST_WORKLOAD=shaped LOADTEST_LANES=2 \
LOADTEST_TURN_PERIODS_MS="14 16 18 20 24 28" LOADTEST_SHAPED_HOLD_MS=auto \
SKIP_ESTIMATE_BENCH=1 \
WARM_GW_P95_MS=<...> COLD_GW_P95_MS=<...> \
WARM_CLIENT_P95_MS=<...> COLD_CLIENT_P95_MS=<...> \
sudo -E bash e2e/latency-benchmark-load.sh
The two microVM reuse capacity reports ran this same script on their own hosts; the exact command and environment behind every table are in each report’s Reproduction section and its *-load-raw/MANIFEST.md:
| bare-metal | nested-virt | |
|---|---|---|
| Pinning | PIN_MODE=1: the host stack roles on 12 reserved whole physical cores named by PIN_STACK_LAYOUT, the microVM pool on the remaining 36 | PIN_MODE=0 — the L1 vCPUs are already outer-scheduled |
| Pool / mapping | 32 VMs with PIN_SLICE_CPUS=2 (the physical-core mapping, the recommendation) and 64 with PIN_SLICE_CPUS=1 (the logical-core mapping, two VMs per core, comparison only) | 2 VMs |
| Latency tiers | 1.05 1.5 2.0 | 1.0 2.0 (the default) |
| Window | 20 s measured / 10 s warm-up; the first offered level is excluded | 20 s / 5 s |
Bare-metal runs auto-pin (e2e/pin.sh; PIN_MODE=auto resolves to 1 on bare metal, 0 under a hypervisor, PIN_MODE=0 forces it off): the host stack on whole reserved physical cores and every microVM on its own whole physical core — a 1-vCPU VM uses one thread and its vhost/virtiofs threads use the other, so no VM spans two physical cores — with numactl --membind per slice. PIN_PLAN=1 prints the computed core partition (reserve, layout, pool, slices) and exits without deploying, for sizing your own host.
Useful environment knobs (all optional):
| Knob | Default | Meaning |
|---|---|---|
WARM_N / COLD_N | 100 / 50 | warm samples / cold rounds (each round = 2 cold samples, so 100 cold) |
SWITCH_GAP_MS | 300 | gap between agent switches (must exceed the sticky window) |
POOL_SIZE | 4 | sandlot VM pool size |
VM_MEMORY_MB | 256 | microVM memory |
PIN_MODE | auto | CPU/NUMA pinning: bare metal on, hypervisor off; 0/1 force |
PIN_STACK_CORES | 8 | whole cores reserved for the host stack (aiagent/unmount) |
PIN_SLICE_CPUS | 2 | CPUs per microVM slice (the load benchmark derives it from BENCH_VCPUS); 2 = one whole physical core (both SMT threads), 1 = one logical core (two VMs per core, comparison only) |
PIN_RESERVE_CORES | 0 | load benchmark: whole cores reserved for loadgen/gateway/sandlot (auto = compute) |
PIN_STACK_LAYOUT | — | load benchmark: explicit role:cpus sets for the reserved cores, e.g. "loadgen:0-3;gateway:4-7;sandlot:8-11"; gateway also carries the human gateway + etcd, sandlot the provision instance |
PIN_VERIFY_EACH_LEVEL | 1 | load benchmark: re-verify the VM-to-slice map after every level; the report runs set 0 |
PIN_PLAN | 0 | print the computed core partition (reserve, layout, pool, slices) and exit without deploying |
LOADTEST_AGENT_LEVELS | 10 20 40 80 | load: offered agent counts (one flat volume per agent); capacity runs set the report’s larger lists |
LOADTEST_STICKY_MS | 1 | load: gateway sticky window; the follow-ups are sent immediately and must land inside it |
LOADTEST_THINK_MIN_MS / LOADTEST_THINK_MAX_MS | 900 / 1500 | per-turn LLM think time |
LOADTEST_REQUESTS_PER_TURN | 1:40,2:35,3:25 | requests per turn: N for a fixed count (1 = every turn is one cold request), or size:weight pairs |
LOADTEST_BUSY_RETRIES / LOADTEST_BUSY_RETRY_BACKOFF_MS | 50 / 10 | additional attempts after 503 NO_AVAILABLE_SANDBOX and the full-jitter backoff base (0 = surface the refusal) |
LOADTEST_INTRA_TURN_GAP_MS | 0 | spacing between requests inside a turn (0 = immediate, warm under the 1 ms sticky) |
LOADTEST_MAX_INFLIGHT | 1024 | client-side in-flight request cap (not a server limit) |
LOADTEST_LATENCY_TIERS | 1.0 2.0 | load: latency budgets as multiples of the no-load baseline P95, for BOTH classes (warm and cold) and both surfaces (gateway E2E and client); the per-report tiers are in the contrast above |
LOADTEST_WORKLOAD | agents | agents = the realistic AI agent model (closed loop); ideal = the ideal paced-lane test, one lane per microVM (capability test); shaped = the traffic-shaped test, a full agent population queued per lane, one in-flight request per VM |
LOADTEST_LANES / LOADTEST_TURN_PERIODS_MS | POOL_SIZE / 20 22 24 26 28 30 35 | ideal/shaped: lane count (must equal the pool) and the turn-period sweep. The knee is the highest completed cold rate among periods that pass the test’s gate — ideal: zero failed requests and busy refusals within the 0.5 % noise tolerance; shaped: zero failed requests and the cold (for mixed turns also warm) gateway and client P95 inside the last tier |
LOADTEST_SHAPED_AGENTS_PER_LANE / LOADTEST_SHAPED_HOLD_MS | auto / auto | shaped: agents aggregated per lane (auto = rate x mean think + margin) and the VM-slot hold after the sticky window — auto measures this stack’s unmount P95 (cold execs, then the /stats unmount recent P95) and uses ceil(P95)+2ms clamped [1,100]; bare metal calibrates several ms lower than nested |
LOADTEST_RECYCLE_POLICY | capacity | capacity disables all lifetime recycle caps + memory recycle; shipped uses the product defaults (300 distinct agents / 10000 mounts / 100000 uses) |
LOADTEST_MAX_USES / LOADTEST_MAX_MOUNTS / LOADTEST_MAX_DISTINCT_AGENTS | policy | explicit lifetime-cap overrides |
LOADTEST_DURATION_SEC / LOADTEST_WARMUP_SEC | 30 / 10 | measurement window / excluded warm-up |
SKIP_ESTIMATE_BENCH / WARM_GW_P95_MS / COLD_GW_P95_MS / WARM_CLIENT_P95_MS / COLD_CLIENT_P95_MS | 0 / — | skip the baseline run and supply its four no-load P95 values |
LOADTEST_AGENT_CREATE_CONCURRENCY / LOADTEST_PROV_MAX_OPS | 32 / 16 | parallel agent creation / prov-instance concurrency |
SANDLOT_EXTRA_FLAGS | — | extra sandlot flags, e.g. -warden-log debug for warden-side timing |
KEEP_LOGS | 0 | keep the component and guest logs in $LOG_DIR after the self-cleaning exit (post-mortem) |
Further details per script: header comments in e2e/, plus the Reproduction section of each report for the exact command set behind each table.
6. Reading your own numbers
- The latency scripts print warm/cold P50/P95 per mode/topology, and the unmount script prints per-scenario gateway/sandlot/warden layer breakdowns.
- The load benchmark prints, per offered-agent level, the completed cold reuses/s, the per-microVM rate, the warm share, the retry pressure and the gateway E2E / client P95 of both classes; the capacity section selects the highest level with zero failed requests, within each latency tier. A
503 NO_AVAILABLE_SANDBOXis retried like a real agent, so it appears as retry pressure, not a failed turn. LOADTEST_WORKLOAD=idealis the ideal paced-lane test, not a business model: one lane per microVM, each issuing a turn every period with no between-turn think gap. It sweeps the turn period and reports the highest cold rate with zero failed requests and busy refusals within the 0.5 % noise tolerance — the collision-free ceiling with the VMs continuously fed.LOADTEST_WORKLOAD=shapedis the realistic version of that ceiling: every lane aggregates a full AI agent population (own sandfs, own think time) whose turns queue at the lane; the lane serves one request at a time and holds the VM slot for sticky +LOADTEST_SHAPED_HOLD_MS(the unmount budget) between turns. Busy retries are part of the shaper’s flow control; its knee is zero failed requests with the cold (for mixed turns also warm) gateway and client P95 inside the last latency tier. Compare all three — theagentsno-failure maximum,shaped,ideal— to see how much of the ideal reuse capability each model recovers.- The first offered level of every realistic run is a warm-up and is excluded from the tables: on bare metal the host kernel is still writing back the per-agent volume creations (multi-GB at thousands of agents), which stalls guest I/O (see the bare-metal capacity report).
- On a nested-virt host the aiagent script labels its output
[NESTED-VIRT]; treat such runs as comparative only. - Numbers are only comparable within one tag and one hardware class — cross-tag or cross-host comparisons should be re-measured with the scripts from the newer tag.