Benchmark Report — Bare Metal
Bare-metal KVM on a TencentOS Server 4 cloud bare-metal host, measured with the prebuilt release package; raw logs are in
baremetal-raw/beside this report. All numbers are from real KVM microVMs (no mocks), read from the gateway’s own ns-precision metrics in every/agent/executeresponse.Companions: the nested-virtualization run (same benchmarks inside a GCP VM, ~3-4x higher numbers with the measured explanation). The
execfield is the warden-side mount+exec total on the cold path and spawn-to-wait on the warm path.
1. Executive summary
| Topology | Path | client P95 | gateway P95 | sandlot P95 | exec P95 |
|---|---|---|---|---|---|
| http single | Warm (sticky lease hit) | 0.997 | 0.880 | 0.758 | 0.663 |
| http single | Cold (mount_and_exec) | 2.128 | 1.993 | 1.884 | 1.696 |
| http distributed | Warm (sticky lease hit) | 1.045 | 0.891 | 0.761 | 0.668 |
| http distributed | Cold (mount_and_exec) | 2.324 | 2.194 | 2.066 | 1.782 |
| mcp single | Warm (sticky lease hit) | 1.311 | 0.947 | 0.820 | 0.733 |
| mcp single | Cold (mount_and_exec) | 2.328 | 2.093 | 1.969 | 1.743 |
| mcp distributed | Warm (sticky lease hit) | 1.341 | 0.958 | 0.821 | 0.731 |
| mcp distributed | Cold (mount_and_exec) | 2.494 | 2.086 | 1.935 | 1.740 |
All table values in ms — each cell is the P95, verbatim from the per-topology result blocks in §3 (http) and §4 (mcp); n=100 warm / n=100 cold samples per run; no re-computation or rounding (min / P50 / max live in the per-topology blocks and baremetal-raw/). gateway = the gateway E2E metric. The mcp rows exclude the LLM-driven tool-call measurements (§4, n=5, cold only). Every run in this report is CPU/NUMA pinned (§2): the host stack (gateway, sandlot, benchmark client) runs on eight whole physical cores (16-23 and their SMT siblings 64-71), and each microVM owns one whole physical core (13/61, 14/62, 15/63) — vCPU and vhost/virtiofs threads stay on that core, never spanning two physical cores.
- Warm gateway E2E P95 across topologies: 0.88 ms (http single), 0.89 ms (http distributed), 0.95 / 0.96 ms (mcp single/distributed) — all four topologies under 1 ms through the full gateway stack (gateway → sandlot → warden); sticky-lease reuse, no remount.
- Cold gateway E2E P95 across topologies: 1.99 ms (http single), 2.19 ms (http distributed), 2.09 / 2.09 ms (mcp single/distributed) — within ~0.2 ms of each other: etcd only records ownership / placement (zero writes per execution cycle in steady state); the exec path authenticates from the gateway’s local in-memory snapshot and never touches etcd, and the cold-mount lifecycle gate is served from the local watch snapshot (§3.3), so the distributed topology pays no measurable etcd toll on the normal active cold mount.
- Client round-trip figures are for reference only: in a real deployment the client and sandswap may not be in the same data center, so client-perceived latency varies with the actual network path; the gateway-side numbers are the portion attributable to sandswap itself.
- The pinned baseline keeps the warm gateway P95 in a 0.88–0.96 ms band across the four topologies; CPU/NUMA placement is fixed by the benchmark harness (§2), so the numbers reflect the sandbox path rather than the host scheduler.
2. Test environment
| Item | Value |
|---|---|
| Host | Inspur SA5212M5, systemd-detect-virt=none, no cpuinfo hypervisor flag |
| Chassis / BIOS | Inspur SA5212M5, BIOS 4.3.19 |
| CPU | Intel Xeon Platinum 8255C @ 2.50 GHz, 2 sockets, 96 threads (acpi-cpufreq, intel_pstate=disable, governor performance, intel_idle.max_cstate=1, iommu=pt) |
| RAM | 384 GiB (DIMM total; 375.8 GiB MemTotal), 2 NUMA nodes |
| NIC / disk | 2× Broadcom BCM58802 50GbE (bnxt_en); 2× Intel SSDPE2KX040T8 3.5 TB NVMe |
| Host kernel | 6.6.117-45.14.2.tl4.x86_64 (TencentOS Server 4) |
| Guest kernel | 7.2.8 minimized cloud-hypervisor tree (vmlinux ELF+PVH; CONFIG_IPV6=y, CONFIG_INET_DIAG_DESTROY=y, INIT_ON_ALLOC_DEFAULT_ON=y, runtime init_on_free=0), clocksource=kvm-clock, hrtimers ON |
| Binaries | prebuilt release package — reproducing.md Path 2, scripts bundled with the binaries; cloud-vendor monitoring agents (barad/stargate) stopped for the run |
| CPU / NUMA pinning | On (bare metal): host stack (gateway, sandlot, benchmark client) on eight whole physical cores — 16-23 and their SMT siblings 64-71; each microVM on one whole physical core (13/61, 14/62, 15/63 — both SMT threads) with numactl --membind=0; driven by e2e/pin.sh (PIN_MODE=auto; nested virt is not pinned, PIN_MODE=0 disables) |
| microVM | 1 vCPU, 256 MiB, 5 MiB initramfs, pool of 3 pre-warmed CLEAN VMs |
| Lease | sticky, 100 ms window |
| VM recycle | max_uses=100000 — no VM replacement occurs inside a benchmark window |
| Topologies | single-node (combined gateway) and distributed (etcd + exec gateway + human gateway) |
| Benchmarks | e2e/latency-benchmark-aiagent.sh http {single,distributed} and mcp {single,distributed}, e2e/latency-benchmark-unmount.sh all (script defaults WARM_N=100 / COLD_N=50 → n=100 warm / n=100 cold samples, BENCH_VERBOSE=1; per-sample detail in baremetal-raw/) |
| Warden logging | default info — every per-op hot-path line lives at debug (sandlot -warden-log debug to enable for profiling) |
3. HTTP Command-execution path results
All values in ms, from the gateway’s gateway_duration_ns / sandlot_duration_ns / exec_duration_ns plus the client-observed round trip; P50/P95 over n=100 (warm) and n=100 (cold). Quality gate green, zero 503s, leaseHit classification from X-Sandbox-Lease.
3.1 Single-node topology (http single)
Warm — sticky lease reuse (leaseHit=true, n=100) [ms]
metric min P50 P95 max
client round-trip 0.884 0.922 0.997 1.206
gateway E2E 0.805 0.825 0.880 1.009
sandlot 0.710 0.724 0.758 0.841
exec 0.624 0.637 0.663 0.714
Switch — cross-agent cold mount_and_exec (leaseHit=false, n=100) [ms]
metric min P50 P95 max
client round-trip 1.920 2.009 2.128 2.210
gateway E2E 1.821 1.900 1.993 2.100
sandlot 1.711 1.786 1.884 1.970
exec 1.513 1.602 1.696 1.770
3.2 Distributed topology (etcd + exec gateway + human gateway, http distributed)
Warm — sticky lease reuse (leaseHit=true, n=100) [ms]
metric min P50 P95 max
client round-trip 0.896 0.946 1.045 1.174
gateway E2E 0.814 0.848 0.891 0.999
sandlot 0.720 0.737 0.761 0.868
exec 0.633 0.647 0.668 0.718
Switch — cross-agent cold mount_and_exec (leaseHit=false, n=100) [ms]
metric min P50 P95 max
client round-trip 2.033 2.111 2.324 3.756
gateway E2E 1.926 1.985 2.194 3.626
sandlot 1.798 1.854 2.066 2.645
exec 1.606 1.661 1.782 2.376
(The cold maxima come from a couple of outlier samples — whole-request stalls spread across all layers; the P95 rows are unaffected.)
3.3 Layer decomposition
Single-node (P50 / P95):
| Segment | Warm | Cold | What it is |
|---|---|---|---|
| client − gateway | 0.09 / 0.13 | 0.11 / 0.13 | HTTP/client stack |
| gateway − sandlot | 0.10 / 0.14 | 0.11 / 0.14 | auth, lease, pool, gRPC |
| sandlot − exec | 0.09 / 0.10 | 0.19 / 0.21 | sandlot proc + set-fs + vsock (cold only) |
| exec (warden) | 0.64 / 0.66 | 1.60 / 1.69 | virtio-fs mount + fork+execve inside the guest |
Distributed (P50 / P95):
| Segment | Warm | Cold | What it is |
|---|---|---|---|
| client − gateway | 0.10 / 0.13 | 0.12 / 0.14 | HTTP/client stack |
| gateway − sandlot | 0.11 / 0.14 | 0.13 / 0.16 | auth, lease, pool, gRPC (+ cold lifecycle/ownership checks) |
| sandlot − exec | 0.09 / 0.10 | 0.20 / 0.22 | sandlot proc + set-fs + vsock (cold only) |
| exec (warden) | 0.65 / 0.67 | 1.66 / 1.78 | virtio-fs mount + fork+execve inside the guest |
(Segment values are the P50/P95 of the per-sample differences between adjacent layers; the exec row is the warden’s own exec_duration_ns.) The distributed topology’s warm exec path never touches etcd — auth resolves from the gateway’s local in-memory identity snapshot, and etcd only records ownership / placement — so warm is within noise of single-node. The cold path’s lifecycle gate is snapshot-first: recheckLifecycleGate reads the local watch snapshot and only a non-active snapshot (detaching/detached/deleting) performs the fresh quorum read (an attach may have completed while the request waited); a transition that lands after the check is handled by the record watcher’s drain. A normal active cold mount therefore performs only the local cache lookup — the cold gateway − sandlot segment is 0.13 / 0.16 ms (P50/P95), within noise of single-node (0.11 / 0.14) — and the ownership claim is a cache fast path (in-memory after the first mount; etcd only on a cold cache miss or a placement write). BeginDispatch/EndDispatch are in-memory; the warm path skips the lifecycle recheck entirely, which is why its segment is unchanged.
4. MCP Command-execution path results
Same stack plus the MCP endpoint (/mcp, Streamable HTTP), driven by openai-sandswap with DeepSeek (deepseek-flash). All values in ms, from the same ns-precision metrics as §3 plus the client-observed round trip; P50/P95 over n=100 (warm), n=100 (cold) and n=5 (LLM-driven). Quality gate green, zero 503s. The LLM’s own round trip (not on the tool path): min / P50 / P95 = 778 / 968 / 1058 ms (single) and 626 / 871 / 1182 ms (distributed) — three orders of magnitude above any sandbox segment.
4.1 Single-node topology (mcp single)
Warm — sticky lease reuse (n=100) [ms]
metric min P50 P95 max
client round-trip 1.063 1.130 1.311 1.435
gateway E2E 0.883 0.917 0.947 0.963
sandlot 0.787 0.802 0.820 0.829
exec 0.698 0.713 0.733 0.746
Switch — cross-agent cold mount_and_exec (n=100) [ms]
metric min P50 P95 max
client round-trip 2.035 2.204 2.328 3.040
gateway E2E 1.825 1.974 2.093 2.792
sandlot 1.700 1.848 1.969 2.649
exec 1.502 1.655 1.743 2.363
LLM-driven tool calls (cold, n=5) [ms]
metric min P50 P95 max
client round-trip 2.189 2.358 6.989 6.989
gateway E2E 1.945 2.085 6.539 6.539
sandlot 1.802 1.927 5.422 5.422
exec 1.605 1.717 2.462 2.462
4.2 Distributed topology (etcd + exec gateway + human gateway, mcp distributed)
Warm — sticky lease reuse (n=100) [ms]
metric min P50 P95 max
client round-trip 1.063 1.131 1.341 1.529
gateway E2E 0.889 0.917 0.958 1.075
sandlot 0.786 0.800 0.821 0.909
exec 0.697 0.710 0.731 0.737
Switch — cross-agent cold mount_and_exec (n=100) [ms]
metric min P50 P95 max
client round-trip 2.089 2.184 2.494 3.066
gateway E2E 1.866 1.945 2.086 2.814
sandlot 1.750 1.814 1.935 2.025
exec 1.563 1.621 1.740 1.770
LLM-driven tool calls (cold, n=5) [ms]
metric min P50 P95 max
client round-trip 2.330 2.393 7.561 7.561
gateway E2E 2.066 2.125 7.129 7.129
sandlot 1.914 1.959 5.491 5.491
exec 1.712 1.749 2.411 2.411
4.3 Layer decomposition
Single-node (P50 / P95):
| Segment | Warm | Cold | What it is |
|---|---|---|---|
| client − gateway | 0.21 / 0.36 | 0.23 / 0.31 | HTTP/client stack + Streamable-HTTP MCP endpoint |
| gateway − sandlot | 0.11 / 0.14 | 0.12 / 0.15 | auth, lease, pool, gRPC |
| sandlot − exec | 0.09 / 0.10 | 0.19 / 0.22 | sandlot proc + set-fs + vsock (cold only) |
| exec (warden) | 0.71 / 0.73 | 1.66 / 1.74 | virtio-fs mount + fork+execve inside the guest |
Distributed (P50 / P95):
| Segment | Warm | Cold | What it is |
|---|---|---|---|
| client − gateway | 0.21 / 0.40 | 0.24 / 0.41 | HTTP/client stack + Streamable-HTTP MCP endpoint |
| gateway − sandlot | 0.11 / 0.14 | 0.13 / 0.16 | auth, lease, pool, gRPC (+ cold lifecycle/ownership checks) |
| sandlot − exec | 0.09 / 0.10 | 0.19 / 0.21 | sandlot proc + set-fs + vsock (cold only) |
| exec (warden) | 0.71 / 0.73 | 1.62 / 1.74 | virtio-fs mount + fork+execve inside the guest |
(Segment values are the P50/P95 of the per-sample differences between adjacent layers. The LLM-driven rows (n=5) are not decomposed: at n=5 the P95 is the max, so a percentile difference has no statistical meaning.)
The client − gateway segment is larger than on the plain-HTTP path (0.21–0.24 ms P50 vs 0.09–0.12): it carries the Streamable-HTTP MCP endpoint on top of the agent API. gateway − sandlot and below are identical machinery to §3.
5. Unmount / recycle path
e2e/latency-benchmark-unmount.sh all (ITER_N=20 per scenario, Ubuntu rootfs with python3, data from /stats unmount_stats). Every iteration runs one exec workload, then lets the sticky lease expire so the real lease-expiry unmount fires — the warden’s OP_UNMOUNT (cgroup.kill + rule-A process sweep + teardown + umount(2) + drop_caches + the cross-tenant residue sweep + the final-state gate) — and the three scenarios stack one cost layer at a time:
- baseline —
echo ok(shell): the fixed-cost floor of the unmount path itself (teardown, umounts, cache drop over an essentially clean state). - cache16m — python3 writes 16 MB to
/workspace, fsyncs, and reads it back: leaves a clean 16 MB page cache. Goal: measure howdrop_cachesscales with page-cache size, independent of writeback. - dirty16m — python3 writes 16 MB with no fsync: pages stay dirty past exec return (dirty_background ≈ 25 MB at 256 MB VM RAM), so
umount(2)itself pays the writeback over virtio-fs. Goal: measure writeback cost on top of the cache drop.
| Scenario | n | success | fail | <5 ms | <10 ms |
|---|---|---|---|---|---|
| baseline | 20 | 20 | 0 | 100% | 100% |
| cache16m | 20 | 20 | 0 | 100% | 100% |
| dirty16m | 20 | 20 | 0 | 100% | 100% |
5.1 Per-layer breakdown (slowest samples from /stats unmount_stats, ms)
The gateway records every unmount at ns precision in three layers: gateway_ms (RPC E2E) / sandlot_ms (handler total, incl. set-fs clear) / warden_ms (kill + teardown + umount + drop_caches + the cross-tenant residue sweep and final-state gate). The table below shows the per-layer range of each scenario’s slowest recorded samples (the top-slow list keeps the 10 slowest per node; dirty16m lands one entry in it — its samples cluster just under the cache16m maxima):
| Scenario | gateway_ms | sandlot_ms | warden_ms | slowest-sample trigger |
|---|---|---|---|---|
| baseline | 2.291–2.423 | 2.059–2.170 | 1.856–1.944 | lease-expiry |
| cache16m | 3.036–3.106 | 2.773–2.841 | 2.545–2.610 | lease-expiry |
| dirty16m | 3.070 | 2.829 | 2.624 | lease-expiry |
Run-wide rolling-window percentiles (n=63 recorded unmounts): gateway P50 / P95 / P99 / max = 2.945 / 3.070 / 3.106 / 3.106 ms; warden P50 / P95 / P99 / max = 2.480 / 2.590 / 2.624 / 2.624 ms.
Reading: every observed unmount is single-digit ms and 100% complete under 5 ms. The warden-side cost is the fixed teardown/sweep sequence — sync execs are not in the agent cgroup (kill_and_reap returns immediately), the rule-A sweep finds no survivors, the child mounts and workspace umount detach cleanly, and the cross-tenant sweep plus final-state gate run on an empty residue set. The clean 16 MiB workload raises the warden maximum by ~0.7 ms (1.944 → 2.610 ms) because drop_caches scales with the pinned page cache; the dirty 16 MiB workload’s slowest sample is 2.624 ms — the writeback is absorbed before the sweep.
6. Reproduction
Developer path (clone → build → run) below; the no-toolchain path — release tarball with the scripts bundled — is in Reproducing the Benchmarks. The numbers in this report were measured through that release path (scripts and binaries bundled together) on a host with the cloud-vendor monitoring agents stopped.
CPU/NUMA pinning is used on bare metal (e2e/pin.sh): the host stack (gateway, sandlot, benchmark client) is pinned to eight whole physical cores (16-23 and their SMT siblings 64-71), and every microVM to its own whole physical core (13/61, 14/62, 15/63), with numactl --membind. A 1-vCPU VM uses one SMT thread; its vhost/virtiofs threads use the other, so the VM never spans two physical cores. Nested-virt hosts are not pinned. Set PIN_MODE=0 to disable pinning, or PIN_STACK_CORES / PIN_SLICE_CPUS to change the partition.
git clone <repo-url> # clone this repository
cd sandswap
bash build/install-deps.sh # one-shot dependency installer (apt + dnf/yum)
bash build/build.sh all # freshness-gated build
BENCH_VERBOSE=1 sudo bash e2e/latency-benchmark-aiagent.sh http single
BENCH_VERBOSE=1 sudo bash e2e/latency-benchmark-aiagent.sh http distributed
# mcp mode (also times the LLM round trip — needs DEEPSEEK_API_KEY /
# OPENAI_API_KEY / GOOGLE_API_KEY):
DEEPSEEK_API_KEY=sk-... sudo bash e2e/latency-benchmark-aiagent.sh mcp single
DEEPSEEK_API_KEY=sk-... sudo bash e2e/latency-benchmark-aiagent.sh mcp distributed
sudo bash e2e/latency-benchmark-unmount.sh all
# single scenario instead of all: baseline | cache16m | dirty16m
sudo bash e2e/latency-benchmark-unmount.sh cache16m
# cache-size curve: workload file size in MB (default 16)
FILE_MB=128 sudo bash e2e/latency-benchmark-unmount.sh cache16m
# per-sample detail (kept in baremetal-raw/ for this run):
BENCH_VERBOSE=1 sudo bash e2e/latency-benchmark-aiagent.sh http single
# warden-side per-op timing lines live at debug level (guest console cost —
# they tax the measured path at info):
SANDLOT_EXTRA_FLAGS="-warden-log debug" sudo bash e2e/latency-benchmark-aiagent.sh http single
Every script is self-cleaning and asserts its own quality gate (≥90% of requested samples must succeed with non-zero durations, or the run fails). The unmount benchmark additionally needs an Ubuntu rootfs with python3 (output/rootfs-ubuntu): it is auto-built on first run (~80 MB download) and kept for reuse across runs — the release package ships it prebuilt.
Appendix: raw data
The complete stdout/stderr of every run (including the per-sample BENCH_VERBOSE=1 detail) is kept in baremetal-raw/ beside this report. The exact commands and SHA256 are in baremetal-raw/MANIFEST.md.