Quick Start
This is a step-by-step command guide for running Sandswap on a single dev machine. Copy-paste each command in order. Two ways to get the software are covered — a self-contained release package (no toolchain) and a source build (developers) — then two deployment modes, plus an operator-extras section (deployment files / client flags):
- Release package: unpack and run — every binary, the guest kernel, and etcd are prebuilt.
- Source build: build everything from the checkout (
build.sh all). - Mode 1 — Single-node mode: one gateway (in-process store, no external persistence) + one sandlot. The simplest way to try the API.
- Mode 2 — Distributed mode: exec gateway + etcd + sandlot, with a human gateway + prov sandlot for CRUD. Exercises the full distributed path (etcd-backed ownership, TLS/mTLS).
Deployment is always done by deploy/baremetal/* (systemd). sandctl is a client: after a deploy it drives the API, DNS and pool/health views.
1. Getting the software
The bare-metal deployers (and every component they start) expect artifacts under output/. Both paths below produce that layout; the commands in the rest of this page are identical for both.
1.1 Using a release package
Download the release tarball, verify it against the release’s SHA256SUMS (sha256sum -c SHA256SUMS — the deploy tree trusts the checksums plus the output/.release-stamp instead of re-deriving the kernel identity), and unpack it into a directory the Sandswap service identities can traverse — /opt/sandswap is the recommended location:
sudo mkdir -p /opt/sandswap
sudo tar xzf sandswap-<ver>-linux-amd64.tar.gz -C /opt/sandswap --strip-components=1
cd /opt/sandswap
1.2 Source builds: get the source
git clone https://github.com/ChaoyiHuang/sandswap.git
cd sandswap
Every command below uses repo-relative paths (build/…, output/…), so run them from the repository root.
1.3 Source builds: one-time host dependencies
Release package users skip this section — the package already carries every binary and the vendored etcd.
sudo bash build/install-deps.sh
# idempotent: gcc/make/go/clang/libbpf-dev/bpftool/busybox/cpio/
# libseccomp-dev/libcap-ng-dev/curl/wget/git + rustup musl target + etcd v3.6.13
# + the e2e/deploy test tools (conntrack, numactl, nfs-kernel-server,
# nfs-common, openssl, iptables, util-linux).
# etcd is auto-downloaded if missing (checks `etcd` on PATH; otherwise downloads
# into a fresh mktemp dir and verifies the built-in SHA-256 pin before installing
# to /usr/local/bin — there is no /tmp cache trust). Override via ETCD_VERSION=v3.x.y.
# Go >= 1.26 is a separate install: https://go.dev/dl/.
1.4 Source builds: build everything
sudo bash build/build.sh all
The Ubuntu rootfs template layer is a chroot package install, so the full build runs as root. An unprivileged bash build/build.sh all keeps an existing complete template and fails with this command when none exists.
Builds in dependency order (kernel → warden → initramfs → vmm → go → rootfs-template → vsockguard) then runs a freshness check, producing output/{vmlinux.bin,initramfs.cpio.gz,sandvmm,warden,gateway,sandlot,boxout,sandns,sandctl} among others. The full artifact table and per-target details are in Building.
1.5 Source builds: verify
bash build/build.sh check # fails if any output/ is missing or stale
ls -lh output/{vmlinux.bin,initramfs.cpio.gz,sandvmm,warden,gateway,sandlot,sandns,sandctl}
which etcd etcdctl # source builds only; release packages use output/etcd/
1.6 The deployed layout
The bare-metal deployers install a systemd stack; nothing is generated into the checkout. The paths every mode shares:
/etc/sandswap/
gateway.yaml, gateway-exec.yaml, gateway-human.yaml # gateway configs
sandlot.yaml, sandlot-prov.yaml # node configs
certs/ # TLS + stream mTLS pairs
identity.env # RESOLVED service ids
sandctl.env # public client contract
/var/lib/sandswap/
volumes/ templates/ flat-templates/ state/ # agent + template storage
/usr/local/bin/sandswap-{gateway,sandlot,sandvmm} # service binaries
/usr/local/lib/sandswap/guest/ # guest TCB (root-only)
sandctl reads /etc/sandswap/sandctl.env automatically (public URLs + CA; never a private key). Its only local state is the agent-key cache run/agents.json under --run-dir (default ./run, 0600, owned by the invoking user).
2. Mode 1 — Single-node mode
The simplest mode: one gateway (in-process MemoryStore, no external persistence) + one sandlot, installed as systemd services. The node stream always uses per-node mTLS: the gateway binds the hello’s node_id to the client certificate CN and refuses a certificate-less stream. No etcd, no DNS needed — the gateway holds all state in memory and the sandlot connects to it directly. HTTP listeners use TLS by default.
2.1 Deploy gateway + sandlot
sudo bash deploy/baremetal/deploy-single-node.sh --skip-build
The deployer prepares the host (service identities, udev, sysctls), mints the certificates, seeds the rootfs:v1 template, installs and enables sandswap-gateway.service + sandswap-sandlot.service, waits for the pool (CLEAN >= 3), and only then writes /etc/sandswap/sandctl.env. --no-tls makes the HTTP listeners plaintext (stream mTLS stays on); --pool SPEC changes the warm pool.
2.2 Check status
systemctl status sandswap-gateway sandswap-sandlot --no-pager
journalctl -u sandswap-sandlot -n 50 --no-pager
deploy/baremetal/sandswap-ops.sh status # cluster/health view
output/sandctl health-all # probes /health + /ready
2.3 Create an agent
output/sandctl api agent create --name demo
Returns agent_id + api_key. The key is cached in run/agents.json, so subsequent --agent <id> calls resolve it automatically — no need to copy it.
2.4 Create a SandFS
output/sandctl api sandfs create --agent <agent_id> --name s1 \
--template rootfs:v1 --mode flat --vcpus 2 --memory-mb 512
rootfs:v1 is the busybox template included with build.sh all — the deployer seeds it to /var/lib/sandswap/flat-templates/rootfs/v1/. --vcpus / --memory-mb are concrete resource requirements. Note the returned sandfs_id — every execution call addresses its target sandbox with it explicitly.
2.5 Execute code
output/sandctl api execute --agent <agent_id> --sandfs <sandfs_id> \
--code 'echo hi' --language shell
Returns stdout, exit_code, and per-stage timings.
2.6 File I/O
output/sandctl api file --agent <agent_id> --sandfs <sandfs_id> \
--action write --path hello.txt --data 'hi'
output/sandctl api file --agent <agent_id> --sandfs <sandfs_id> \
--action read --path hello.txt
output/sandctl api file --agent <agent_id> --sandfs <sandfs_id> \
--action list_dir --path / --depth 1
2.7 Background exec
# Start a background task (returns exec_id immediately)
output/sandctl api execute --agent <agent_id> --sandfs <sandfs_id> \
--code 'sleep 30' --background --lease 60
# Poll status
output/sandctl api exec-status <exec_id> --agent <agent_id> --sandfs <sandfs_id>
# Kill it
output/sandctl api exec-kill <exec_id> --agent <agent_id> --sandfs <sandfs_id>
2.8 Pool & health
output/sandctl pool # cluster pool view (single-node: local /stats)
output/sandctl health-all # probe gateway + human /health + /ready
deploy/baremetal/sandswap-ops.sh logs gateway 50 # journalctl tail
2.9 List / delete
output/sandctl api sandfs list
output/sandctl api agent list
output/sandctl api template list
output/sandctl api sandfs delete <sandfs_id>
output/sandctl api agent delete <agent_id>
2.10 Clean up
sudo bash deploy/baremetal/uninstall.sh
Stops and disables the units, detaches vsockguard, and removes the host mutations. The default removes /var/lib/sandswap (all agent data); --keep-data preserves the agent volumes.
3. Mode 2 — Distributed mode
Distributed mode: the gateway uses EtcdStore (persists to etcd) and the whole chain is TLS/mTLS. One exec gateway serves /agent/*; one human gateway serves the control plane (/human/*, SandFS/template CRUD, ownership). This guide runs both on one machine with separate ports, sharing one etcd.
3.1 Deploy etcd + exec gateway + sandlot (+ human gateway + prov sandlot)
sudo bash deploy/baremetal/deploy-one-machine-etcd.sh --skip-build
The deployer installs and enables sandswap-etcd.service, sandswap-gateway@exec.service (:3000 agent, :9998 stream), sandswap-gateway@human.service (:3100 human, :9996 prov stream), sandswap-sandlot-instance@main.service (exec pool) and sandswap-sandlot-prov.service (storage plane). It gates on the pool and on the human gateway reporting provisioners >= 1, then writes /etc/sandswap/sandctl.env with the exec/human URLs and the etcd endpoint. --no-tls makes HTTP and etcd plaintext (stream mTLS stays on).
3.2 Check status
systemctl status 'sandswap-gateway@exec' 'sandswap-gateway@human' \
sandswap-etcd sandswap-sandlot-instance@main sandswap-sandlot-prov --no-pager
deploy/baremetal/sandswap-ops.sh status
output/sandctl health-all
sandswap-ops.sh status shows both gateways and the pool; the human gateway reports the connected prov instances:
curl --cacert /etc/sandswap/certs/ca.crt https://127.0.0.1:3100/health
# → {"provisioners":1,"status":"ok"}
The shipped human listener is anonymous on loopback (the unit passes -allow-anonymous-human); a production human gateway must use OIDC or mTLS instead.
3.3 The human gateway and prov sandlot
sandswap-gateway@human— the human gateway:mode: human, serving the human API on:3100(SandFS/template CRUD, ownership ops), with a prov-only stream on:9996.sandswap-sandlot-prov— the prov sandlot: a sandlot instance running the provision role (node idprov-1, its own per-node certificate) that dials:9996and executes every sandfs/template storage op. It has no VM pool.
3.4 Create agent + SandFS (via the human gateway)
output/sandctl api agent create --name dist-demo
output/sandctl api sandfs create --agent <agent_id> \
--name s1 --template rootfs:v1 --mode flat --vcpus 2 --memory-mb 512
sandctl reads human_url from the client contract, so CRUD commands hit the human gateway automatically. Creating the SandFS writes its ownership to etcd. Note the returned sandfs_id.
3.5 Execute code (via the exec gateway, owner-routed)
output/sandctl api execute --agent <agent_id> --sandfs <sandfs_id> \
--code 'echo distributed-ok'
The exec gateway (:3000) receives the request, looks up ownership in etcd, and routes to the sandlot holding the SandFS. The agent key is resolved from run/agents.json (created in 3.4), so no --api-key is needed.
3.6 File I/O
File I/O goes through the exec gateway (/agent/file on :3000), same as execute. --sandfs is required — requests address their sandbox explicitly:
output/sandctl api file --agent <agent_id> --sandfs <sandfs_id> \
--action write --path d.txt --data dist
output/sandctl api file --agent <agent_id> --sandfs <sandfs_id> \
--action read --path d.txt
3.7 Background exec
output/sandctl api execute --agent <agent_id> --sandfs <sandfs_id> \
--code 'sleep 20' --background --lease 60
output/sandctl api exec-status <exec_id> --agent <agent_id> --sandfs <sandfs_id>
output/sandctl api exec-kill <exec_id> --agent <agent_id> --sandfs <sandfs_id>
3.8 Pool (multi-gateway)
output/sandctl pool # both gateways and their nodes
output/sandctl pool --raw
pool tries etcd first. The deploy’s etcd client key belongs to the sandserv identity, so an ordinary user sees the /stats fallback (the same aggregate view, served anonymously on the human listener). Run it as root if you want the direct-etcd path.
You can also curl the human gateway’s /stats:
curl --cacert /etc/sandswap/certs/ca.crt https://127.0.0.1:3100/stats | python3 -m json.tool
The exec gateway (:3000) gates /stats and /metrics behind -ops-token-file — disabled (404) by default; see gateway configuration.
3.9 Other commands
output/sandctl health-all
output/sandctl api sandfs list
output/sandctl api ownership release <sandfs_id>
This topology does not run sandns; the dns commands need a DNS-bearing deployment.
3.10 Clean up
sudo bash deploy/baremetal/uninstall.sh
Stops and disables every unit (both gateways, both sandlots, etcd), removes the host mutations, and deletes /var/lib/sandswap unless --keep-data. Verify no residuals:
pgrep -af 'sandvmm|sandswap-(gateway|sandlot|etcd)' || echo "clean"
ss -ltn | grep -E ':(3000|3100|9998|2379)' || echo "ports clean"
4. Operator extras: deployment files & client flags
Everything above installs through deploy/baremetal; this section is for operating it by hand.
Deployment files. The deployers generate /etc/sandswap/*.yaml (plain YAML, fully editable) and the systemd units. Edit, then restart:
sudo vi /etc/sandswap/gateway.yaml
sudo systemctl restart sandswap-gateway
sudo journalctl -u sandswap-gateway -n 50 --no-pager
Client contract. /etc/sandswap/sandctl.env is root:root 0644 and carries public material only (URLs, TLS toggle, CA path; plus the etcd endpoint in the distributed shape). sandctl reads it automatically; set SANDSWAP_SANDCTL_ENV to point at another file. It never names a private key, and sandctl never reads a service identity’s key. --client-cert / --client-key are for mTLS deployments and must be caller-owned 0600 files.
Global flags. Global flags go before the subcommand and override the contract:
--run-dir --bin-dir --gateway-url --human-url --dns-addr --etcd
--tls --ca --client-cert --client-key --insecure --raw --dry-run
--dry-run prints the HTTP request without sending; --raw skips JSON pretty-printing. See sandctl for the full command & flag reference.
Measured latency (command-execution path + asynchronous unmount), with reproduction scripts, lives in Benchmark.