Architecture
Sandswap serves every AI agent request with an isolated sandbox on demand: Sandswap attaches a warm pooled microVM with the agent’s persistent filesystem, and releases the VM when it goes idle. This page is the structural view: the components, the optional ones, the connections, and the flow of one request.
1. System context
Everything inside the Sandswap boundary is shipped as a Sandswap binary or build artifact:
| Component | Kind | Present when | Role |
|---|---|---|---|
| gateway | Service | Always | The entry point: auth, SandFS registry, ownership, sticky lease, dispatch to nodes. Deployed as one combined process (single-node), or split into a human gateway (ops) and exec gateways (agent traffic) |
| sandlot | Daemon | Always — one per node | The node daemon: keeps a pool of warm microVMs ready, swaps the agent’s SandFS in and out of a VM, and performs sandfs/template lifecycle on shared storage. Deployed as one combined instance (-roles exec,prov), or split into exec sandlot instances (compute) and prov sandlot instances (storage) |
| sandvmm | Process | Always — one per microVM | The KVM virtual machine monitor |
| warden | Process | Always — PID 1 in every microVM | Mounts the SandFS, runs the agent’s commands |
| sandns | Service | Multi-gateway deployments | The owner-aware DNS service: answers “which gateway serves this SandFS?” and “which gateway serves this sandlot?” on DNS :53 |
| boxout | Sidecar | Kubernetes (boxout mode) | Privileged helper so sandlot can run unprivileged |
| kernel + initramfs | Artifacts | Always | The boot image of every microVM (warden inside) |
| vsockguard | BPF LSM | Optional host hardening | Pins the host-side vsock transport against local users |
Around the boundary sit third-party systems Sandswap integrates with but does not replace — none of them is required in the default single-node deployment:
| External system | Used for | Needed when |
|---|---|---|
| etcd | Ownership, health, and routing records shared between gateways and sandns | Multi-gateway deployments |
| Cloud FS / NFS / CephFS / … | Shared volumes storage | Multi-node deployments |
| OCI registry | Pull/push of rootfs templates | When templates are pulled or published |
The actors are AI agents (HTTP or MCP, the same listener) and operators (human API: SandFS CRUD, templates).
2. Logical architecture
Solid boxes are Sandswap components; [EXT] marks a third-party system, [opt.] a component needed only in specific deployments.
AI agents operators
(HTTP :3000 / MCP, same listener) (human API :3100 when split)
│ │
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ exec gateway │ │ human gateway │
│ agent+MCP :3000 │ │ human API :3100 │
│ stream :9998 │ │ prov stream :9998 │
│ auth · ownership │ │ SandFS CRUD · │
│ sticky lease │ │ templates CRUD │
└──────────────┬────┘ └───┬───────────────┘
▲ │ │ ▲
│ │ownership SandFS│ │ gRPC bidi stream
│ │routing state state │ │ (sandlot dial out)
│ │ ┌───────────────┐ │ │
│ │ │ sandns [opt.] │ │ │
│ │ │ on DNS :53 │ │ │
│ │ └──────▲────────┘ │ │
│ │ │watch,read-only │ │
│ │ ┌╌╌╌╌╌╌╌╌╌┴╌╌╌╌╌╌╌╌╌┐ │ │
│ └─►│ etcd [EXT] │◄──────┘ │
│ │ (multi-gw) │ │
│ └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘ │
│ gRPC bidi stream(sandlot dial out) │
┌─────▼─────────────────────────┐ ┌──────────────▼──────────────────┐
│ node × N — exec sandlot │ │ instances × M — prov sandlot │
│ sandvmm · microVMs (warden) │ │ sandfs + template lifecycle │
│ one SandFS per microVM │ │ OCI pull/push [EXT] │
│ at a time │ └──────────────┬──────────────────┘
└─────┬─────────────────────────┘ │
│ virtio-fs passthrough │ volume + template writes
│ — only the active agent's │
│ directory │
▼ ▼
┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
│ volumes + templates — one directory per SandFS │
│ local disk, or Cloud FS / NFS / CephFS / ... [EXT] │
└╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
Notes on lines not drawn:
- Single-node mode is one combined gateway and one combined sandlot. The diagram draws the split deployment — human gateway + exec gateway, prov sandlot + exec sandlot — because the two roles have different responsibilities and peers. Both splits are optional: in single-node mode one gateway process serves both listeners and accepts both stream roles, and one sandlot carries both roles (
-roles exec,prov). - Agents and sandlots query sandns (
:53) to find the gateway they should talk to — multi-gateway deployments only; in single-node mode they address the gateway directly. - Gateways never touch the storage plane. SandFS creation/seeding, template pull/push/save, and volume deletion are ProvisionCommands the gateway routes over the stream to a prov sandlot instance (deployed on storage-attached hosts, or the same host as the gateway in compact topologies). No gateway mounts anything from the volumes/templates trees or holds registry egress.
2.1 The three planes
- Control plane — the human gateway, the prov sandlot instances, and etcd: manage SandFS identities and templates, track ownership, and perform all sandfs/template storage work. Agent code never travels through it.
- Exec plane — the exec gateway, sandns (optional), and the exec sandlot nodes: gateway → sandlot → sandvmm → microVM (warden). Every command an agent runs travels this chain.
- Storage plane — the volumes + templates trees: one directory per SandFS on local disk or shared storage. Written ONLY by prov sandlot instances (creation, seeding, deletion, template pull/push) and the serving FUSE daemon (sandvmm); exec sandlot nodes read it via virtio-fs passthrough.
The central invariant lives in the exec plane: one SandFS per microVM at a time — a microVM runs at most one SandFS, and Sandswap mounts one SandFS in at most one microVM.
3. Inside a sandlot exec node
The exec sandlot node from the logical architecture — the one that runs microVMs. A prov sandlot runs the same sandlot binary with no VM side: no sandvmm, no warden clients, no pool — just storage operations against the volumes/templates trees.
▲ to gateway :9998
│ (gRPC bidi stream — sandlot dials out,
│ the gateway never connects in)
┌────────┼──────────────────────────────────────────┐
│ node │ │
│ │ │
│ ┌────┴───────────────────┐ │
│ │ sandlot │ │
│ │ VM pool · SandFS swap ├──►┌──────────────┐ │
│ │ warden client (vsock) │ │ boxout │ │
│ └─┬─────────────────────┬┘ │ [opt., K8s] │ │
│ │ HTTP over UDS │ └──────────────┘ │
│ ┌─▼──────────────┐ │ vsock :1024 │
│ │ sandvmm │ │ direct to warden — │
│ │ KVM · virtio-fs│ │ no sandvmm hop │
│ └─┬──────────────┘ │ │
│ │ KVM │ │
│ ┌─▼──────────────────┐ │ │
│ │ microVM │ │ │
│ │ warden (PID 1) │◄─┘ │
│ │ chroot: /mnt/rootfs│ │
│ │ eth0 → host bridge │ │
│ └────────┬───────────┘ │
│ │ virtio-fs passthrough │
│ │ — only the active │
│ │ agent's directory │
└────────────┼──────────────────────────────────────┘
│
│
┌─╌╌╌╌╌╌╌╌╌╌─▼─╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌──┐
│ volumes — one directory per SandFS │
│ local disk, or Cloud FS / NFS / │
│ CephFS / ... [EXT] │
└──╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌──┘
The connections, one by one:
- gRPC stream (up,
▲) — sandlot’s only link to the rest of the platform. One persistent outbound connection to the gateway; the stream multiplexes all VM commands, events, and heartbeats. - HTTP over UDS (control) — sandlot drives sandvmm: create, boot,
set-fs, ping, shutdown. Rare, milliseconds, never on the hot path. - KVM — sandvmm runs the microVM: its own kernel, warden as PID 1.
- vsock
:1024(business path) — sandlot talks to the warden directly. The connection is carried by the kernel’s vhost-vsock backend; sandvmm is not in the data path. Mount, exec, unmount, stats all travel here. - UDS to boxout (optional, Kubernetes) — the privileged broker for sandvmm lifecycle, networking, and host-side overlay mounts.
- virtio-fs passthrough (down) — the guest sees exactly one SandFS directory; every write lands in the agent’s persistent volume.
- eth0 → host bridge — guest egress to the internet; on egress-enabler nodes the traffic carries the Sandswap egress identity (see Egress Enabler Contract).
4. Deployment views
4.1 Single-node (default)
One gateway process, one node, no external systems — identity state is an in-process store persisted to a snapshot file:
External AI Agents/Human (HTTP/MCP)
│
▼
┌─────────────────┐ gRPC bidirectional stream ┌──────────────────┐
│ gateway │◄───────────────────────────────►│ sandlot (node) │
│ :3000 agent+MCP │ (sandlot dials out :9998) │ exec+prov roles │
│ +human API │ │ ┌──────────────┐ │
│ :9998 exec+prov │ │ │ sandvmm │ │
│ in-process │ │ │ ┌────────┐ │ │
│ identity store │ │ │ │microVM │ │ │
│ (snapshot file) │ │ │ │ warden │ │ │
└─────────────────┘ │ │ └────────┘ │ │
│ └──────────────┘ │
└──────────────────┘
Volumes live on local disk. This is the smallest complete deployment: every component above except etcd, sandns, boxout, and the OCI registry.
4.2 Distributed
The human gateway (SandFS/template management) + exec gateway instances (agent traffic) + etcd + sandns; N exec node hosts dial the exec gateway, M prov sandlot instances dial the human gateway:
┌────────────┐ ┌─╌╌╌╌╌╌╌╌╌╌╌╌╌╌─┐
│ sandns │◄───│ etcd cluster │
│ on DNS :53 │ │ (state store) │
└────────────┘ └─╌╌┬─╌╌╌╌╌╌╌╌─┬─┘
│ │
┌─────────────────▼──┐ ┌────▼───────────────┐
│human gateway │ │exec gateway │
│:3100 /human/* │ │:3000 /agent/* │
│prov stream :9996 │ │:9998 node stream │
└─────────▲──────────┘ └─────────▲──────────┘
│ │
│ │
│ gRPC stream │ gRPC stream
│ (sandlot dial out) │ (sandlot dial out)
│ │
┌─────────▼──────────┐ │
│prov sandlot ×M │ │
│storage + OCI ops │ │
└────────────────────┘ │
┌──────────────────┬─────┴────────────┐
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│exec sandlot │ │exec sandlot │ │exec sandlot │
│node-1 │ │node-2 │ │node-3 │
│KVM microVMs │ │KVM microVMs │ │KVM microVMs │
└─────────────┘ └─────────────┘ └─────────────┘
The human gateway and exec gateway share SandFS state via etcd: the human gateway does SandFS CRUD with distributed locks; exec gateways serve agent traffic — execution and GET /agent/sandfs discovery alike — from a read-only cache synced by watch. Multiple exec gateways sit behind owner-aware DNS (see Ownership & DNS). The prov sandlot instances under the human gateway dial its stream listener (:9996) and execute the storage operations behind SandFS CRUD and template commands.
4.3 Sandlot privilege modes: embedded vs boxout
Which modes a platform supports:
| Platform | Embedded | Boxout |
|---|---|---|
| Bare metal | ✓ (the only mode) | — |
| Kubernetes | ✓ | ✓ |
The node diagrams show embedded mode: sandlot is privileged and manages sandvmm directly — on bare metal this is the only mode.
In boxout mode (Kubernetes) — experimental — sandlot runs unprivileged and delegates the privileged path (sandvmm lifecycle, network provisioning, host-side overlay mounts) to a privileged boxout sidecar over a Unix socket. The separation has not delivered the expected benefit: boxout itself still needs excessive privileges, and the split adds complexity and recurring permission problems between sandlot and boxout. The hot path stays in sandlot with zero privileges — details in Install — Kubernetes.
5. Interface catalog
| Path | Transport | Initiator | Purpose |
|---|---|---|---|
| Agent → gateway | HTTP :3000 (MCP on the same listener) | agent | execute, file I/O, background exec status/kill, SandFS discovery |
| Operator → gateway | human API (same listener single-node; :3100 split modes) | operator | SandFS CRUD, template pull/push/save |
| exec sandlot → gateway | gRPC bidi stream, TCP :9998 (TLS/mTLS) | exec sandlot | VM commands, async events, heartbeats — one outbound connection per exec instance |
| prov sandlot → gateway | gRPC bidi stream, TCP :9998 (TLS/mTLS) | prov sandlot | ProvisionCommands + responses (volume/template lifecycle). Roles are advertised in NodeHello; the gateway routes each plane only to matching role connections |
| exec sandlot → sandvmm | HTTP/1.1 over Unix socket | exec sandlot | VM create/boot, vsock attach, set-fs, ping, shutdown |
| exec sandlot → warden | vsock (vhost-vsock), port 1024 | exec sandlot | mount/exec/unmount, memory stats, background exec |
| exec sandlot → boxout | newline-JSON over Unix socket | exec sandlot | privileged lifecycle ops (boxout mode only) |
| gateway → etcd | gRPC read/write/watch | gateway | ownership, health, placement, gateway registry |
| sandns → etcd | gRPC watch (read-only) | sandns | routing tables |
| prov sandlot ↔ OCI registry | OCI distribution API | prov sandlot | template pull/push — the egress lives on the provision host, never on the gateway |
| exec sandlot → volumes | filesystem (virtio-fs) | exec sandlot | read-only overlay lowerdirs; the serving FUSE daemon (sandvmm) writes guest data |
| prov sandlot → volumes/templates | filesystem | prov sandlot | the ONLY writer of volume trees (create/seed/delete) and the template store |
| agents, sandlots → sandns | UDP/TCP :53 | agent / sandlot | gateway address resolution (multi-gateway) |
| microVM → internet | guest eth0 → host bridge | guest | egress; identity-stamped on enabler nodes |
The three internal paths that carry all the traffic, in detail:
5.1 Gateway ↔ sandlot (gRPC bidirectional stream)
Sandlot ─── outbound TCP ───→ Gateway (:9998)
│ │
│ NodeHello{roles:[exec|prov]} → │ (handshake; the gateway rejects an
│ ←── HelloAck{…|reject_reason} │ instance whose roles it cannot accept,
│ │ naming the reason)
│ │
│ ←──────── VMCommand{...} ───── │ (exec role only)
│ CommandResponse{...} ────────→ │
│ ←── ProvisionCommand{…} ────── │ (prov role only)
│ ProvisionResponse{…} ────────→ │
│ │
│ VMEvent{VSOCK_READY} ────────→ │ (async events)
│ Heartbeat ───────────────────→ │ (periodic)
No ports are exposed on sandlot. It initiates a single persistent TCP connection to the gateway (TLS/mTLS), and the gateway sends commands on the response half of the stream. The message catalog is in Internal Protocols.
5.2 sandlot ↔ sandvmm (HTTP + raw frames over Unix socket)
Sandlot ─── HTTP/1.1 | raw frames ───→ /run/sandswap/vm/<id>/sandvmm.sock
HTTP/JSON — VM lifecycle (once per VM):
PUT /api/v1/vm.create
PUT /api/v1/vm.add-vsock (vhost-vsock device, body {cid})
PUT /api/v1/vm.boot
PUT /api/v1/sandvmm.shutdown
raw control frames — per-request hot ops (no HTTP/JSON framing):
vm.set-fs (virtio-fs allowed_dirs swap — every cold attach)
sandvmm.ping (health probes)
One socket, two protocols: sandvmm dispatches each connection by the first byte on the wire (0x01 → raw frame, ASCII method letter → HTTP). The hot ops default to length-prefixed binary frames because they run on every cold attach and probe cycle, where HTTP/JSON framing overhead is measurable (sandlot -setfs-transport=http selects the legacy REST form). Frame layout and budgets: Internal Protocols.
No containerd, no shim — direct control of the VMM.
5.3 sandlot ↔ warden (raw vsock protocol)
Sandlot ─── kernel AF_VSOCK (guest CID, port 1024) ───→ MicroVM
raw vsock protocol:
OP_MOUNT_AND_EXEC (1) — mount + exec in one RTT (cold)
OP_UNMOUNT (2) — unmount + cgroup.kill
OP_EXEC (3) — exec only (warm)
OP_MEM_STATS (4) — heartbeat: guest memory stats
OP_ASYNC_EXEC (5) — start background exec (warm)
OP_MOUNT_AND_ASYNC_EXEC (6) — mount + start background exec (cold, 1 RTT)
OP_STATUS_EXEC (7) — poll background exec status
OP_KILL_EXEC (8) — kill background exec
OP_FILE (9) — file I/O (unified sub-opcode dispatch)
vsock backend: kernel vhost-vsock — chosen for ultra-low latency. The sandlot↔warden channel is the hottest transport in the system: every exec, file op, and stat crosses it, so the transport itself must add close to nothing. The two conventional alternatives each add measurable cost — emulating the vsock device inside the VMM process (userspace virtio) forces a VMEXIT and a VMM context switch on every packet, and going over the guest NIC (virtio-net + TCP/IP) adds a full network stack — sockets, TCP, a host bridge — to the data path. With kernel vhost-vsock, the entire data path is offloaded to the Linux kernel via /dev/vhost-vsock: at device activation the VMM hands guest memory + virtqueues to a kernel vhost worker and never touches a packet afterward — zero VMEXIT per packet, no VMM context switch, and a bare kernel AF_VSOCK socket pair instead of a network stack. This keeps the vsock hop off the critical path: millisecond-scale end-to-end ops are dominated by the work being done, not by the transport. The vhost ioctl ABI is frozen UAPI (kernel 4.8+), so one VMM binary runs across kernel versions. (Runtime prerequisite: /dev/vhost-vsock readable — sandlot checks at startup.)
Connection pooling: the warden accepts multiple concurrent vsock connections. Business ops use a per-VM pool (up to exec_slots resident connections, one in-flight request each); each connection’s receive timeout is re-set per request to the op’s duration + 30 s margin (capped at the warden countdown), so a wedged connection fails fast instead of blocking forever. Heartbeat probes use separate short-lived connections (fresh every 3 s) so they never queue behind a long sync exec.
Security hardening: vsockguard. The warden performs zero authentication on this port — whatever process manages to connect() to (guest CID, 1024) can send mount/exec frames and act as the sandlot. Device-node permissions cannot close this: an AF_VSOCK connect() goes through no device node, and /dev/vhost-vsock only gates CID registration, not connecting to an existing guest — so any local user on the node can dial the warden. vsockguard closes this gap in the kernel: a BPF LSM program on the socket_connect hook denies connect() to AF_VSOCK port 1024 for every UID except the sandlot service UID (held in a BPF map, written by the loader at attach time); all other address families and ports pass through untouched. Two operational notes: the kernel’s bpf LSM must be active (/sys/kernel/security/lsm) or the attached program is a silent no-op — vsockguardctl refuses to attach otherwise — and vsockguard covers active impersonation only; passive sniffing of an existing connection is prevented by the host security invariants. Full story in Vsockguard.
6. Life of a request
One tool call, end to end — the cold path (idle VM, mount first):
agent gateway sandlot warden
│ POST /agent/execute │ │ │
├─────────────────────►│ │ │
│ │ api_key → agent │ │
│ │ owns sandfs_id ✓ │ │
│ │ claim ownership │ │
│ │ pick node · reserve│ │
│ │ start sticky lease │ │
│ ├───────────────────►│ │
│ │ │ bind idle pool VM│
│ │ │ vm.set-fs │
│ │ │ allowed_dirs swap│
│ │ │ OP_MOUNT_AND_EXEC│
│ │ ├─────────────────►│
│ │ │ │ mount /mnt/rootfs
│ │ │ │ chroot → run cmd
│ │ │◄─────────────────┤ status + output
│ │◄───────────────────┤ response │
│◄─────────────────────┤ result JSON │ │
│ │ │ │
A follow-up call from the same agent inside the sticky-lease window skips the middle of the diagram: the VM is still mounted, so the gateway sends one command and sandlot issues a single OP_EXEC on the cached vsock connection — the warm path.
The lifecycle around a request:
1. SandFS creation (one-time, human API). The human gateway generates a sandfs_id and creates the volume on the volumes storage — no mount yet. Flat mode is seeded either by copying a local template directory or, for OCI-origin templates, by streaming the layer blobs straight into the volume with permissions applied inline (no blob cache — dedup is the storage backend’s job). Overlay mode creates rootfs/ · workspace/ · home/ plus a .template marker, with the template’s OCI layers extracted once into the shared content-addressed layers/ cache. The SandFS is owned by an agent identity: that agent authenticates with its provisioned api_key (config-file injection — zero code changes) and names this sandfs_id on every request.
2. The warm pool (continuous, per node — not per request). Each sandlot keeps its own pool of CLEAN microVMs per spec: pre-created at startup and refilled asynchronously as VMs are consumed or recycled. Creating a VM is purely local node work — the gateway never commands it: sandlot launches the sandvmm process with a permanent virtio-fs device (tag sandswap, shared dir = volumes dir, writable), boots kernel + initramfs, and warden starts as PID 1 listening on vsock:1024. The request path contains no boot: if no CLEAN VM fits, the request fails fast (NO_AVAILABLE_SANDBOX) while the pool refills in the background.
3. Code execution (every tool call). The sequence above. Gateway side: authenticate the api_key, resolve sandfs_id and its ownership, pick a node (filter → affinity → score, reserving one clean slot), open the lease, and send a single mount-and-execute command. Sandlot side: bind a CLEAN VM to the sandfs (one sandfs → one VM; a warm bound VM is reused), lazily host-mount the read-only lowerdir in overlay mode, and swap the virtio-fs filter (allowed_dirs = [agent's dir], ~0.20 ms); the warden then performs the guest mount (bind rootfs → /mnt/rootfs, plus workspace/home/tmpfs in overlay mode) and runs the command in a chroot — mount and exec travel in a single vsock round trip. Only the active agent’s directory is visible to the guest; every write lands on the agent’s persistent volume.
4. Sticky lease (same agent, consecutive calls). Within the lease window (default 100 ms) the VM is still mounted — the next call takes the warm path. The 100 ms is a latency/VM-hold trade-off, not a functional floor: -sticky-ms accepts down to 1 ms (the minimum; 0 is rejected), and the load benchmark ran a 1 ms window with back-to-back follow-up calls still landing warm — see baremetal-load.
5. Background exec (long-running tools). background: true returns an exec_id immediately; the warden spawns the process under setsid, logs to a temp file, and adds it to the agent’s cgroup. The agent polls status / kills by id. The lease_duration is an absolute run-time cap (polling does not extend it). Details in the Agent API Reference.
6. Release (async). After the lease expires, the gateway sends the unmount off the response path: the warden kills every agent process (cgroup.kill on the lazily created agent cgroup), tears down mounts, umounts, and drops caches; sandlot clears allowed_dirs. The VM returns to the pool CLEAN — or sandlot destroys and replaces it when the unmount fails (the exclusive-ownership guarantee) — and recycles at the max_uses / memory-pressure thresholds rather than running forever.
7. Agent B on the same microVM. A new mount-and-execute with agent B’s volume: filter swap + bind-mount swap after killing A’s processes. B cannot see any of A’s data — enforced at both the VMM level (allowed_dirs) and inside the guest (chroot), with the memory-clearing chain in between (see Isolation and Swap).
7. Key design decisions
| Decision | Choice | Rationale |
|---|---|---|
| Isolation tech | KVM microVM (sandvmm) | Hardware-level isolation, no containerd needed |
| Boot method | Initramfs cold boot | No snapshot needed; kernel VFS cache naturally consistent |
| Agent volume role | is the rootfs | All writes inside the chroot persist naturally |
| Hot-swap mechanism | mount/umount + virtio-fs allowed_dirs | bind-mount swap + VMM-level directory filter; set-fs ~0.20 ms |
| OverlayFS mode | Read-only overlay + writable workspace/ · home/ | No upperdir/workdir, so it works on any shared filesystem — NFS, CephFS, Cloud FS, or newer storage systems; lazy mount, stateless overlay |
| Volume storage | Local disk or shared (Cloud FS / NFS / CephFS / …) | Sandswap only sees directories — the storage backend is replaceable; shared storage lets any node serve any SandFS |
| Agent identity | API key (Bearer) | Config-file injection, zero code changes |
| microVM reuse | Sticky lease + async clean | Same-agent reuse skips remount |
| sandlot connectivity | Outbound gRPC stream | No ports exposed on sandlot; NAT-friendly |
| VMM control | HTTP + raw frames over Unix socket | Direct control, no containerd/shim layer; binary frames for per-request hot ops |
| Background exec | Per-agent cgroup + async API | Long-running tools survive across calls; cgroup.kill on release |
8. Failure behavior
The tolerance-first model answers every component failure: ownership does not move, agents get retryable 503s, and Sandswap switches only after living participants confirm the old mount is released — the Resilience page is the full story (timelines, takeovers, self-fencing, and the both-dead manual backstops).