Threat Model
Sandswap’s security story has two halves.
Inside the microVM — arbitrary agent code vs the warden, the VMM filter, another agent on a reused VM — is Isolation and Swap: the per-exec confinement, the two isolation boundaries, and the switch-clearing guarantee that a reused microVM carries zero residue of its previous agent.
This page covers the other half: every adversary outside the microVM isolation boundary, and what contains them. The running principle is the four-identity split (sandlot 985, sandserv 986, sandvmm 987, sandprov 983 — see Host Prerequisites): every privileged surface is fenced to exactly one identity, so compromising one component does not confer its neighbors’ powers.
The adversaries:
- A non-root local user on a node — reaching the warden, the VMM control socket, the vsock device nodes, or sniffing existing traffic.
- The network — what can reach the guest, the internal control channels, and what spoofing the DNS costs.
- A compromised gateway process (the perimeter) — the external-facing process parses untrusted HTTP; assume an RCE there and design so it does not become data compromise.
- A compromised sandvmm process (the VMM) — the host-side process that serves the guest’s filesystem and device models.
1. Adversary 1 — a non-root local user on a node
The node’s privileged surfaces, and what admits a process to each. Every row is enforced by a different mechanism (LSM hook, file DAC, kernel socket-cred check), so no single bypass opens more than one door:
| Surface | Who may use it | Closed by |
|---|---|---|
Warden vsock port (guest CID:1024) | the sandlot holding the per-VM session key | per-frame HMAC-SHA256 — every request carries a nonce + tag over the fixed header and an op-specific body-field MAC (the warm-op mounted-volume identity or the cold mount target), keyed by the VM’s warden.session_key (generated by the sandlot at creation and delivered over the first host vsock connection with an unauthenticated OP_KEY_DELIVERY frame whose response is tagged with the delivered key; never on the kernel command line; verified before the body is read), and every response is tagged for the request nonce, with a separate MAC over the mounted-volume identity bytes. vsockguard (BPF LSM on socket_connect, AF_VSOCK port 1024 only for the sandlot UID) remains defense-in-depth, not the sole gate (Vsockguard) |
sandvmm API socket (/run/sandswap/vm/<id>/sandvmm.sock) | the sandvmm process; sandlot via the shared group where deployed | file DAC: socket is owner-sandvmm 0600, or 0660 with --api-gid widening the group to the shared group (a setgid sandbox dir; microvm.api_gid) — a local user can neither connect nor even locate a usable fd |
boxout socket (/run/sandswap-boxout/boxout.sock, K8s mode) | the sandlot pod only | a boxout-private root-owned 0711 directory (the sandlot traverses but cannot rename/replace entries) whose socket is allowed-uid 0600, secured via directory-fd Fchownat/Fchmodat with an inode re-check; SO_PEERCRED UID verification at accept time plus a per-connection HMAC-SHA256 challenge-response (-client-token-file/-boxout-token-file) from a pod-private emptyDir — the UID alone is a host-wide constant, so a foreign uid-65536 process can neither substitute the socket nor answer the challenge without the token |
/dev/vhost-vsock (CID registration / VM vsock devices) | the sandvmm identity (0600 owner) | file DAC — a non-root identity cannot create vsock devices or claim CIDs |
/dev/vsock (the node’s own vsock device) | locked to the sandlot identity (0600) | device hygiene — AF_VSOCK sockets themselves need no device node; the warden port is fenced by vsockguard regardless |
| Sniffing an established vsock connection | nobody non-root | kernel-vhost data path (no user-space intermediary to tap), ptrace_scope ≥ 1, perf_event_paranoid ≥ 2 (Host Prerequisites) |
Service configs / credentials (/etc/sandswap, cert keys) | root + the owning service group | file modes (0600/0640 root:<group>); this is also why registry credentials belong in YAML, not CLI flags — a flag is world-readable in /proc/<pid>/cmdline |
| Identity snapshot file | the gateway process only | mode 0600, hash-only credentials at rest, flock exclusivity, tmp+fsync+rename atomic writes |
VM state files (/run/sandswap/vm/<id>/) | root + the service identities | dir 0711 / state 0640; re-adopt adopts only survivors whose sandvmm is alive and whose API socket answers — a forged state entry cannot resurrect access |
| The volumes filesystem | the service identities (group access) | mounted nosuid,nodev on every consumer host — guest-planted setuid bits and device nodes are stored verbatim by the passthrough and made inert by the mount flags + the warden’s NO_NEW_PRIVS execs (Storage row 9) |
Two complementary fences cover the vsock transport, and neither alone suffices. Warden reach — connect() to any guest CID on port 1024 — is fenced by vsockguard for every UID except the sandlot’s, explicitly including the VMM identity: vsockguard-verify.sh probes uid 987 and asserts the denial (the VMM must never reach the warden port). CID registration — creating vsock devices — is fenced by /dev/vhost-vsock 0600 sandvmm, so a compromised sandlot cannot spawn VMs or claim CIDs. AF_VSOCK sockets themselves bypass device nodes, which is exactly why the first fence is an LSM hook rather than a permission bit. Asserted live by security/vsockguard-verify.sh (allowed/denied/sandvmm probes), e2e/seccomp.sh (device ownership), and security/verify-host-security.sh (check 4).
Known limits of this layer: root bypasses all of it (the KVM trust model — whoever owns the hypervisor host owns the VMs), and the LSM gate keys on host UIDs, so it is not itself a guest-to-guest control (measured posture in Vsockguard operational notes).
2. Adversary 2 — the network
- Guest inbound: none. The TAP is the only path in or out, under host-side enforcement — the network baseline and the per-attachment identity binding (Egress Enabler Contract); the guest is bound by the segment’s L3 policy.
- Gateway ↔ sandlot stream (
:9998): mutual TLS — TLS 1.3 withRequireAndVerifyClientCertin both directions (the stream CA is a deployment input; the shipped deploys always configure it), certificates scoped per service identity. A peer without a key cannot join the node fabric: an impostor sandlot cannot receive VMCommands, an impostor gateway cannot serve them. - Agent → gateway: HTTPS + api-key / OIDC auth (server-auth TLS; the credential, not the transport, is the client’s proof).
- etcd: the cluster’s coordination store — loopback or TLS (client-cert-auth) endpoints; see trust roots below.
- sandns: UDP/TCP 53 is unauthenticated — it has no auth and holds no secrets. That is bounded by design: answers carry only addresses (recommendations), the gateway’s own TLS pins the agent↔gateway channel so a spoofed A record cannot become a MITM, and it writes no ownership state. A poisoned sticky entry is a soft hint (last-writer-wins, self-healing, bounded by
sticky_max); its worst case is one extra 307 redirect or a migrated takeover — no data loss (Ownership & DNS).
3. Adversary 3 — the perimeter process
The gateway (uid sandserv) is treated as compromisable. An attacker with code execution in the gateway process reaches the following, and no more:
| Capability | Reachable from a compromised gateway? | Why / what bounds it |
|---|---|---|
| Mounts of shared storage | No | It mounts nothing: no volumes/templates filesystem, no fsGroup, no supplementary groups. vol_root exists in its config as a path convention for command payloads only |
| Arbitrary file operations on volumes | No | Every sandfs/template operation is a ProvisionCommand executed on a prov sandlot instance, which validates ids/refs node-side (single-path-element charset, no traversal) and resolves all paths itself |
| OCI registry egress (pull/push anything) | No | Registry access lives on provision hosts; the gateway’s network position is agent traffic + etcd + internal stream |
| The legitimate storage operation vocabulary | Yes — by abuse | A compromised gateway can still issue VALID operations (delete any sandfs, save-as-template + push as a coarse exfiltration channel). This is containment, not prevention: the blast radius is a bounded, auditable op set instead of arbitrary filesystem writes, and it is bounded further by the registry allowlist and by storage-plane instances being a separate identity (sandprov) that never holds agent-facing authority |
| Identity credentials of agents | Hash-only at rest; verification happens in-process | A full gateway compromise yields whatever the process can do via its legit API (mint agent keys in human mode) — the response is revocation through the identity store, which is the acknowledged limit of a compromised authority |
| Node exec path (warden vsock) | No | The gateway never speaks vsock; it only relays VMCommands over the authenticated stream, and the node applies its own policy (allowed_dirs, mount pinning, seccomp) regardless of who asked |
| The nodes’ VM lifecycle | No directly | VM create/destroy/mount requests require the mTLS stream identity; from the gateway’s own position they are its legitimate commands — the bound is that the node re-derives every isolation decision itself, so a malicious command still cannot cross a SandFS boundary |
Cluster-wide effect: because the storage plane is a SEPARATE identity class (sandprov, no caps, shared-group only) and a separate process on separate hosts, a compromised gateway cannot plant files that the execution fleet will serve — planted content requires compromising the provision plane itself.
The same split bounds a stolen credential. An agent api-key identifies exactly one agent and every per-sandfs request re-derives ownership from the record, so its blast radius is exactly the sandfs that agent owns — execute and file I/O inside those volumes and nothing else: not other agents’ sandfs, not the control plane (creating sandfs, minting agents, and template operations are human-gateway endpoints, so a leaked agent key cannot even grow its own footprint), not the node fabric, not any warden. A leaked human-side credential (OIDC token, or an mTLS client cert carrying all three scopes) unlocks only what its scope allows — sandswap:human (sandfs/template CRUD), sandswap:admin (credential minting), sandswap:ops (observability and fleet reclamation), checked per route, with lifetime, MFA, and revocation left to the IdP’s standard machinery. In both cases the response to a leak is revocation through the identity store, and the owned volumes’ contents must be treated as compromised — inherent, since the agent is their owner. The compromise-relevant asymmetry holds throughout: the human gateway holds the authority but mounts no storage.
4. Adversary 4 — a compromised sandvmm (the VMM process)
sandvmm is the host process that emulates the guest’s devices, including the virtio-fs file server (in-process virtiofsd) — so it is the data path for whatever its VM sees. What a full RCE in it holds, and what stays out of reach:
- Holds: the guest’s memory and console; the ability to serve any volume whose path it knows to its own guest (the
allowed_dirsfilter lives inside this process, and the daemon learns each path the VM is switched to). With the shared-group grant it can read/write those volumes directly on the host. It cannot enumerate the volumes root (2710, group execute-only) and cannot guess a name (sf-+ 128-bit HMAC-SHA256(seed, id)); the seed isroot:sandprov0640 and every sandvmm spawn replaces its supplementary set with exactlymicrovm.vmm_groups, so the seed never reaches it. sandvmm is therefore a data-confidentiality TCB for the volumes it has served, not for the whole tree — which is why it is a separate identity from the warden-trusted sandlot and is spawned with zero capabilities: the TAP is created and pre-configured (owner set to the sandvmm uid, MTU/bridge/up) by the network provider, and the VMM only attaches to it by owner match. - Cannot reach the warden transport: the vsock data path is the kernel’s vhost-vsock module (the process never touches guest vsock frames),
/dev/vsockfor host-side sockets is sandlot-owned 0600 (§1), and vsockguard denies uid 987 the warden port even if it could. Guest processes inside its VM also cannot escalate through it to the host (guest-planted setuid files land owned by 987 — an identity with no warden access; §1 and Host Prerequisites). - Cannot cross VMs: no shared memory with other VMs, ptrace fenced by
ptrace_scope, and vhost-vsock enforces CID uniqueness in-kernel — a second device cannot claim a live VM’s CID, so it cannot impersonate another guest at the vsock layer. - Cannot hold the host: zero capabilities (the TAP is created by the network provider), no
DAC_OVERRIDE; its supplementary groups are exactlykvm+ the shared storage group (it must serve volume I/O). File DAC still applies to everything outside the volumes it serves: the share root is traverse-only and sibling volume directories are reachable only by name, so the shared group does not become a cross-tenant read grant. Nor is it insandtmpl(the template-refs metadata group — memberssandprov+sandlotonly): it reads overlay layer content through the shared group but cannot enumerate the template stores’refs/, so the global template catalog stays out of a compromised VMM’s reach.
5. Trust roots and what each unlocks
The model names its trust roots — the components the rest of the system builds on, so nothing within the model can compromise them:
| Trust root | What its compromise unlocks | Standing protection |
|---|---|---|
| Host root | Everything on the node (all VMs’ memory, all identities) | the KVM trust model — out of scope by definition; the identity split exists to make non-root compromise insufficient |
| etcd | Cluster control: identity records (hash-only), sandfs records, ownership/placement keys — a writer can re-route and claim ownerships | loopback/TLS client-cert endpoints; treat etcd as a production secret, not a shared service |
| Gateway human authority | Mint/revoke agent keys, CRUD sandfs | the perimeter model (§3): revocation is the response |
| Stream CA + certs | Impersonate gateways or sandlots on the control fabric | per-identity key material, 0600 root-scoped; rotation is redeploy |
| Guest kernel + initramfs (the warden) | A boundary breach, not an in-sandbox event: an implant survives the tenant switch and reads every volume later attached to that VM — per-VM, never fleet-wide | the pinned minimized build, the load/runtime posture, the capability/seccomp fence, and the reuse limits: Guest Kernel Hardening |
6. The boundary this page does not re-derive
What happens inside the microVM — the exec profiles and seccomp deny-list, the warden’s self-hardening against same-uid attack, the pinned mount protocol and the host-CID heartbeat gate, the virtio-fs allowed_dirs filter, and the switch-clearing chain (processes, mounts, caches, kernel objects, sockets) that guarantees zero residue between two agents on one VM — is the subject of Isolation and Swap, and deliberately not restated here: this page assumes that boundary holds and asks what is left outside it. The guest kernel as a trust root — the minimized build, the allowed/denied syscall surface, the blast radius of a compromise, and the reuse limits that bound it — is Guest Kernel Hardening. The host-side invariants it leans on are asserted live by e2e/seccomp.sh, security/verify-host-security.sh, and security/vsockguard-verify.sh.
Two channels sit deliberately OUTSIDE the in-VM boundary and are recorded here as deployment duties:
- Guest → host vsock (CID 2): the agent seccomp filter denies
socket(AF_VSOCK)(EPERM), so sandswap guests cannot create a vsock socket at all; the warden’s own listener is created before the filter is installed. This is code-level blocking, not a deployment duty. If a future image ships without that rule, any host-side vsock service must still treat guest peers as unauthenticated-hostile (the warden’s own port applies the same rule to ITS peers — isolation-and-swap.md §7.3). - Guest console logs on the node: the VMM captures the guest kernel console into per-VM
console.log/serial.logon the host. Guest code can induce kernel printk at some rate, so the sandlot enforces a size cap (monitor.go— truncate past 128 MB on a slow ticker); a node’s log partition must still be provisioned for the capped aggregate across its VM fleet.