Threat Model

Sandswap’s security story has two halves.

Inside the microVM — arbitrary agent code vs the warden, the VMM filter, another agent on a reused VM — is Isolation and Swap: the per-exec confinement, the two isolation boundaries, and the switch-clearing guarantee that a reused microVM carries zero residue of its previous agent.

This page covers the other half: every adversary outside the microVM isolation boundary, and what contains them. The running principle is the four-identity split (sandlot 985, sandserv 986, sandvmm 987, sandprov 983 — see Host Prerequisites): every privileged surface is fenced to exactly one identity, so compromising one component does not confer its neighbors’ powers.

The adversaries:

  1. A non-root local user on a node — reaching the warden, the VMM control socket, the vsock device nodes, or sniffing existing traffic.
  2. The network — what can reach the guest, the internal control channels, and what spoofing the DNS costs.
  3. A compromised gateway process (the perimeter) — the external-facing process parses untrusted HTTP; assume an RCE there and design so it does not become data compromise.
  4. A compromised sandvmm process (the VMM) — the host-side process that serves the guest’s filesystem and device models.

1. Adversary 1 — a non-root local user on a node

The node’s privileged surfaces, and what admits a process to each. Every row is enforced by a different mechanism (LSM hook, file DAC, kernel socket-cred check), so no single bypass opens more than one door:

Surface Who may use it Closed by
Warden vsock port (guest CID:1024) the sandlot holding the per-VM session key per-frame HMAC-SHA256 — every request carries a nonce + tag over the fixed header and an op-specific body-field MAC (the warm-op mounted-volume identity or the cold mount target), keyed by the VM’s warden.session_key (generated by the sandlot at creation and delivered over the first host vsock connection with an unauthenticated OP_KEY_DELIVERY frame whose response is tagged with the delivered key; never on the kernel command line; verified before the body is read), and every response is tagged for the request nonce, with a separate MAC over the mounted-volume identity bytes. vsockguard (BPF LSM on socket_connect, AF_VSOCK port 1024 only for the sandlot UID) remains defense-in-depth, not the sole gate (Vsockguard)
sandvmm API socket (/run/sandswap/vm/<id>/sandvmm.sock) the sandvmm process; sandlot via the shared group where deployed file DAC: socket is owner-sandvmm 0600, or 0660 with --api-gid widening the group to the shared group (a setgid sandbox dir; microvm.api_gid) — a local user can neither connect nor even locate a usable fd
boxout socket (/run/sandswap-boxout/boxout.sock, K8s mode) the sandlot pod only a boxout-private root-owned 0711 directory (the sandlot traverses but cannot rename/replace entries) whose socket is allowed-uid 0600, secured via directory-fd Fchownat/Fchmodat with an inode re-check; SO_PEERCRED UID verification at accept time plus a per-connection HMAC-SHA256 challenge-response (-client-token-file/-boxout-token-file) from a pod-private emptyDir — the UID alone is a host-wide constant, so a foreign uid-65536 process can neither substitute the socket nor answer the challenge without the token
/dev/vhost-vsock (CID registration / VM vsock devices) the sandvmm identity (0600 owner) file DAC — a non-root identity cannot create vsock devices or claim CIDs
/dev/vsock (the node’s own vsock device) locked to the sandlot identity (0600) device hygiene — AF_VSOCK sockets themselves need no device node; the warden port is fenced by vsockguard regardless
Sniffing an established vsock connection nobody non-root kernel-vhost data path (no user-space intermediary to tap), ptrace_scope ≥ 1, perf_event_paranoid ≥ 2 (Host Prerequisites)
Service configs / credentials (/etc/sandswap, cert keys) root + the owning service group file modes (0600/0640 root:<group>); this is also why registry credentials belong in YAML, not CLI flags — a flag is world-readable in /proc/<pid>/cmdline
Identity snapshot file the gateway process only mode 0600, hash-only credentials at rest, flock exclusivity, tmp+fsync+rename atomic writes
VM state files (/run/sandswap/vm/<id>/) root + the service identities dir 0711 / state 0640; re-adopt adopts only survivors whose sandvmm is alive and whose API socket answers — a forged state entry cannot resurrect access
The volumes filesystem the service identities (group access) mounted nosuid,nodev on every consumer host — guest-planted setuid bits and device nodes are stored verbatim by the passthrough and made inert by the mount flags + the warden’s NO_NEW_PRIVS execs (Storage row 9)

Two complementary fences cover the vsock transport, and neither alone suffices. Warden reach — connect() to any guest CID on port 1024 — is fenced by vsockguard for every UID except the sandlot’s, explicitly including the VMM identity: vsockguard-verify.sh probes uid 987 and asserts the denial (the VMM must never reach the warden port). CID registration — creating vsock devices — is fenced by /dev/vhost-vsock 0600 sandvmm, so a compromised sandlot cannot spawn VMs or claim CIDs. AF_VSOCK sockets themselves bypass device nodes, which is exactly why the first fence is an LSM hook rather than a permission bit. Asserted live by security/vsockguard-verify.sh (allowed/denied/sandvmm probes), e2e/seccomp.sh (device ownership), and security/verify-host-security.sh (check 4).

Known limits of this layer: root bypasses all of it (the KVM trust model — whoever owns the hypervisor host owns the VMs), and the LSM gate keys on host UIDs, so it is not itself a guest-to-guest control (measured posture in Vsockguard operational notes).

2. Adversary 2 — the network

  • Guest inbound: none. The TAP is the only path in or out, under host-side enforcement — the network baseline and the per-attachment identity binding (Egress Enabler Contract); the guest is bound by the segment’s L3 policy.
  • Gateway ↔ sandlot stream (:9998): mutual TLS — TLS 1.3 with RequireAndVerifyClientCert in both directions (the stream CA is a deployment input; the shipped deploys always configure it), certificates scoped per service identity. A peer without a key cannot join the node fabric: an impostor sandlot cannot receive VMCommands, an impostor gateway cannot serve them.
  • Agent → gateway: HTTPS + api-key / OIDC auth (server-auth TLS; the credential, not the transport, is the client’s proof).
  • etcd: the cluster’s coordination store — loopback or TLS (client-cert-auth) endpoints; see trust roots below.
  • sandns: UDP/TCP 53 is unauthenticated — it has no auth and holds no secrets. That is bounded by design: answers carry only addresses (recommendations), the gateway’s own TLS pins the agent↔gateway channel so a spoofed A record cannot become a MITM, and it writes no ownership state. A poisoned sticky entry is a soft hint (last-writer-wins, self-healing, bounded by sticky_max); its worst case is one extra 307 redirect or a migrated takeover — no data loss (Ownership & DNS).

3. Adversary 3 — the perimeter process

The gateway (uid sandserv) is treated as compromisable. An attacker with code execution in the gateway process reaches the following, and no more:

Capability Reachable from a compromised gateway? Why / what bounds it
Mounts of shared storage No It mounts nothing: no volumes/templates filesystem, no fsGroup, no supplementary groups. vol_root exists in its config as a path convention for command payloads only
Arbitrary file operations on volumes No Every sandfs/template operation is a ProvisionCommand executed on a prov sandlot instance, which validates ids/refs node-side (single-path-element charset, no traversal) and resolves all paths itself
OCI registry egress (pull/push anything) No Registry access lives on provision hosts; the gateway’s network position is agent traffic + etcd + internal stream
The legitimate storage operation vocabulary Yes — by abuse A compromised gateway can still issue VALID operations (delete any sandfs, save-as-template + push as a coarse exfiltration channel). This is containment, not prevention: the blast radius is a bounded, auditable op set instead of arbitrary filesystem writes, and it is bounded further by the registry allowlist and by storage-plane instances being a separate identity (sandprov) that never holds agent-facing authority
Identity credentials of agents Hash-only at rest; verification happens in-process A full gateway compromise yields whatever the process can do via its legit API (mint agent keys in human mode) — the response is revocation through the identity store, which is the acknowledged limit of a compromised authority
Node exec path (warden vsock) No The gateway never speaks vsock; it only relays VMCommands over the authenticated stream, and the node applies its own policy (allowed_dirs, mount pinning, seccomp) regardless of who asked
The nodes’ VM lifecycle No directly VM create/destroy/mount requests require the mTLS stream identity; from the gateway’s own position they are its legitimate commands — the bound is that the node re-derives every isolation decision itself, so a malicious command still cannot cross a SandFS boundary

Cluster-wide effect: because the storage plane is a SEPARATE identity class (sandprov, no caps, shared-group only) and a separate process on separate hosts, a compromised gateway cannot plant files that the execution fleet will serve — planted content requires compromising the provision plane itself.

The same split bounds a stolen credential. An agent api-key identifies exactly one agent and every per-sandfs request re-derives ownership from the record, so its blast radius is exactly the sandfs that agent owns — execute and file I/O inside those volumes and nothing else: not other agents’ sandfs, not the control plane (creating sandfs, minting agents, and template operations are human-gateway endpoints, so a leaked agent key cannot even grow its own footprint), not the node fabric, not any warden. A leaked human-side credential (OIDC token, or an mTLS client cert carrying all three scopes) unlocks only what its scope allows — sandswap:human (sandfs/template CRUD), sandswap:admin (credential minting), sandswap:ops (observability and fleet reclamation), checked per route, with lifetime, MFA, and revocation left to the IdP’s standard machinery. In both cases the response to a leak is revocation through the identity store, and the owned volumes’ contents must be treated as compromised — inherent, since the agent is their owner. The compromise-relevant asymmetry holds throughout: the human gateway holds the authority but mounts no storage.

4. Adversary 4 — a compromised sandvmm (the VMM process)

sandvmm is the host process that emulates the guest’s devices, including the virtio-fs file server (in-process virtiofsd) — so it is the data path for whatever its VM sees. What a full RCE in it holds, and what stays out of reach:

  • Holds: the guest’s memory and console; the ability to serve any volume whose path it knows to its own guest (the allowed_dirs filter lives inside this process, and the daemon learns each path the VM is switched to). With the shared-group grant it can read/write those volumes directly on the host. It cannot enumerate the volumes root (2710, group execute-only) and cannot guess a name (sf- + 128-bit HMAC-SHA256(seed, id)); the seed is root:sandprov 0640 and every sandvmm spawn replaces its supplementary set with exactly microvm.vmm_groups, so the seed never reaches it. sandvmm is therefore a data-confidentiality TCB for the volumes it has served, not for the whole tree — which is why it is a separate identity from the warden-trusted sandlot and is spawned with zero capabilities: the TAP is created and pre-configured (owner set to the sandvmm uid, MTU/bridge/up) by the network provider, and the VMM only attaches to it by owner match.
  • Cannot reach the warden transport: the vsock data path is the kernel’s vhost-vsock module (the process never touches guest vsock frames), /dev/vsock for host-side sockets is sandlot-owned 0600 (§1), and vsockguard denies uid 987 the warden port even if it could. Guest processes inside its VM also cannot escalate through it to the host (guest-planted setuid files land owned by 987 — an identity with no warden access; §1 and Host Prerequisites).
  • Cannot cross VMs: no shared memory with other VMs, ptrace fenced by ptrace_scope, and vhost-vsock enforces CID uniqueness in-kernel — a second device cannot claim a live VM’s CID, so it cannot impersonate another guest at the vsock layer.
  • Cannot hold the host: zero capabilities (the TAP is created by the network provider), no DAC_OVERRIDE; its supplementary groups are exactly kvm + the shared storage group (it must serve volume I/O). File DAC still applies to everything outside the volumes it serves: the share root is traverse-only and sibling volume directories are reachable only by name, so the shared group does not become a cross-tenant read grant. Nor is it in sandtmpl (the template-refs metadata group — members sandprov + sandlot only): it reads overlay layer content through the shared group but cannot enumerate the template stores’ refs/, so the global template catalog stays out of a compromised VMM’s reach.

5. Trust roots and what each unlocks

The model names its trust roots — the components the rest of the system builds on, so nothing within the model can compromise them:

Trust root What its compromise unlocks Standing protection
Host root Everything on the node (all VMs’ memory, all identities) the KVM trust model — out of scope by definition; the identity split exists to make non-root compromise insufficient
etcd Cluster control: identity records (hash-only), sandfs records, ownership/placement keys — a writer can re-route and claim ownerships loopback/TLS client-cert endpoints; treat etcd as a production secret, not a shared service
Gateway human authority Mint/revoke agent keys, CRUD sandfs the perimeter model (§3): revocation is the response
Stream CA + certs Impersonate gateways or sandlots on the control fabric per-identity key material, 0600 root-scoped; rotation is redeploy
Guest kernel + initramfs (the warden) A boundary breach, not an in-sandbox event: an implant survives the tenant switch and reads every volume later attached to that VM — per-VM, never fleet-wide the pinned minimized build, the load/runtime posture, the capability/seccomp fence, and the reuse limits: Guest Kernel Hardening

6. The boundary this page does not re-derive

What happens inside the microVM — the exec profiles and seccomp deny-list, the warden’s self-hardening against same-uid attack, the pinned mount protocol and the host-CID heartbeat gate, the virtio-fs allowed_dirs filter, and the switch-clearing chain (processes, mounts, caches, kernel objects, sockets) that guarantees zero residue between two agents on one VM — is the subject of Isolation and Swap, and deliberately not restated here: this page assumes that boundary holds and asks what is left outside it. The guest kernel as a trust root — the minimized build, the allowed/denied syscall surface, the blast radius of a compromise, and the reuse limits that bound it — is Guest Kernel Hardening. The host-side invariants it leans on are asserted live by e2e/seccomp.sh, security/verify-host-security.sh, and security/vsockguard-verify.sh.

Two channels sit deliberately OUTSIDE the in-VM boundary and are recorded here as deployment duties:

  • Guest → host vsock (CID 2): the agent seccomp filter denies socket(AF_VSOCK) (EPERM), so sandswap guests cannot create a vsock socket at all; the warden’s own listener is created before the filter is installed. This is code-level blocking, not a deployment duty. If a future image ships without that rule, any host-side vsock service must still treat guest peers as unauthenticated-hostile (the warden’s own port applies the same rule to ITS peers — isolation-and-swap.md §7.3).
  • Guest console logs on the node: the VMM captures the guest kernel console into per-VM console.log/serial.log on the host. Guest code can induce kernel printk at some rate, so the sandlot enforces a size cap (monitor.go — truncate past 128 MB on a slow ticker); a node’s log partition must still be provisioned for the capped aggregate across its VM fleet.