Boxout
1. Overview
Boxout is the privileged sidecar for Kubernetes deployments: it takes over everything privileged so the sandlot container can run unprivileged (UID 65536 — the reserved sandbox identity, all capabilities dropped). It is a posture choice, not a default. Source: boxout/ (Go, one binary). See Topologies.
2. Key Responsibilities
- sandvmm lifecycle for the pod’s VMs: launch, kill, shutdown.
- Network provisioning: build the network provider (bridge / routed-ipv6) here instead of in sandlot — TAP/IP setup and the static egress baseline.
- Egress identity stamping: apply and remove the routed-IPv6 identity SNAT stamps on the sidecar side (netlink needs
CAP_NET_ADMIN). - Host-side OverlayFS: mount/unmount, tracking, and the periodic stale-overlay sweep — the same
core.OverlaySetimplementation the embedded mode uses. - BPF binding-region handoff: create and pin the egress identity region, then chown it to the unprivileged sandlot (
0660). - /proc re-adopt scanning and crash detection for surviving VMs — parity with embedded mode.
3. Interfaces / Data Flow
Inputs:
- Newline-delimited JSON requests from sandlot over
/run/sandswap-boxout/boxout.sock. The directory is boxout-private (root-owned0711, separate from the sandlot-writable/run/sandswap); SO_PEERCRED rejects any UID except-allowed-uid; the socket is0600, ownerallowed-uid, secured through directory-fdFchownat/Fchmodatwith an inode re-check. With-client-token-fileset, a per-connection HMAC-SHA256 challenge-response (fresh nonces both directions, token never on the wire) runs before any request is read. - The shared sandlot YAML: boxout watches it and hot-applies the
microvm.*subset, so its create_vm parameter whitelist tracks the sandlot’s live values. - Its own flags (socket path, allowed UID).
Outputs:
- sandvmm processes spawned/killed on the host.
- Kernel state: TAP devices, addresses, nftables rules, identity SNAT stamps, OverlayFS mounts, the pinned bpffs region.
- JSON replies on the socket.
Sandlot (UID 65536) Boxout (UID 0) Host
│ {op:"launch_vm",...} │ │
│─────────────────────────▶│ verify peer UID │
│ │ check param whitelist │
│ │ spawn sandvmm ──────────▶ KVM process
│ {ok} │ │
│◀─────────────────────────│ │
│ {op:"mount_overlay",...} │ │
│─────────────────────────▶│ mount on host ──────────▶ OverlayFS
│ {ok} │ │
│◀─────────────────────────│ │
4. How It Processes
- Start as a privileged sidecar in the sandlot pod. Create/verify the private socket directory, bind
/run/sandswap-boxout/boxout.sock, enforce the entry’s owner/mode/inode, load the shared YAML. - Verify every request: read the peer’s UID via SO_PEERCRED, reject anything but the configured sandlot UID.
- Whitelist parameters: only the
microvm.*values the shared YAML allows pass through — the caller cannot smuggle arbitrary kernel paths, memory sizes, binaries, or guest addresses. Pre-configured kernel/initramfs only; vcpus/memory/CID are range- and integrality-checked; sandbox dirs are pinned to<instance root>/<vm_id>; volume paths are opened withopenat2(RESOLVE_BENEATH|RESOLVE_NO_SYMLINKS)and must be direct children of the volumes root; network release/reclaim require the caller’s own lease and re-derive the spec from persisted state. - Execute the op: spawn/kill/shutdown sandvmm, provision or tear down networking, mount/unmount overlays, apply/remove identity stamps, create the bpf region, or scan
/procfor re-adoption. - Reply with a bounded JSON result. A wedged sidecar never blocks a mount or unmount: the stamp path degrades to unstamped egress (see Egress Enabler Contract §5).
The hot path stays in sandlot: vm.set-fs, the warden vsock exec, and leases need zero privileges and never cross this socket. Everything boxout does is privileged management-path work — off the response path.
5. State & Storage
Boxout is effectively stateless. It tracks host-side OverlayFS mounts in memory for the stale-overlay sweep (overlay-only, capped, persisted only on change to <private state root>/boxout-overlays.json); the durable state it manages lives in the kernel (TAPs, mounts, bpffs pins) and is re-derived after a sidecar restart by the pidfd//proc re-adopt scan and the overlay re-scan. Its instance lock and token file live in a private root-only state directory. Configuration comes from the shared sandlot YAML.
Component configuration: Boxout configuration.