Sandlot Configuration
Sources:
sandlot/cmd/main.go(CLI flags) +sandlot/config/config.go(YAML structure, defaults, reload classification) +sandlot/core/reload.go(reload application) +sandlot/core/runner.go(resilience reconnect state machine). Two entry points: CLI flags alone, or-config <yaml>plus flags. Only flags explicitly set on the command line override the YAML value (unset flags fall through to the file, then to built-in defaults).Hot-reload vs restart: when
-configis set, the file is watched (fsnotify, debounced) and hot-reloadable fields apply live; everything else is logged as “requires restart”. CLI flags always take effect at process start. See theclassifyFieldsummary at the bottom.The Gateway page documents the other end of the stream; the cross-component invariants (stream windows, 16 MB FileOp chain, exec-timeout ceiling) involve both.
Role applicability. Which sections matter for which -roles value. A knob outside an instance’s role list is accepted but inert there — and the refusals in §11 are the hard edges (a prov-only instance rejects exec machinery outright).
| Section | exec | prov | exec,prov (merged) |
|---|---|---|---|
| §1 Node identity / top level | ○ (per-row) | ○ (per-row) | ○ |
| §2 microVM defaults | ○ | — | ○ |
| §3 Specs and pool | ○ | — | ○ |
| §4 Recycle | ○ | — | ○ |
| §5 Resilience timings | ○ | ○ (no mounts to clean) | ○ |
| §6 Stream transport | ○ | ○ | ○ |
| §7 Network | ○ | — (prov refuses non-none) | ○ |
| §8 Egress binding + baseline | ○ | — | ○ |
| §9 TLS | ○ | ○ | ○ |
| §10 Go runtime GC pacing | ○ | ○ | ○ |
| §11 Roles + provision block | roles only | ○ (the whole provision.* block) | ○ |
| §12–§14 Logging / example / reload summary | ○ | ○ | ○ |
1. Node identity / top level — restart-required
Scope: all roles; the per-row Used by column marks the exec-only knobs (vmm_binary, vm_mode, boxout_socket, setfs_transport).
Background. A sandlot is one node: it owns the microVMs on a host, dials exactly one outbound gRPC stream to a gateway, and identifies itself in every heartbeat. node_id names the node in etcd records (ownership placement, presence), so it must be stable across restarts — the hostname default works on bare metal.
| CLI flag | YAML key | Default | Meaning | Used by |
|---|---|---|---|---|
-config | — | ”” | Path to the YAML config file (optional; enables the hot-reload watcher). | all |
-version | — | — | Print the version (version / commit / build time, ldflags-stamped by build/build.sh) and exit. | all |
-gateway-addr | gateway_addr | localhost:9998 | Gateway stream address (host:port or DNS name; production uses the stream DNS name, single-node uses localhost). | all |
-node-id | node_id | ”” (hostname) | Unique node identifier. Keys under /sandswap/node/{id}/ hang off it. | all |
-vmm-binary | vmm_binary | sandvmm | Path to the sandvmm binary. | exec |
-vm-mode | vm_mode | embedded | embedded = sandlot is privileged and manages sandvmm directly; boxout = sandlot is unprivileged (UID 65536, the reserved sandbox identity) and delegates privileged lifecycle ops to a Boxout sidecar over a Unix socket. | exec |
-boxout-socket | boxout_socket | /run/sandswap-boxout/boxout.sock | Unix socket path for boxout mode (must match boxout’s -socket). The default lives in boxout’s own root-owned 0711 directory, NOT the sandlot-writable /run/sandswap. | exec |
-boxout-token-file | boxout_token_file | ”” (UID-only admission) | File holding the per-pod admission token presented to boxout on every socket connection. Boxout verifies it in addition to SO_PEERCRED: the peer credential authenticates a UID, not the sandlot process, so without the token any process that merely shares the sandbox uid could drive root boxout. Must match boxout’s -client-token-file; set both sides in production boxout deployments. | exec |
-roles | roles | ["exec"] | Capability sets this instance serves: exec (microVM lifecycle/mount/exec), prov (sandfs + template lifecycle on shared storage — §11), or exec,prov (merged, single-node). Restart-required. One sandlot process per host (role-lock on /run/sandswap/sandlot.role; the refusal names the three blessed shapes). -allow-colocated-instances is the simulation-only escape hatch — a duplicate node_id is refused even with it. | all |
-setfs-transport | setfs_transport | raw | Transport for the sandvmm hot ops (vm.set-fs on every cold attach + lease-expiry clear, sandvmm.ping probes): raw = binary control frames on the same API socket (no HTTP/JSON framing — the fastest path; budgets aligned with the HTTP envelope, see protocols), http = the legacy REST form. Restart-required (fixed at process start before any VMM client exists). | exec |
| — | instance.id | ”” | Multi-instance suffix: run several sandlots on one host. Effective node id becomes <node_id>-<instance.id>, and per-VM state moves under instance.base_dir/<id>/vm. | all |
| — | instance.base_dir | /run/sandswap | Parent directory for per-instance VM state (used only when instance.id is set). | all |
| — | instance.cid_range_start | 3 | First guest vsock CID the instance may allocate (avoid overlap between instances). Minimum 3 (0/1/2 are reserved AF_VSOCK CIDs) — lower values are rejected by the config validation. | all |
| — | instance.cid_range_end | 0 (unbounded) | Last guest vsock CID; 0 = no upper bound. When set it must be > cid_range_start (an inverted range exhausts allocation immediately) — validated. | all |
2. microVM defaults — kernel/size hot-reload, paths restart-required
Scope: exec instances (a prov instance touches no VMM).
Background. These are the defaults for VMs created outside a named spec (pool pre-create uses spec entries instead; see §3). The kernel/initramfs/CPU/memory values hot-reload and affect newly created VMs only — running VMs keep what they booted with. The two directory paths shape host-side state and are restart-required.
| CLI flag | YAML key | Default | Reload | Meaning |
|---|---|---|---|---|
-kernel | microvm.kernel | /usr/local/lib/sandswap/guest/vmlinux.bin | hot | Path to the PVH kernel image every microVM boots. The default is the root-owned guest TCB directory (0755 root:root, outside every service identity’s write reach) — the kernel executes every in-VM clear, so it must never live in the service-writable state root. When both TCB files resolve under that directory, an exec-role start requires a root-owned SHA256SUMS file beside them and verifies both hashes before booting; a mismatch is fatal. Paths outside the directory (dev/e2e) log that the hash check was skipped. |
-initramfs | microvm.initramfs | /usr/local/lib/sandswap/guest/initramfs.cpio.gz | hot | Path to the initramfs containing the warden; same TCB directory and hash verification as -kernel. |
-default-vcpus | microvm.default_vcpus | 2 | hot | Default vCPUs. Minimum 1 — a negative CLI flag would silently wrap in the int→uint32 conversion and every VM creation would fail; validated at startup and on hot-reload. |
-default-memory | microvm.default_memory_mb | 512 | hot | Default memory (MiB). Minimum 64 (MinBootMemoryMB — a smaller VM cannot boot the kernel + warden); same wrap/validation note as vcpus. |
-volumes-dir | microvm.volumes_dir | /var/sandswap/volumes | restart | Host directory shared into every VM through the single virtio-fs device (tag=sandswap). Must be the same shared filesystem the gateway’s -vol-root points at. |
-allow-insecure-volumes | — | false | DEV/TEST ONLY: start even when -volumes-dir is not on a nosuid,nodev mount. The virtio-fs passthrough stores guest-planted setuid bits and mknod verbatim, and only those mount flags make them inert on the host (storage contract row 9, storage). Production fixes the mount instead — self-bind + remount,bind,nosuid,nodev, which the bare-metal deployer’s ss_harden_volumes_mount performs; e2e trees pass the flag deliberately. | exec |
-allow-missing-vsockguard | — | false | DEV/TEST ONLY: start an embedded exec node without a valid root-written vsockguard attestation. The sandlot service uid cannot read bpffs (/sys/fs/bpf is root-only), so security/vsockguard-attach.sh (root) writes /run/sandswap/vsockguard.allowed_uid after a successful attach and the systemd unit’s root ExecStartPre re-proves the live pin at every start; a non-root sandlot requires that file to be root-owned, non-writable and to name its own uid. A missing/mismatched attestation refuses unless this flag is set (loud WARN). Root-run sandlots skip the check (root is a trust root and the BPF hook admits root); boxout-mode sandlots skip it because the host-setup unit owns the guard. | exec |
-overlayfs-templates | microvm.overlayfs_templates | ”” (falls back to the sibling templates/ dir beside -volumes-dir) | restart | Overlay template cache directory for OverlayFS lazy mount. When empty, the exec role reads lowerdirs from the same sibling store the prov role writes — keep the two flags pointing at one place or set both explicitly. |
-warden-max-exec-timeout-sec | microvm.warden_max_exec_timeout_sec | 0 (warden default 180) | restart | Passed to the warden as warden.max_exec_timeout via the kernel cmdline. See Warden. |
| — | microvm.vmm_uid | 987 | restart | sandvmm identity uid (contract-pinned). The spawn wraps through setpriv: the VMM + FUSE passthrough daemon run as this uid with zero capabilities. The network provider creates the persistent TAP owned by this uid and pre-configures its MTU/bridge/up state, so the VMM attaches by owner match (TUNSETIFF) and skips the privileged MTU/enable ioctls. 0 (with vmm_gid) = no transition. |
| — | microvm.vmm_gid | 987 | restart | sandvmm primary gid; must be set together with vmm_uid. |
| — | microvm.vmm_groups | kvm,985 | restart | Supplementary groups for the sandvmm child — numeric gids or group NAMES (names resolve per host, e.g. kvm; numbers carry contract pins, e.g. 985 = the sandlot shared group for volume writes + the API-socket group grant). The spawn REPLACES the child’s supplementary set with exactly this list (setpriv --groups; --clear-groups when empty) — it never inherits the parent’s, so the merged exec,prov sandlot’s sandprov grant (for reading the volume-name seed) does not reach the VMM/FUSE passthrough daemon. |
| — | microvm.api_gid | 985 | restart | Group granted connect access to the sandvmm API sockets (0660 owner:group via --api-gid). Must be a group vmm_groups contains and one sandlot belongs to. Mandatory >0 when vmm_uid is set (validated). k8s/boxout: set to the pod’s shared group (fsGroup). |
| — | microvm.vmm_seccomp | true | restart | sandvmm --seccomp mode: true enforces the v28 per-thread allow-lists (Trap/SIGSYS), log installs them in audit mode (a missed rule is logged and allowed), false = no filter (the explicit escape hatch). process (KillProcess) is rejected: it installs empty per-thread filters. The default was promoted after e2e/vmm-seccomp.sh proved both phases green (log mode: zero RET_LOG hits across the battery; enforce mode: whole battery passes). A future VMM syscall that the allow-list misses traps that thread — re-run the gate before changing the lists. |
-vmm-api-auth | microvm.vmm_api_auth | true | Enable the per-VM sandvmm API token/challenge handshake (default ON: DAC + --api-allowed-uid alone put the API behind one uid boundary, shared by the FUSE passthrough daemon). Set false only for test tooling that drives the socket directly. When on, each VM gets a 32-byte random token delivered over an inherited fd, and every canonical-socket connection starts with a transport-independent challenge-response preface (it applies to both raw and HTTP framing). When off, the socket relies on DAC + --api-allowed-uid alone. Restart-required; in boxout mode set the YAML key (boxout reads the shared config). | exec |
-warden-log | microvm.warden_log | ”” (warden default info) | restart | Warden log level via the kernel cmdline. Deliberately NOT hot-reloadable: debug puts per-op timing lines on the guest console (hvc0 → console.log; the legacy UART cost ~83 µs/char), on the measured path — an online flip would enable that across every new VM at fleet scale, so the restart is the safety gate. A microvm.* hot-reload preserves the startup value (never drops it). |
-init-on-free | microvm.init_on_free | false | restart | Guest kernel free-time page wipe, written as an explicit init_on_free= token on every VM’s kernel cmdline (the effective cmdline is persisted in state.json, so the posture is auditable per VM). The guest kernel builds with INIT_ON_FREE_DEFAULT_ON, whose boot-time bulk clear of all guest RAM is redundant in a fresh microVM — the VMM’s guest RAM is host anonymous mmap (untouched pages are already host-zero) and no tenant has run — while the measured boot cost is linear in RAM (~0.65 ms/MB on nested virt; boot becomes RAM-independent with the override). init_on_alloc stays on either way (separate kernel static key) and keeps the cross-tenant “next allocation reads the previous tenant’s freed pages” channel closed. Set true only for deployments that require the stronger wipe (contents zeroed at free time as well — shorter lifetime after free, stale-mapping defense); that mode also keeps the per-free memset on exec teardown. Security-posture knob, restart-required like vmm_seccomp/vmm_api_auth; boxout mode reads the shared YAML. |
3. Specs and pool — pool hot-reload, specs restart-required
Scope: exec instances.
Background. A spec is a named VM shape; the pool pre-creates CLEAN VMs per spec so first exec on a node is warm. pool is a spec:count list parsed against the specs map; editing it live resizes the pool in both directions (scale-up creates, scale-down destroys idle excess; busy VMs are destroyed on their next unmount). Specs themselves are read at startup; a restarted sandlot also re-converges the live pool to the configured target.
| CLI flag | YAML key | Default | Meaning |
|---|---|---|---|
-pool | pool | ”” (no pool) | Comma-separated spec:count pairs, e.g. small:2,medium:3,large:1. Empty = no pre-created pool (VMs are created on demand). |
-vm-create-concurrency | concurrency.vm_create | 5 | Maximum microVMs booting at once during a pool scale-up, the 60s reconcile fill, or startup pre-create. Restart-required. |
-vm-destroy-concurrency | concurrency.vm_destroy | 10 | Maximum idle microVMs destroyed at once during a pool scale-down. Restart-required. |
| — | specs.<name>.vcpus | (below) | vCPUs for the spec. Minimum 1 — a zero-vCPU spec is rejected at parse time (both startup and hot-reload go through the parser) instead of failing later inside KVM with a confusing per-VM error. |
| — | specs.<name>.memory_mb | (below) | Memory (MiB) for the spec; minimum 64 (smaller specs are rejected at parse time). |
| — | specs.<name>.exec_slots | (below) | Per-VM sync-command concurrency = warden connection-pool size (one in-flight request per vsock connection). 0/omitted = 1 (serial — the memory-safe fallback for custom specs that don’t declare it). |
Built-in spec defaults (bare-metal sizing; nested-virtualization hosts override via -config):
| Spec | vcpus | memory_mb | exec_slots |
|---|---|---|---|
small | 2 | 256 | 3 |
medium | 4 | 1024 | 4 |
large | 8 | 2048 | 5 |
Hard limits enforced at startup and at hot-reload (a rejected reload keeps the previous pool target):
- Memory overcommit: total pool memory (
Σ count × memory_mb) must fit within 80% of hostMemTotal, or the node refuses to start / refuses the edit. - Boot floor:
memory_mb≥ 64 per spec. - A pool self-heal pass runs every 60s to refill gaps left by transient creation failures.
4. Recycle — hot-reload
Scope: exec instances.
Background. VMs are recycled (destroyed and replaced at the next unmount) when they wear out. max_uses bounds the cumulative commands served (execs and file ops, warm and cold, counted per command); the two memory thresholds catch guest memory fragmentation/leakage. The effective memory threshold is the lower of the enabled ones (percent decides on small VMs, absolute MiB on large ones), clamped to ≤ 50% of MemTotal.
| CLI flag | YAML key | Default | Unit | Meaning |
|---|---|---|---|---|
-max-uses | recycle.max_uses | 100000 | count | Destroy+recreate a VM after it serves this many commands (execs and file ops alike, warm and cold; -1 = unlimited, 0 = disable; any other negative is rejected at startup and on hot-reload). Default raised from 10000 (itself raised from 50): a freshly recycled VM’s first ops carry a cold-guest tax (and an occasional ~20 ms tail, ~1/160 cold ops) — at 100000 the recycle effectively only fires on the memory thresholds, and a node restart/redeploy replaces VMs anyway. |
| — | recycle.max_mounts | 10000 | count | Destroy+recreate a VM after this many cold mount generations (one per successful attach; warm ops do not count; -1 = unlimited, 0 = disable). |
| — | recycle.max_distinct_agents | 300 | count | Cap on distinct agents one VM may serve over its lifetime; a new agent on a VM at the cap is refused at selection (or the VM is recycled). -1 = unlimited, 0 = disable. |
| — | recycle.affinity_vms | 3 | count | How many recently-used VMs a sandfs prefers to reuse (best-effort in-memory affinity; 0 disables). |
-recycle-mem-percent | recycle.mem_percent | 10 | % | Recycle when guest MemAvailable < MemTotal × this / 100 (0 = disable). |
-recycle-mem-abs-mb | recycle.mem_abs_mb | 100 | MiB | Absolute recycle threshold of guest MemAvailable (0 = disable). |
-min-idle-floor-frac | recycle.min_idle_floor_frac | 0 (disabled) | fraction | Maintain per-spec CLEAN headroom of max(1, floor(frac × target)) on top of the pool target, so a spare CLEAN VM always absorbs a burst (dampens zero-clean load shedding). 0.05 = 5%. |
5. Resilience timings — restart-required
Scope: all roles — the values ride NodeHello.timing on every connection; the tolerance/cleanup semantics bite where mounts exist (exec instances; a prov instance holds no mounts).
Background. The tolerance-first model: when the stream to the gateway breaks, the sandlot first tolerates (mounts are kept so in-flight vsock commands finish), then cleans (releases mounts within a budget), then isolates (waits before reconnecting). The warden’s in-VM countdown is the last-resort backstop. These four values are sent to the gateway in NodeHello.timing; the gateway derives its presence-key TTL T_safe = T_tolerance + T_warden + 60s from them and rejects the connect if the ordering invariants below are violated. The phase is persisted across sandlot restarts (a restart never resets the tolerance clock).
| CLI flag | YAML key | Default | Meaning |
|---|---|---|---|
-tolerance-sec | resilience.tolerance_sec | 300 | T_tolerance: stream-outage grace before the sandlot starts cleaning mounts. During it, only the preferred gateway is re-dialed and ownership is untouched. |
-cleanup-sec | resilience.cleanup_sec | 300 | T_clean: time budget for the concurrent unmount sweep. Unmount failures and overruns are force-destroyed (destroy-and-replace). |
-warden-countdown-sec | resilience.warden_countdown_sec | 300 | T_warden: the warden’s single countdown, reset only by OP_MEM_STATS heartbeats; expiry with a live mount exits the warden (killing the VM). Passed to each VM via the kernel cmdline as warden.warden_countdown_sec. |
-isolation-sec | resilience.isolation_sec | 720 | T_isolation: wait after cleanup before reconnecting — past T_safe, so no new gateway can mount this node’s shared sandfs before every old mount is gone. |
-allow-short-resilience-timings | — (CLI only) | false | DEBUG/TEST ONLY: permits sub-300s T_tolerance/T_clean/T_warden. Short timings collapse the tolerance window and reintroduce the double-mount race the tolerance model eliminates — never set in production. |
Invariants (validated at startup; the gateway re-checks them at connect):
T_clean ≤ T_warden— cleanup must always finish before the countdown backstop fires atT_tolerance + T_warden(fatal at startup if violated).- 300s production floor: T_tolerance, T_clean, T_warden must each be ≥ 300s unless
-allow-short-resilience-timingsis set (fatal otherwise). T_isolation > T_safe = T_tolerance + T_warden + 60s.- Exec vs countdown bounds (fatal unless
-allow-short-resilience-timings):warden-countdown-secmust be strictly greater than the warden’s effective max exec timeout (-warden-max-exec-timeout-sec, or the warden default 180 when unset — the warden raises its countdown to it, which would invalidate the gateway’s T_safe proof), and greater thanwarden-max-exec-timeout-sec + 30s(the transport read margin: a max-length exec’s read racing the warden’s timeout response could let a retry re-execute a command that already ran). Under the escape flag the read-margin violation downgrades to a warning. - Hard ceilings (fatal):
-warden-max-exec-timeout-sec≤ 3600 s and-warden-countdown-sec≤ 4200 s — the warden refuses larger values at boot too.
6. Stream concurrency / transport — restart-required
Scope: all roles — exec streams carry VM commands, prov streams carry provision commands.
Background. All VM commands from the gateway are multiplexed over one bidi gRPC stream. These knobs size that pipe — how many in-flight commands it absorbs, how long a write may block (half-open TCP detection), and the HTTP/2 flow-control windows / max message size. Defaults serve 2000 concurrent commands and 16 MB FileOp payloads; dial options and the command semaphore are fixed at startup.
| CLI flag | YAML key | Default | Unit |
|---|---|---|---|
-stream-send-buffer | stream.send_buffer_size | 2000 | commands (sendCh capacity). Minimum 1 |
-stream-recv-buffer | stream.recv_buffer_size | 2000 | commands (recvCh capacity). Minimum 1 |
-max-concurrent-commands | stream.max_concurrent_commands | 2000 | commands (cmdSem; the (N+1)th blocks). Minimum 1 |
-stream-write-timeout-sec | stream.write_timeout_sec | 30 | seconds. Minimum 1 |
-stream-initial-window-mb | stream.initial_window | 64 | MB (HTTP/2 per-stream window; must be ≥ max message size). Minimum 1 |
-stream-conn-window-mb | stream.initial_conn_window | 128 | MB (HTTP/2 per-connection window). Minimum 1 |
-stream-max-msg-size-mb | stream.max_msg_size | 16 | MB (gRPC max send/recv message size). Minimum 16 (the FileOp chain floor) |
Deployment rule: gateway -stream-send-buffer ≥ sandlot -max-concurrent-commands (both 2000); keep the 16 MB cap aligned with the warden limits and the gateway’s stream_concurrency.max_msg_size (see Troubleshooting).
Local bounds are validated at startup (restart-required knobs — no hot-reload case) via the shared common/config.ValidateStreamCaps: every cap ≥ 1, windows within int32 range, initial_window ≥ max_msg_size, initial_conn_window ≥ initial_window, max_msg_size ≥ 16 MB.
7. Network — restart-required
Scope: exec instances — a prov-only instance refuses any non-none backend at startup (§11).
Background. Outbound guest networking for pooled microVMs. none keeps VMs offline; bridge attaches each VM to a host bridge with NAT; routed-ipv6 gives each VM a routed static IPv6 address (identity-in-address). All of it is built once at startup. In boxout mode sandlot does no networking itself — these keys are then read by the boxout sidecar instead (see Boxout).
| CLI flag | YAML key | Default | Meaning |
|---|---|---|---|
-network-backend | network.backend | none | none | bridge | routed-ipv6. |
-network-bridge | network.bridge | br-sandswap | Host bridge (bridge backend). Startup fails if the bridge is missing or net.ipv4.ip_forward is off — run deploy/setup-network.sh. |
-network-subnet | network.subnet | 10.188.0.0/16 | Guest subnet (CIDR, bridge backend). |
-network-gateway | network.gateway | 10.188.0.1 | Guest gateway / host bridge IP. Must be an IPv4 inside network.subnet, at its first host address (.1 by convention — the address allocator reserves exactly .0/.1); validated at provider construction, so a typo fails at startup instead of silently breaking every guest’s default route. This is guest-network plumbing, NOT how the sandlot finds the gateway service — that is gateway_addr (DNS name or IP both work). |
-network-dns | network.dns | 1.1.1.1,8.8.8.8 | Comma-separated guest DNS nameservers. |
-network-v6-prefix | network.v6_prefix | fd00:5341:4e44::/48 | Routed-ipv6 site prefix (multiple-of-8 length, ≤ /48; “SAND” in ASCII hex). |
-iid-template | network.iid_template | ”” (stamping disabled) | IPv6 IID layout for identity-in-address, e.g. "agent:32,epoch:32" or "deployment:16=42,agent:32,epoch:16" (identity fields are pinned to exactly 32 bits — a narrower agent/sandfs field is rejected at startup; deployment is the one operator-assigned field, its value written into the template and width-checked at startup). |
routed-ipv6requires a guest kernel withCONFIG_IPV6=y— the defaultkernel/build.shbuild includes it; only the dev-onlykernel/build.sh downloadvariant ships the official prebuilt without IPv6.
8. Egress identity binding + baseline — restart-required
Scope: exec instances.
Background. Two independent egress features share the network. block. The binding region publishes (tap-ifindex → agent/sandfs) at mount time for third-party network policy enforcers to read (see Egress Enabler Contract); off (default) leaves the hot path byte-identical to a build without it. The egress baseline is nftables anti-spoof + VM isolation (not policy) and is on by default.
| CLI flag | YAML key | Default | Meaning |
|---|---|---|---|
-binding-backend | network.bind_backend | off | Binding region carrier: off | shm (tmpfs file) | bpf (pinned mmapable BPF array). |
-require-egress-binding | network.require_egress_binding | false | Refuse to start an exec-role instance when no egress identity enabler is configured (Carrier A -binding-backend=shm\|bpf, or Carrier B -network-backend=routed-ipv6 + -iid-template). Unset = a loud startup WARN plus egress_bindings[].binding = "none" in /stats; production should set it. |
-binding-path | network.bind_path | ”” (per-backend default) | Region location override. Defaults: /dev/shm/sandswap-bindings (shm), /sys/fs/bpf/sandswap/bindings (bpf). |
-egress-baseline | network.egress_baseline | on | on | off. Requires nftables. Fail-closed: when nftables is missing or the baseline tables cannot be installed, the node refuses to start — set off explicitly to accept a per-VM-only deployment (logged as a startup WARN). |
-vm-to-vm-policy | network.vm_to_vm | deny | Guest-to-guest traffic posture: deny | allow (multi-agent systems may allow direct VM-to-VM traffic). |
9. TLS (gateway stream client) — restart-required
Scope: all roles.
Background. The sandlot is the TLS client of the one outbound stream to the gateway. Material is loaded once at startup and reused across reconnects; the SAN verified is the gateway address’s hostname (service name), even when a specific gateway IP is dialed. The client certificate’s CN must equal this instance’s node_id exactly — the gateway binds the stream hello’s declared node_id to the presented certificate’s CN and refuses a mismatch by name, so one shared pair across nodes does not connect (deploy/gen-certs.sh --client-cn <node-id>, deploy/baremetal/gen-baremetal-certs.sh with SANDLOT_NODE_ID=<effective node id>; k8s mints one Secret per node).
| CLI flag | YAML key | Default | Meaning |
|---|---|---|---|
-tls-cert | tls.cert | ”” | Client certificate for mTLS. Empty = plaintext (dev only). |
-tls-key | tls.key | ”” | Client private key (required if tls.cert is set). |
-tls-ca | tls.ca | ”” | CA bundle to verify the gateway server cert. Empty = skip verification. |
10. Go runtime GC pacing — restart-required
Scope: all roles.
Same mechanism as the gateway (gateway config §16): -gogc (0 = auto — GOGC env var wins if set, else 200; positive = force) and -gomemlimit (e.g. 512MiB; empty = GOMEMLIMIT env var wins, else no limit). The collector’s mark/assist cycles share the node’s cores with the VM command handlers and the VMM threads, so on busy nodes the default 200 halves GC frequency for lower tail latency. On hosts shared with many microVMs, set -gomemlimit to a fraction of the sandlot-side memory budget so VM RAM and the Go heap cannot collide unnoticed.
11. Roles (exec / prov) — restart-required
Scope: roles applies to every instance; the whole provision.* block is prov-only.
Background. A sandlot can serve two capability sets. exec is the classic per-node daemon (VM pool, mounts, exec). prov is the storage side: sandfs + template lifecycle on the shared volumes/templates trees — the work the gateway would otherwise do in-process. The gateway never mounts storage; it relays every sandfs/template API call to a prov instance over the stream.
| CLI flag | YAML key | Default | Meaning | Used by |
|---|---|---|---|---|
-roles | roles | ["exec"] | exec | prov | exec,prov (see §1). | all |
-templates-dir | provision.templates_dir | ”” (resolves to -overlayfs-templates when set, else the sibling templates/ dir beside -volumes-dir — same filesystem as the volumes tree) | The layers/ + refs/ template store root (shared with the exec role, which reads overlay lowerdirs from it). | prov |
-oci-registry | provision.oci_registry | ”” | Registry for pull/push/auto-pull. The egress lives on the PROV host — gateway hosts never reach a registry. | prov |
-registry-username | provision.registry_username | ”” | Registry basic-auth username (requires oci_registry). | prov |
-registry-password | provision.registry_password | ”” | Registry basic-auth password. Prefer the YAML key over the flag: a flag value is visible in /proc/<pid>/cmdline. | prov |
-registry-token | provision.registry_token | ”” | Registry bearer token; alternative to username/password (basic wins when both are set). | prov |
-registry-cert | provision.registry_cert | ”” | TLS client certificate file (mTLS); must be paired with registry_key. | prov |
-registry-key | provision.registry_key | ”” | TLS client key file (mTLS); must be paired with registry_cert. | prov |
-registry-ca | provision.registry_ca | ”” | CA bundle for server verification against a private CA; combines with any credential (basic + private CA is the common Harbor shape). | prov |
-flat-templates | provision.flat_templates_dir | ”” | Startup scan dir for local flat templates (<dir>/<name>/<version>/ → ref <name>:<version>). | prov |
-max-provision-ops | provision.max_concurrent_ops | 2 | Concurrent provision commands (GB-scale work must not starvation the host). | prov |
-provision-deadline-sec | provision.default_deadline_sec | 300 | Per-op deadline when the gateway sends none (auto-pulls are cancelled at it). | prov |
-provision-reconnect-sec | provision.reconnect_sec | 300 | PROV connection recycle age (±20% jitter) for spread rebalance across gateway addresses. Exec connections are sticky and NEVER recycled. | prov |
-volume-seed-next-file | provision.volume_seed_next_file | ”” | Flag form of the staged next seed (rotation step 1) — see the YAML key’s row below. | prov |
-volume-seed-file | provision.volume_seed_file | — (REQUIRED for prov) | File holding the volume-name seed (hex or base64, decoded ≥ 32 bytes). Every sandfs directory name is sf- + base32(HMAC-SHA256(seed, id)) — see isolation §4.6. Prefer the YAML key: the flag value (a path) is visible in /proc/<pid>/cmdline; the seed VALUE must never be a flag or a ConfigMap entry. Scope the file like the prov config (root:sandprov 0640): the prov identity reads it at startup AND on every reload, while the sandvmm identity must never be able to (a leaked seed makes every name guessable again); security/verify-host-security.sh FAILS when a sandvmm process carries the seed’s group. Bare metal: minted once by sandswap_ensure_volume_seed; k8s: the sandswap-volume-seed Secret, created once by deploy-lib and never regenerated. | prov |
| — | provision.volume_seed | ”” | Inline seed value (same validation). Mutually exclusive with volume_seed_file — both set is refused at load. Prefer the file form. | prov |
| — | provision.volume_seed_next_file | ”” | Stages the next seed generation (rotation window step 1): with it set, the prov instance refuses NEW volume creation (ErrRotationStaged — a volume created mid-window would strand in the old generation) while existing volumes keep resolving (current name first, staged name as fallback). Managed by rotate-volume-seed.sh --stage; cleared by --promote. | prov |
| — | provision.volume_seed_next | ”” | Inline staged seed; same relation to volume_seed_next_file as volume_seed to volume_seed_file. | prov |
-allow-seed-generation-mismatch | — (CLI only) | false | Start even when the volumes root carries a naming generation NO configured seed derives — deliberate re-key / disaster recovery only, paired with the one-time migration (see Volume seed); volumes stay unresolvable until that migration runs. Does not weaken the promote completeness gate. | prov |
| — | provision.seed_poll_sec | 30 (0 = default) | How often (seconds) the prov instance re-reads its seed sources. fsnotify covers local writes; the poll is what still notices a change on filesystems where inotify never fires. Restart-required. | prov |
| — | provision.quota_stamp_cmd | ”” (disabled) | Storage plane’s quota hook, executed as <cmd> <sandfs-id> with the volume path on stdin (one line; never argv — /proc/<pid>/cmdline is world-readable and the path is the seed-derived tenant name) once per freshly seeded volume (empty = no-op; non-empty must be an absolute path; 10s deadline). Exit 0 = stamped; failure refuses the create fail-closed (the provisioning marker stays, so a retry self-heals). The concrete implementation belongs to the storage operator — see the contract in sandlot/provision/volume/quota.go and storage contract row 13. Restart-required. | prov |
| — | provision.max_sparse_bytes | 8589934592 (8 GiB) | Cap on one file’s APPARENT size when SaveAsTemplate copies a tenant volume into a template layer. A sparse file’s hole is materialized as zeros host-side, so without a cap one file-API write at a huge offset could fill shared storage; the copy also refuses to write more than the destination filesystem’s free space minus a reserve. 0 = built-in default. Restart-required. | prov |
The four volume_seed* keys are the ONLY provision.* keys that hot-reload — staging and promotion are config edits by design (the rotation window must not restart the prov instance; see deploy/baremetal/rotate-volume-seed.sh). The reload applies a strict transition table (stage / abort / promote); anything else — a current seed changing without a staged promotion, a current seed disappearing, editing while staged — is refused by name and the live namer keeps its previous state. Seed values are redacted ((redacted)) in the reload log; the observability surface is a one-way 8-hex fingerprint of the generation.
Validation matrix: a prov-only instance refuses exec machinery (a pool, vm_mode: boxout, or a non-none network backend) at startup with a named error; an exec,prov instance is only meaningful against a UNIFIED gateway (single-node / one-gw topologies). Placement, presence, and sandns never see prov sandlot instances — they carry zero etcd footprint. Registry credentials are refused at startup when oci_registry is empty (nothing to authenticate to) or when registry_cert/registry_key are not set together. Every other provision.* key — credentials included — is restart-required: the registry client is built once at startup, and a credential edit is redacted ((redacted)) in the reload log.
12. Logging
Scope: all roles.
| CLI flag | Default | Meaning |
|---|---|---|
-debug | false | Per-op success-path logging (mount/exec/unmount/file_op timings, VM boot/ready). Off by default: a busy node performs thousands of ops/sec and these lines flood the log AND sit on the response path (unbuffered stdout write before the handler returns). Error/warning lines are always logged. |
-warden-log | ”” (warden default info) | In-VM warden log level passed via kernel cmdline; also settable (restart-required) as microvm.warden_log — see §2. Set debug for profiling sessions — per-op hot-path timing lines live at debug because their console drain taxes the measured path. |
13. Worked example: sandlot.yaml
Scope: merged-shape example; the comments mark the prov-only and exec-only parts.
A representative embedded-mode config (the bare-metal deployer generates this shape — see Install (bare metal)):
gateway_addr: "sandswap-stream.local:9998" # DNS name in production;
# localhost:9998 single-node
node_id: "sandlot-X"
vm_mode: embedded
vmm_binary: /usr/local/bin/sandswap-sandvmm
# Merged roles (the single-node / all-in-one shape): this instance is both the
# exec node and the storage side. Compute hosts use roles: [exec]; dedicated
# storage hosts use roles: [prov] + the provision block below.
roles: [exec, prov]
provision:
templates_dir: /var/sandswap/templates # layers/ + refs/ store —
# default (unset): sibling templates/
# beside volumes_dir, same filesystem
flat_templates_dir: /var/sandswap/flat-templates
# Volume-name seed (REQUIRED on prov): derives every sandfs directory name.
# File-scoped secret, never inline in a shared file and never a ConfigMap
# value; minted once by the deploy tooling. Rotation stages a second key:
# volume_seed_next_file: /etc/sandswap/volume-seed-next
volume_seed_file: /etc/sandswap/volume-seed
# oci_registry: registry.example.com # egress lives on THIS host
# Credentials for a private registry (only with oci_registry; this file
# must then be root-owned 0640 — the deploy tooling scopes it via
# ss_protect_config):
# registry_username: robot
# registry_password: "…"
# registry_ca: /etc/sandswap/certs/registry-ca.pem # private CA
# registry_cert: /etc/sandswap/certs/registry-client.crt # mTLS (with key)
# registry_key: /etc/sandswap/certs/registry-client.key
microvm:
# The deploy tooling installs the TCB pair as vmlinux.bin /
# initramfs.cpio.gz in this root-only directory (with SHA256SUMS) and pins
# the paths here. Omit both keys to use the sandlot defaults, which name
# defaults name .../guest/vmlinux.bin (the deployed artifact's name) — see §2.
kernel: /usr/local/lib/sandswap/guest/vmlinux.bin
initramfs: /usr/local/lib/sandswap/guest/initramfs.cpio.gz
default_vcpus: 2
default_memory_mb: 512
volumes_dir: /mnt/sandswap/volumes
# warden_max_exec_timeout_sec: 0 # 0 = warden default 180; restart-required
# warden_log: "" # "" = warden default info; restart-required (debug taxes the guest console)
# init_on_free: false # guest free-wipe posture; false boots with init_on_free=0
# (fast, RAM-independent boot; init_on_alloc stays on). true = the
# stronger wipe for high-security deployments; restart-required
specs:
small: { vcpus: 2, memory_mb: 256, exec_slots: 3 }
medium: { vcpus: 4, memory_mb: 1024, exec_slots: 4 }
large: { vcpus: 8, memory_mb: 2048, exec_slots: 5 }
pool: "medium:5"
recycle:
# max_uses omitted = default 100000 (recycle effectively fires on the memory
# thresholds only; a low use-count makes every recycled VM's first ops pay
# the cold-guest tax)
mem_percent: 10
mem_abs_mb: 100
network:
backend: bridge
bridge: br-sandswap
subnet: 10.188.X.0/24
gateway: 10.188.X.1
dns: "1.1.1.1,8.8.8.8"
tls:
cert: /etc/sandswap/certs/sandlot.crt
key: /etc/sandswap/certs/sandlot.key
ca: /etc/sandswap/certs/ca.crt
14. Hot-reload vs restart-required summary
Scope: all roles.
- Hot-reload (YAML edit applies live):
microvm.kernel,microvm.initramfs,microvm.default_vcpus,microvm.default_memory_mb(new VMs only; the reload is a partial application —microvm.volumes_dir/overlayfs_templatesand the warden cmdline fields keep their startup values),pool(resize both directions, overcommit-checked),recycle.*,provision.volume_seed*(the rotation window — stage/abort/promote transition table enforced) - Restart-required (logged as a warning, applied on next start): everything else — top level (
gateway_addr/node_id/vmm_binary/vm_mode/boxout_socket),instance.*,specs,microvm.volumes_dir/microvm.overlayfs_templates,microvm.warden_max_exec_timeout_sec/microvm.warden_log(warden debug logging is deliberately restart-gated: an online flip would tax the guest console on the measured path at fleet scale),microvm.init_on_free(security-posture knob, like the other posture keys),stream.*,resilience.*,network.*(incl. binding/baseline),tls.*,provision.*(the registry, its credentials, and the store dirs are all read once at startup)