Observability
What Sandswap exposes for monitoring and debugging, and how to read it. Scraping /metrics with Prometheus is covered in Prometheus Integration (the metric catalogue lives there); this page is the field view: the JSON stats, the per-request latency fields, and where each component logs.
1. Collection surfaces
| Surface | Location | Precision | Content |
|---|---|---|---|
| execute response fields | POST /agent/execute response body | ns | gateway_duration_ns (gateway E2E), sandlot_duration_ns (dispatch→response, incl. set-fs + vsock RTT), exec_duration_ns (warden share: spawn→wait warm / mount+exec cold) |
| lease classification header | X-Sandbox-Lease: hit on the same response | per request | hit = warm (sticky lease); absent = cold (cross-agent cold mount / first call) |
| unmount statistics | GET /stats → per-node unmount_stats | ms (integer-truncated) | unmount_count/success/fail, under_5ms/under_10ms, top_slow[10] (with gateway_ms/sandlot_ms/warden_ms/command/reason three-layer breakdown) |
| MCP tool results | the execute tool result | ns | same three *_duration_ns fields as the HTTP response |
| Prometheus | /metrics | scrape-time | the same stats projected to text — catalogue in Prometheus Integration |
All timings come from monotonic clocks (Go time.Since, Rust Instant); no wall-clock drift. Logging overhead on measured paths is bounded by design — per-op success logs are debug-gated in all three processes (below), so a production default log costs < 0.01 ms per request.
2. Derived latency relations
gRPC + gateway processing ≈ gateway_duration_ns − sandlot_duration_nsvsock RTT + set-fs + sandlot processing ≈ sandlot_duration_ns − exec_duration_ns
Finer segments than these are only visible in component logs.
3. Reading /stats
GET /stats (human listener in single-node/human mode; agent listener in exec mode, where it is -ops-token-file-gated and disabled without the flag) serves the full cluster report: per-gateway health (healthy, stale_sec, last_healthy_ts), node inventory with pool state and VM counts, duty/load signals, the per-node unmount_stats block, and exec-slot signals (peak_concurrent_execs, slot_wait_ms, slot_wait_events, recycle_triggers). The same report, field by field, is what Troubleshooting walks through — go there for “what value means what”.
4. Log gating conventions
Per-op success lines are debug-gated everywhere, checked before any formatting (slog args and fmt.Sprintf are eager — the gate wraps the call):
| Component | Default output | Enable debug |
|---|---|---|
| gateway | errors/warnings + sparse lifecycle events | -log-level debug |
| sandlot | errors/warnings + sparse lifecycle events | -debug |
| warden | errors/warnings + lifecycle events | config log_level / SANDSWAP_WARDEN_LOG_LEVEL; debug! is a compile-time no-op without the slog max_level_debug feature |
The reason for the gate: per-op lines tax the cold path through the slog-async drain (the legacy serial UART cost ~83 µs/char; warden userspace now uses hvc0 → console.log), and at scale they flood the log.
5. Per-segment log lines (debug on)
| Segment | Log source | Example line |
|---|---|---|
| Gateway E2E + warm/cold | gateway.log (JSON) | "msg":"execute complete","leaseHit":true,"latency":"3.4ms" |
| set-fs (allowed_dirs hot-swap) | sandlot.log | mount-rootfs: set-fs allowed_dirs=[…] took 1.2ms |
| vsock RT (cold/warm path) | sandlot.log | mount-rootfs: cold mount_and_exec took 2.1ms / mount-rootfs: warm exec took 0.8ms |
| warden unmount | sandlot.log | warden unmount /mnt/rootfs took …ms |
| warden exec / mount | warden console log (console.log) | exec done in 3ms (spawn=1ms wait=1ms exit=0), mount_and_exec done in 19ms (mount=7ms exit=0) |
6. Verifying data integrity
# Read response fields directly
curl -s -X POST "$GATEWAY_URL/agent/execute" -H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{"sandfs_id":"...","code":"echo hi","language":"shell"}' \
| python3 -m json.tool | grep duration
# Unmount statistics (human listener)
curl -s http://localhost:3100/stats | python3 -m json.tool | grep -A8 unmount_stats
# Component log samples (debug enabled)
grep '"execute complete"' /tmp/sandswap-logs/gateway.log | tail -5
grep "took" /tmp/sandswap-logs/sandlot.log | tail -8
Collection guarantees: unmount statistics are booked by the gateway at mount/unmount completion (unmount_count + 1 means the unmount finished and its three-layer latencies are recorded — the benchmark iterates strictly serially on that); top_slow keeps the slowest 10 per node, so multi-scenario benchmarks snapshot the diff at each scenario boundary and samples do not pollute each other.