Sandns Configuration
Source:
dns/config.go(YAML keys, defaults, validation) +dns/main.go(CLI, reload, signals).sandns is a YAML-only component:
-config <yaml>(required) is the only meaningful flag (-versionprints the version stamp and exits). There is no static gateway list — every gateway self-registers its routing addresses into/sandswap/gateway/{id}at startup, and sandns builds its candidate set entirely from etcd (registry = candidacy, health = liveness, load = scoring). Before any gateway has registered, stream/agent queries SERVFAIL and the sandfs zone falls back to its no-owner branch.
The design this configures (owner-aware per-sandfs zone, client-IP stickiness, scheduling policy) is described in Ownership & DNS.
1. Listener / etcd / forwarding — restart-required
Background. sandns serves UDP+TCP on one address, keeps its state (sticky mappings) in etcd, and forwards everything outside the Sandswap zones to upstream resolvers. These keys are read once at startup and are not re-read by the reload watcher.
| CLI flag | YAML key | Default | Meaning |
|---|---|---|---|
| — (required) | listen_addr | 0.0.0.0:53 | DNS listen address (UDP + TCP). Port 53 needs CAP_NET_BIND_SERVICE or a non-privileged port. When the address is a wildcard, dedicated loopback UDP listeners are bound on the same port (reported as a warning if they cannot bind) — they carry the ingress signal the recursive ACL trusts, and their loss fails host-local recursion closed, not open. |
| — | etcd | ”” (required) | etcd endpoints (comma-separated, e.g. https://etcd1:2379,...). Gateway candidates, sticky state, and ownership all come from here. |
| — | etcd_ca | ”” | CA bundle for etcd TLS. |
| — | etcd_cert | ”” | Client certificate for etcd mTLS. |
| — | etcd_key | ”” | Client private key matching etcd_cert. |
| — | forwarders | [] | Upstream resolvers (host:port) for non-Sandswap names. Empty = no forwarding (unresolved names fail). A forwarder that would re-enter this resolver — the same port on an address the configured listener actually receives on (a wildcard listener: every local address; an IP-literal listener: that exact address; localhost: loopback) — is refused at startup: the query would re-enter the process and each in-flight forward would hold a slot until timeout. A local stub resolver on a different port, or on a different specific address the listener does not bind, is fine. |
| — | guest_cidrs | [] | Sandbox guest source ranges. These sources are not control-plane consumers: every <sandfs_id>.<sandfs> query from one is answered NXDOMAIN (the answer reveals neither existence nor ownership — DNS carries no ownership proof), the stream/agent zones are REFUSED, and foreign zones are REFUSED instead of relayed (the resolver must not double as an open recursive resolver behind the host’s egress identity). Set the sandbox subnet (10.188.0.0/16 by default); empty disables the gate (dev) and logs a startup warning. |
| — | recursive_acl | ["127.0.0.1/32","::1/128"] | Non-guest source ranges allowed to relay foreign zones. A loopback entry counts only when the packet also arrived over a loopback-address socket: sandns binds dedicated loopback listeners alongside a wildcard listen_addr, enables the IP_PKTINFO / IPV6_RECVPKTINFO socket options on them and requires the datagram’s ingress interface to be lo (TCP uses the accepted connection’s local address). net.ipv4.route_localnet=1 source spoofing — and a packet addressed 127.0.0.1 that actually arrived on another interface — therefore cannot claim the host’s recursion; if the control message cannot be enabled the socket binding remains the signal (logged as a warning), never the claimed source address. Non-loopback entries match by source address (UDP DNS has no stronger client identity). Empty ACL = loopback-only (defensive default). |
2. Domains — restart-required
Background. sandns serves up to three zones with different routing semantics: the stream domain places sandlots onto gateways; the per-sandfs zone is owner-aware (reads /sandswap/owner/{id} then /sandswap/health/{gw}); the agent domain resolves to one healthy exec gateway for the GET /agent/sandfs bootstrap discovery call.
| YAML key | Default | Meaning |
|---|---|---|
domains.stream | ”” (required) | Sandlot stream domain: resolves to the gateway a sandlot should dial (placement). Validation fails without it. |
domains.sandfs | ”” (zone disabled) | Per-sandfs owner-aware zone: <sandfs_id>.<domain> resolves to its OWNER gateway (CNAME for hostname targets, A for IPs); falls back to staleness-guarded/presence-directed routing or scheduling when there is no fresh owner. |
domains.agent | ”” | Shared agent domain (e.g. sandswap.local) — the bootstrap discovery entry: resolves to one healthy exec gateway for GET /agent/sandfs (any exec gateway answers from the cluster-wide etcd identity store; the call carries no sandfs_id so no per-sandfs domain can serve it); also the sandctl dns agent diagnostics target. |
Routing gates: the sandfs-domain fallback additionally requires
node_count > 0andclean_vms > 0(a gateway with no connected sandlots is in a failure state and must not receive new sandfs placements); the stream and agent-discovery domains do NOT gate on compute — a compute-less gateway is still a valid sandlot target (the sandlot connecting is what brings compute back) and a valid discovery target (the identity list is read from etcd regardless of sandlots).
3. TTL / sticky records — restart-required
Background. Client-IP stickiness maps a resolver’s source IP to the gateway it was first routed to, so an agent’s repeated connections keep landing on the gateway that holds its warm VMs. Sticky mappings persist to etcd (survive sandns restarts) and expire when unused.
| YAML key | Default | Meaning |
|---|---|---|
ttl | 10 | DNS TTL (seconds) on served answers. Valid range [1, 86400]. |
sticky_expire | 1800 | Sticky records expire after this many seconds without a query. |
sticky_sync_interval | 5 | Seconds between sticky → etcd sync passes. Must be ≥ 1 (0 would panic the sync ticker). |
sticky_sync_batch | 256 | MAXIMUM sticky entries flushed to etcd per sync pass (0 = unlimited). Each pass then fits inside its etcd timeout regardless of dirty-set size; the remainder is picked up by the next pass. |
4. Scheduling (gateway routing policy) — hot-reload
Background. How sandns picks a gateway for UNSOWNED sandfs and for stream-domain placement (owned sandfs always route to their owner — affinity, orthogonal to policy). binpack packs new work onto the busiest healthy gateway so lightly-loaded ones go idle (drain/scale-away); spread sends it to the most idle gateway. The duty cap excludes saturated gateways in BOTH policies, so binpack can never pile onto an overloaded gateway. Changes apply live via the config-file watcher.
| YAML key | Default | Meaning |
|---|---|---|
scheduling.policy | binpack | binpack | spread. Any value other than spread normalizes to binpack. |
scheduling.duty_hard | 0.85 | Gateway-level agg_duty_long hard cap (0–1): a gateway at or above it is excluded from routing for new sandfs in both policies. 0 = disabled. |
5. Staleness (owner death threshold) — hot-reload
Background. sandns is a separate process, so the gateway’s -route-stale-sec flag does not reach it — it has its own threshold with the same semantics: staleness past which the owner branch stops routing to an owner (falls back to presence-directed routing or scheduling). Judging wrong here is cheap (a napping owner serves the request fine once it wakes), so this is the fastest death threshold. A live owner refreshes its health key every 5s and is never stale at 20s.
| YAML key | Default | Meaning |
|---|---|---|
staleness.route_stale_sec | 20 | Seconds since an owner gateway’s last_healthy_ts past which it is treated as dead for routing. 0 = keep the default; negative values are rejected at load. Keep aligned with (≥) the gateway’s -route-stale-sec — see Gateway §12. |
6. Operational sizing — restart-required
Background. The serving paths read these as plain values and the semaphores are constructed once at startup, so they are restart-required by design. Defaults are chosen to stay reasonable at large fleets (the owner cache holds a million-sandfs working set in ~300 MB at ~250 B per entry; smaller fleets only ever hold what they query). All bounds below are enforced at load — an out-of-range value fails startup.
| YAML key | Default | Valid range | Meaning |
|---|---|---|---|
owners.cache_max | 1,000,000 | ≥ 1000 | Owner-cache entry bound (per-sandfs state). Must hold the hot working set or the hit rate collapses into an etcd read storm. |
owners.cache_ttl_sec | 30 | [1, 3600] | How stale a positive owner entry may be before revalidation. Larger cuts the etcd read rate proportionally at the cost of slower ownership-change visibility. |
owners.neg_cache_ttl_sec | 1 | [1, 300] | Negative (no-owner) entry TTL. Short so a real first-claim is picked up promptly. |
owners.etcd_timeout_ms | 2000 | [100, 60000] | Bound on one on-demand owner etcd Get. |
owners.get_concurrency | 256 | [0, 100000], 0 = unbounded | Concurrent owner etcd Gets on the query path; past the cap lookups fail fast (SERVFAIL/stale) instead of queueing. |
owners.rate_limit_per_source | 100 | [0, 1000000], 0 = unbounded | Per-client-IP owner-lookup rate cap (lookups/s). A random-subdomain flood from one source is refused at this bound instead of reaching etcd; the response is a normal miss, not an error. |
owners.rate_limit_global | 50000 | [0, 10000000], 0 = unbounded | Fleet-wide owner-lookup rate cap (lookups/s) across all sources; the backstop when the per-source cap is disabled or the flood is distributed. |
sticky_max | 65536 | ≥ 1 | Client-IP sticky table bound (in-memory). |
max_concurrent_queries | 2048 | ≥ 16 | Overall in-flight query bound (semaphore). |
forward_concurrency | 128 | ≥ 0, 0 = unbounded | Concurrent upstream forwards. |
tcp_max_conns | 256 | ≥ 1 | Concurrent TCP DNS connections. |
7. Hot-reload and signals
- Hot-reload: the config file is watched (fsnotify, debounced — same
common/configReloadWatcher sandlot uses). Onlyscheduling.*andstaleness.*re-apply live; invalid values on reload (e.g.duty_hardoutside [0,1]) are logged and ignored, keeping the previous settings. Everything else (listen_addr,domains.*,etcd*,ttl,sticky_*,forwarders, the §6 sizing knobs) requires a restart. Kubernetes ConfigMap mounts work too: the watcher detects the symlinked..datalayout and reacts to kubelet’s atomic updates. - Strict parsing: unknown YAML keys fail at startup (an obsolete
gateways:section is rejected, never accepted silently). Remove obsolete keys when upgrading. - SIGHUP: clears all sticky records (
ClearAllSticky) — the ops escape hatch when every client should be re-placed (e.g. after a topology change). Does not reload config and does not stop the process. - SIGINT/SIGTERM: shutdown.
8. Worked example: sandns.yaml
The production shape from Install (bare metal):
listen_addr: "0.0.0.0:53"
domains:
agent: sandswap.local # agent discovery bootstrap (GET /agent/sandfs; any exec GW answers)
stream: sandswap-stream.local # sandlot stream domain
sandfs: sandfs.sandswap.local # per-sandfs owner-aware zone:
# <sandfs_id>.sandfs.sandswap.local → owner GW A record
# (candidates come entirely from etcd self-registration —
# no static gateway list)
etcd: "https://etcd1:2379,https://etcd2:2379,https://etcd3:2379"
etcd_ca: /etc/sandswap/certs/ca.crt
etcd_cert: /etc/sandswap/certs/etcd-client.crt
etcd_key: /etc/sandswap/certs/etcd-client.key
ttl: 10 # DNS TTL (seconds)
sticky_expire: 1800 # sticky records expire after 30min without queries
sticky_sync_interval: 5 # sticky → etcd sync interval (seconds)
sticky_sync_batch: 256 # max sticky entries flushed per sync pass
forwarders:
- 8.8.8.8:53
- 8.8.4.4:53
(The forwarders block, plus scheduling:/staleness: if you deviate from the defaults, is where you extend it.)