Scaling, Slots & Pool Manager

Every sx invocation schedules cooperatively against shared local lockfiles (~/.local/state/spotexec/) and cloud instance tags—requiring zero central server or cloud coordinator.

  sx (worktree 1)    sx (worktree 2)    sx (worktree 3)    sx (worktree 4)
         │                  │                  │                  │
         └──────────── pool.json + pool.lock (local flock) ───────┘
                                       │
   ┌───────────────────────────────────┴───────────────────────────────────┐
   ▼                                                                       ▼
homelab (ssh · priority 10)                                 aws (aws-spot · priority 20)
 └─ [slot 1 ●][slot 2 ●][slot 3 ●][slot 4 ●]                 ├─ i-04a1 (oldest) [slot 1 ●][slot 2 ●] ◀─ bin-packed first
                                                             └─ i-08f2 (newest) [slot 1 ○][slot 2 ○] ──▶ idles out → $0/hr

Client-Side Scheduling & Bin-Packing#

1. Static runners first. ssh runners in the active pool are probed first in ascending priority order without blocking. Your local build server or Mac Studio absorbs work at zero cloud cost until all its slots are occupied.
2. Oldest-first Spot bin-packing. When jobs spill onto an aws-spot fleet, sx reads running instances (refreshing EC2 state when older than 8s or on error) and tries them oldest-first (ties broken by instance ID). Concentrating new jobs onto older VMs leaves newer VMs empty so their idle_timeout clocks run uninterrupted and scale the fleet back in automatically.
3. Fair local queueing. When every slot across the pool is busy, sx takes a ticket in a fair local queue ordered by (priority, arrival_time). Interactive terminals display queued #3 of 9, while coding agents receive queue_position NDJSON events.
4. Demand-driven scale-out. Queued jobs register waiter files locally. Under pool.lock, sx launches new Spot instances only when waiting slot demand exceeds capacity already booting in the launch ledger, up to max_instances. Lower-priority fleets launch only for demand that higher-priority fleets cannot cover.
5. Autonomous scale-in. Each Spot VM powers itself off after idle_timeout (default 10m) once no spotexec-helper session holds /var/tmp/spotexec/lifecycle.lock.

Sizing Test Workers with $SPOTEXEC_CPUS#

When a 16-vCPU Spot VM is configured with max_slots = 4, each slot's fair share is 4 vCPUs—exposed automatically via $SPOTEXEC_CPUS. Point your test runners at $SPOTEXEC_CPUS so four concurrent agents don't each spawn 16 workers (64 threads total) and thrash the runner:

# Vitest (Vitest 2+ needs --minWorkers=1 on small shares)
vitest run --minWorkers=1 --maxWorkers=$SPOTEXEC_CPUS

# Jest
jest --maxWorkers=$SPOTEXEC_CPUS

# Pytest (pytest-xdist)
pytest -n $SPOTEXEC_CPUS

# Go
go test -p $SPOTEXEC_CPUS ./...

When a single heavy build or benchmark needs the entire machine, pass sx --exclusive <cmd> (or set exclusive = true on a task in .spotexec.toml) to claim every slot on a host and set SPOTEXEC_CPUS to the full machine vCPU count.

Mixing Small and Large Fleets#

Define multiple fleets in ~/.config/spotexec/config.toml to match instance shapes to workloads—for example, fast 2-vCPU instances for linters/unit tests and 8-vCPU instances for overflow or E2E suites:

default_pool = ["aws-fast", "aws-heavy"]

[runners.aws-fast]
type = "aws-spot"
region = "us-east-2"
profile = "spotexec"
instance_types = ["c7a.large", "c6a.large"]
max_slots = 1
max_instances = 4
priority = 10

[runners.aws-heavy]
type = "aws-spot"
region = "us-east-2"
profile = "spotexec"
instance_types = ["c7a.2xlarge", "m7a.2xlarge"]
max_slots = 4
max_instances = 2
priority = 20

Optional Local Pool Manager (spotexec daemon)#

sx scales from zero to max_instances and back to zero completely on its own. spotexec daemon is an optional background process running on your laptop (never in the cloud) that shares the exact same pool.json lockfiles as sx:

spotexec daemon start          # run in background; logs to ~/.local/state/spotexec/daemon/daemon.log
spotexec daemon status         # view warm instances, health probes, and recent scaling actions (--json)
spotexec daemon stop           # stop daemon; any running VMs simply idle out after idle_timeout
spotexec daemon run            # run in foreground (for launchd / systemd user services)

What spotexec daemon adds when running:

• Warm capacity (min_instances > 0). Keeps min_instances booted and refreshes their idle timers so the first sx command of the morning starts in milliseconds instead of waiting for EC2 boot. Fifteen minutes before a warm VM hits max_lifetime, the daemon drains it and boots a fresh replacement seamlessly.
• Proactive Spot interruption replacement. Probes fleet VMs every 30s; if a warm VM receives a 2-minute Spot notice while idle between sx runs, the daemon drains it and boots a replacement immediately.
• Unhealthy VM replacement & circuit breaking. Terminates and replaces VMs that fail to finish first-boot setup or stop responding for 8 minutes (throttled to 1 replacement per 30m if all VMs are unreachable due to local Wi-Fi outage, and pausing a fleet for 1 hour if 3 consecutive boots fail).
• Safety reaping. Reaps VMs past max_lifetime (if a watchdog ever failed), orphaned instances without IP addresses, and leftover image bake builders.