# Scaling, Slots & Pool Manager

Every `sx` invocation schedules cooperatively against shared local lockfiles (`~/.local/state/spotexec/`) and cloud instance tags—requiring zero central server or cloud coordinator.

```
  sx (worktree 1)    sx (worktree 2)    sx (worktree 3)    sx (worktree 4)
         │                  │                  │                  │
         └──────────── pool.json + pool.lock (local flock) ───────┘
                                       │
   ┌───────────────────────────────────┴───────────────────────────────────┐
   ▼                                                                       ▼
homelab (ssh · priority 10)                                 aws (aws-spot · priority 20)
 └─ [slot 1 ●][slot 2 ●][slot 3 ●][slot 4 ●]                 ├─ i-04a1 (oldest) [slot 1 ●][slot 2 ●] ◀─ bin-packed first
                                                             └─ i-08f2 (newest) [slot 1 ○][slot 2 ○] ──▶ idles out → $0/hr
```

## Client-Side Scheduling & Bin-Packing

1. **Static runners first.** `ssh` runners in the active pool are probed first in ascending `priority` order without blocking. Your local build server or Mac Studio absorbs work at zero cloud cost until all its slots are occupied.
2. **Oldest-first Spot bin-packing.** When jobs spill onto an `aws-spot` fleet, `sx` reads running instances (refreshing EC2 state when older than 8s or on error) and tries them **oldest-first** (ties broken by instance ID). Concentrating new jobs onto older VMs leaves newer VMs empty so their `idle_timeout` clocks run uninterrupted and scale the fleet back in automatically.
3. **Fair local queueing.** When every slot across the pool is busy, `sx` takes a ticket in a fair local queue ordered by `(priority, arrival_time)`. Interactive terminals display `queued #3 of 9`, while coding agents receive `queue_position` NDJSON events.
4. **Demand-driven scale-out.** Queued jobs register waiter files locally. Under `pool.lock`, `sx` launches new Spot instances only when waiting slot demand exceeds capacity already booting in the launch ledger, up to `max_instances`. Lower-priority fleets launch only for demand that higher-priority fleets cannot cover.
5. **Autonomous scale-in.** Each Spot VM powers itself off after `idle_timeout` (default `10m`) once no `spotexec-helper` session holds `/var/tmp/spotexec/lifecycle.lock`.

## Sizing Test Workers with `$SPOTEXEC_CPUS`

When a 16-vCPU Spot VM is configured with `max_slots = 4`, each slot's fair share is 4 vCPUs—exposed automatically via `$SPOTEXEC_CPUS`. Point your test runners at `$SPOTEXEC_CPUS` so four concurrent agents don't each spawn 16 workers (64 threads total) and thrash the runner:

```sh
# Vitest (Vitest 2+ needs --minWorkers=1 on small shares)
vitest run --minWorkers=1 --maxWorkers=$SPOTEXEC_CPUS

# Jest
jest --maxWorkers=$SPOTEXEC_CPUS

# Pytest (pytest-xdist)
pytest -n $SPOTEXEC_CPUS

# Go
go test -p $SPOTEXEC_CPUS ./...
```

When a single heavy build or benchmark needs the entire machine, pass `sx --exclusive <cmd>` (or set `exclusive = true` on a task in `.spotexec.toml`) to claim every slot on a host and set `SPOTEXEC_CPUS` to the full machine vCPU count.

## Mixing Small and Large Fleets

Define multiple fleets in `~/.config/spotexec/config.toml` to match instance shapes to workloads—for example, fast 2-vCPU instances for linters/unit tests and 8-vCPU instances for overflow or E2E suites:

```toml
default_pool = ["aws-fast", "aws-heavy"]

[runners.aws-fast]
type = "aws-spot"
region = "us-east-2"
profile = "spotexec"
instance_types = ["c7a.large", "c6a.large"]
max_slots = 1
max_instances = 4
priority = 10

[runners.aws-heavy]
type = "aws-spot"
region = "us-east-2"
profile = "spotexec"
instance_types = ["c7a.2xlarge", "m7a.2xlarge"]
max_slots = 4
max_instances = 2
priority = 20
```

## Optional Local Pool Manager (`spotexec daemon`)

`sx` scales from zero to `max_instances` and back to zero completely on its own. `spotexec daemon` is an **optional** background process running on **your laptop** (never in the cloud) that shares the exact same `pool.json` lockfiles as `sx`:

```sh
spotexec daemon start          # run in background; logs to ~/.local/state/spotexec/daemon/daemon.log
spotexec daemon status         # view warm instances, health probes, and recent scaling actions (--json)
spotexec daemon stop           # stop daemon; any running VMs simply idle out after idle_timeout
spotexec daemon run            # run in foreground (for launchd / systemd user services)
```

What `spotexec daemon` adds when running:

- **Warm capacity (`min_instances > 0`).** Keeps `min_instances` booted and refreshes their idle timers so the first `sx` command of the morning starts in milliseconds instead of waiting for EC2 boot. Fifteen minutes before a warm VM hits `max_lifetime`, the daemon drains it and boots a fresh replacement seamlessly.
- **Proactive Spot interruption replacement.** Probes fleet VMs every 30s; if a warm VM receives a 2-minute Spot notice while idle between `sx` runs, the daemon drains it and boots a replacement immediately.
- **Unhealthy VM replacement & circuit breaking.** Terminates and replaces VMs that fail to finish first-boot setup or stop responding for 8 minutes (throttled to 1 replacement per 30m if *all* VMs are unreachable due to local Wi-Fi outage, and pausing a fleet for 1 hour if 3 consecutive boots fail).
- **Safety reaping.** Reaps VMs past `max_lifetime` (if a watchdog ever failed), orphaned instances without IP addresses, and leftover `image bake` builders.