Reference

Operations & Troubleshooting

Day-2 operations, diagnosing bootstrap or capacity errors, handling write-back conflicts, unlocking wedged slots, and local Spot simulation.

Day-2 Diagnostic & Operational Commands#

spotexec doctor                      # check config, SSH ControlMaster, embedded helpers, IAM, AMI, VPC, quota
spotexec status [--json]             # list runners, running Spot VMs, slot lock holders, and idle countdowns
spotexec monitor                     # live 2s-refresh dashboard of active slots, CPU/RAM free, and $/hr burn
spotexec up aws --count 2            # pre-warm 2 Spot VMs immediately (respects max_instances)
spotexec down aws                    # terminate all VMs in a fleet now (or --all across every fleet)
spotexec unlock aws/i-0abc --slot 3  # terminate a wedged process and free slot 3 (--all reclaims all slots)
spotexec prune --older-than 7d       # prune stale dependency archives and idle slot workspaces on runners
spotexec sync-plan                   # dry-run manifest: inspect total files, bytes, top 10 largest files
spotexec cloud-init aws              # inspect the exact cloud-init user-data script sent to EC2

Common Symptoms & Fixes#

Symptom / Code	Cause & Resolution
`BOOTSTRAP_FAILED`	Your `bootstrap_script` or `user_bootstrap_script` exited non-zero or exceeded `boot_timeout`. Read the printed tail of `/var/log/spotexec-bootstrap.log`, fix the script in `config.toml`, and run `spotexec down <runner>` (which terminates the failed VM and clears the 1-hour fleet launch hold).
Jobs stay `queued` without launching Spot VMs	Check `max_instances` in `config.toml`, run `spotexec daemon status` to see if the fleet is paused after repeated boot failures, and run `spotexec doctor` to verify your regional Spot vCPU quota (`L-34B43A08`). Failed launches back off for 20s.
`InsufficientInstanceCapacity`	EC2 is temporarily out of Spot capacity for that shape in that AZ/region. Add 2–3 fallback shapes to `instance_types` (e.g. `["c7a.xlarge", "c6a.xlarge", "m7a.xlarge"]`), switch regions, or set `on_demand_fallback = true`.
Sync is slow or rejected as too large	Run `spotexec sync-plan` to list the 10 largest files being included. Add large fixtures, datasets, or untracked build directories to `.spotexecignore` or `[sync].exclude`.
`SYNC_CONFLICT` warning after a run	You or an agent modified a file locally while the remote command also modified it. Your local file was preserved; diff or merge against `.git/spotexec/conflicts/<run-id>/<path>`.
Integration test fails trying to reach `localhost`	The command needs a local database or dev server on your laptop. Prefix with `SPOTEXEC_LOCAL=1 <cmd>` (when using the Claude hook) or invoke the command directly without `sx`.
Expired AWS SSO / `aws login` session	Run `aws sso login` (or `aws login`), or run `spotexec setup` once to provision a non-expiring scoped `spotexec` access key so sessions never expire mid-workflow.

Testing Without AWS Using the Local Spot Simulator#

spotexec ships a built-in Docker Spot fleet simulator (type = "sim") that executes the exact same cloud-init user-data, sshd transport, flock slot tokens, idle self-termination watchdog, and 2-minute Spot interruption flow locally in containers:

toml

# Example simulator fleet in ~/.config/spotexec/config.toml
[runners.sim]
type = "sim"
max_slots = 2
max_instances = 3
idle_timeout = "1m"

spotexec sim build                   # build the local sshd + cloud-init simulator image
sx -r sim uname -a                   # scales out a container instance over real SSH
spotexec sim interrupt sim           # inject a synthetic 2-minute Spot interruption notice
spotexec sim rebalance sim           # inject an advisory EC2 rebalance recommendation
spotexec down sim                    # tear down simulated fleet containers