# Operations & Troubleshooting

## Day-2 Diagnostic & Operational Commands

```sh
spotexec doctor                      # check config, SSH ControlMaster, embedded helpers, IAM, AMI, VPC, quota
spotexec status [--json]             # list runners, running Spot VMs, slot lock holders, and idle countdowns
spotexec monitor                     # live 2s-refresh dashboard of active slots, CPU/RAM free, and $/hr burn
spotexec up aws --count 2            # pre-warm 2 Spot VMs immediately (respects max_instances)
spotexec down aws                    # terminate all VMs in a fleet now (or --all across every fleet)
spotexec unlock aws/i-0abc --slot 3  # terminate a wedged process and free slot 3 (--all reclaims all slots)
spotexec prune --older-than 7d       # prune stale dependency archives and idle slot workspaces on runners
spotexec sync-plan                   # dry-run manifest: inspect total files, bytes, top 10 largest files
spotexec cloud-init aws              # inspect the exact cloud-init user-data script sent to EC2
```

## Common Symptoms & Fixes

| Symptom / Code | Cause & Resolution |
| --- | --- |
| `BOOTSTRAP_FAILED` | Your `bootstrap_script` or `user_bootstrap_script` exited non-zero or exceeded `boot_timeout`. Read the printed tail of `/var/log/spotexec-bootstrap.log`, fix the script in `config.toml`, and run `spotexec down <runner>` (which terminates the failed VM and clears the 1-hour fleet launch hold). |
| Jobs stay `queued` without launching Spot VMs | Check `max_instances` in `config.toml`, run `spotexec daemon status` to see if the fleet is paused after repeated boot failures, and run `spotexec doctor` to verify your regional Spot vCPU quota (`L-34B43A08`). Failed launches back off for 20s. |
| `InsufficientInstanceCapacity` | EC2 is temporarily out of Spot capacity for that shape in that AZ/region. Add 2–3 fallback shapes to `instance_types` (e.g. `["c7a.xlarge", "c6a.xlarge", "m7a.xlarge"]`), switch regions, or set `on_demand_fallback = true`. |
| Sync is slow or rejected as too large | Run `spotexec sync-plan` to list the 10 largest files being included. Add large fixtures, datasets, or untracked build directories to `.spotexecignore` or `[sync].exclude`. |
| `SYNC_CONFLICT` warning after a run | You or an agent modified a file locally while the remote command also modified it. Your local file was preserved; diff or merge against `.git/spotexec/conflicts/<run-id>/<path>`. |
| Integration test fails trying to reach `localhost` | The command needs a local database or dev server on your laptop. Prefix with `SPOTEXEC_LOCAL=1 <cmd>` (when using the Claude hook) or invoke the command directly without `sx`. |
| Expired AWS SSO / `aws login` session | Run `aws sso login` (or `aws login`), or run `spotexec setup` once to provision a non-expiring scoped `spotexec` access key so sessions never expire mid-workflow. |

## Testing Without AWS Using the Local Spot Simulator

`spotexec` ships a built-in Docker Spot fleet simulator (`type = "sim"`) that executes the exact same `cloud-init` user-data, `sshd` transport, `flock` slot tokens, idle self-termination watchdog, and 2-minute Spot interruption flow locally in containers:

```toml
# Example simulator fleet in ~/.config/spotexec/config.toml
[runners.sim]
type = "sim"
max_slots = 2
max_instances = 3
idle_timeout = "1m"
```

```sh
spotexec sim build                   # build the local sshd + cloud-init simulator image
sx -r sim uname -a                   # scales out a container instance over real SSH
spotexec sim interrupt sim           # inject a synthetic 2-minute Spot interruption notice
spotexec sim rebalance sim           # inject an advisory EC2 rebalance recommendation
spotexec down sim                    # tear down simulated fleet containers
```