# spotexec (`sx`)

**Offload tests, typechecks, linters, E2E suites, and builds from local git worktrees to an SSH box or scale-to-zero AWS Spot fleet.**

Prefix any command with `sx` (shorthand for `spotexec run --`):

```console
$ sx pnpm vitest run -u src/auth.test.ts
 ✓ src/auth.test.ts (12 tests) 340ms
● 4.1s on aws#3 · synced 4 files (18 KB) · pulled 2 modified files
```

Files modified remotely (`prettier --write`, `eslint --fix`, `vitest -u`, codegen) return to your local worktree through a **3-way content-ID check**, so edits you or an agent make during the run are never overwritten.

---

## Why `spotexec` Exists: Coding Agents Drove Up CPU & Memory

Modern coding agents (Claude Code, Codex, and parallel worktree workflows) write code across 4–8 branches simultaneously—and then **verify** every edit by running unit tests, Playwright E2E browsers, `tsc --noEmit`, ESLint, Prettier, and builds in every worktree at once:

- Node / Vitest / Jest worker pools (`2–6 GB` RAM per worktree)
- Headless Chromium / Playwright E2E browsers (`3–8 GB` RAM per run)
- `tsc --noEmit`, ESLint, Prettier, Rust `cargo test`, Go `go test -race`, Gradle/Maven verify

Running four or five concurrent agent verification loops on a 32 GB laptop pushes memory into swap, spikes CPU fans, and stalls your editor.

### Two Core Benefits

1. **Low cost (~65–70% Spot savings):** Typical 2 vCPU / 4–8 GB AWS Spot workers (`t3a.large` ~$0.024/hr, `c6i.large` ~$0.028/hr, `c7a.large` ~$0.035/hr, `m7a.large` ~$0.038/hr) cost 65–70% less than On-Demand and scale back to `$0.00/hr` when idle.
2. **Run far more on your local hardware:** Offloading heavy verification lets you run 4–8+ parallel coding agents, editors, and local dev servers on your Mac without thermal throttling or swap thrashing.

### Why Spot Works Here: Offload the Heavy Task, Not the Whole Agent

Running an entire agent workflow on Spot instances is fragile—if AWS reclaims the VM mid-session, agent state and progress are lost. `spotexec` keeps your agents, Git worktrees, and editor on your Mac and only dispatches individual stateless verification tasks. If a Spot VM is reclaimed mid-check, nothing is lost: `sx` replays that single task on another slot.

`spotexec` moves those heavy verification commands off your laptop while keeping `git`, `rg`, file edits, and `localhost` dev servers local:

```
Local Laptop (4+ Agent Worktrees)              Remote Capacity (SSH or AWS Spot)
─────────────────────────────────              ─────────────────────────────────
wt/auth   ─┐                                   homelab (ssh, priority 10)
wt/api    ─┼─ sx / Claude Bash Hook ── SSH ──▶  └─ [slot 1 ●][slot 2 ●][slot 3 ●][slot 4 ●]
wt/e2e    ─┤  (Git-blob delta sync)
wt/lint   ─┘                                   aws-spot (overflow, priority 20)
                                                ├─ i-04a1 [slot 1 ●][slot 2 ●]
                                                └─ scales to $0.00/hr after idle_timeout
```

---

## Core Properties

- **Drop-in two-letter prefix.** `sx <command>` streams stdout/stderr live, preserves exit codes, and rewrites remote paths in stack traces back to local paths.
- **Scale-to-zero AWS Spot or static SSH.** Fill a homelab box first (`priority = 10`), spill overflow to AWS Spot VMs (`priority = 20`), bin-pack jobs onto the oldest VMs, and let idle VMs self-terminate via a cloud-init `systemd` watchdog.
- **3-way safe write-back.** Compares local files against the exact Git blob SHA-1 baseline that was synced. If you edited a file mid-run, your local edit is kept and the remote output is saved under `.git/spotexec/conflicts/<run-id>/`.
- **Delta sync & lockfile caches.** Clean tracked files reuse `.git/index` blob IDs without reading disk; only missing blobs travel over SSH ControlMaster. Dependencies hydrate via `reflink` (CoW) locally and private S3 archives across Spot VMs.
- **Zero cloud control plane.** Single Go binary with embedded Linux/macOS (`amd64`/`arm64`) helpers and hand-signed AWS SigV4 requests. No SaaS account, no daemon in AWS, and **zero IAM credentials on runner VMs**.
- **Built for coding agents.** Emits structured NDJSON events on non-TTY stderr, and includes `spotexec hook install --claude` to rewrite heavy Bash commands automatically when laptop load exceeds `when_load_above`.

---

## Install on macOS (Recommended)

```sh
brew install daintreehq/tap/spotexec
```

Installs `spotexec`, the `sx` shorthand alias, shell completions, and embedded cross-compiled remote helpers.

### Linux & Source Builds

```sh
# Homebrew on Linux
brew install daintreehq/tap/spotexec

# Debian / Ubuntu, Fedora / RHEL, Alpine (from GitHub Releases)
sudo apt install ./spotexec_*_linux_amd64.deb
sudo dnf install ./spotexec_*_linux_amd64.rpm
sudo apk add --allow-untrusted ./spotexec_*_linux_amd64.apk

# From source (Go 1.26+; cross-compiles and embeds remote helpers)
git clone https://github.com/daintreehq/spotexec.git && cd spotexec
make install
```

## 3-Step Quickstart

```sh
# 1. Provision runner config and dedicated per-machine IAM key (once per machine)
aws login
spotexec setup

# 2. Initialize project config in your repository (once per repo)
cd ~/code/my-app
spotexec init
spotexec doctor

# 3. Prefix heavy commands with sx (or install the Claude Code hook)
sx pnpm test
sx pnpm prettier --write .
spotexec hook install --claude
```

## Documentation Index

- [Introduction & Quickstart](/docs.md) — Overview, requirements, install, and first run
- [Setting Up Machines](/docs/setup.md) — `spotexec setup`, SSH vs. AWS Spot runners, multi-machine provisioning, key rotation
- [Running Commands & Write-back](/docs/running.md) — Flags, runner environment variables, exit codes, 3-way write-back, artifacts
- [Coding Agents & Hooks](/docs/agents.md) — Claude Code & Codex hooks, load-aware offloading, NDJSON event stream
- [Containers, Caches & Baked Images](/docs/environments.md) — Project Dockerfiles, `reflink`/S3 dependency cache, `image bake`
- [AWS Spot Fleet Guide](/docs/aws.md) — IAM policy, regions, Spot vCPU quotas, VPC networking, Spot VM lifecycle
- [Scaling, Slots & Pool Manager](/docs/scaling.md) — Slot bin-packing, queueing, `$SPOTEXEC_CPUS`, optional local `daemon`
- [Architecture & Security Model](/docs/architecture.md) — RPC lifecycle, cloud-init watchdog, security boundaries, known limits
- [Configuration Reference](/docs/configuration.md) — Complete `config.toml` and `.spotexec.toml` reference
- [CLI & File Reference](/docs/reference.md) — Every command and local/remote state path
- [Operations & Troubleshooting](/docs/troubleshooting.md) — Day-2 commands, common error codes, Docker Spot simulator

---

# Introduction & Quickstart

`spotexec` (`sx`) runs your tests, typechecks, linters, E2E browsers, formatters, and builds on remote hardware—straight from a local Git worktree. You can point it at a Linux box or spare Mac over SSH, an elastic fleet of AWS Spot instances that boot on demand and terminate themselves when idle, or both at once.

Prefix any command with `sx` (shorthand for `spotexec run --`):

```console
$ sx pnpm vitest run -u src/auth.test.ts
 ✓ src/auth.test.ts (12 tests) 340ms
● 4.1s on aws#3 · synced 4 files (18 KB) · pulled 2 modified files
```

Files modified by the remote command (`prettier --write`, `eslint --fix`, `vitest -u`, codegen) come back to your local worktree through a **3-way content-ID check**, so concurrent local edits are never overwritten.

## Why It Matters for Parallel Coding Agents

AI coding agents (such as Claude Code and Codex) have changed the bottleneck on developer machines. Editing files across four or six Git worktrees in parallel uses practically zero CPU or memory—but **every agent verifies its own work** by invoking test runners, headless E2E browsers, typecheckers, linters, and compilers simultaneously:

| Workload (4 Parallel Worktrees) | Typical Peak RAM | Local Impact |
| --- | --- | --- |
| `vitest` / `jest` worker pools | `8 – 16 GB` | Node V8 heaps multiply per worktree |
| `playwright test` / Cypress E2E | `10 – 24 GB` | Multiple headless Chromium / WebKit instances |
| `tsc --noEmit` + `eslint` | `6 – 12 GB` | Full AST & type-graph resident in each process |
| `cargo test` / `go test -race` | `8 – 20 GB` | Parallel `rustc` / linker spikes + disk I/O |

On a laptop, those concurrent verification loops trigger swap thrashing, thermal throttling, sluggish editors, and out-of-memory kills.

`spotexec` is designed specifically for this workload:

- **Shared remote slots.** Many local worktrees share a pool of remote runners through isolated slots, a fair local queue, and a lockfile-keyed dependency cache.
- **Transparent agent hooks.** `spotexec hook install --claude` rewrites heavy test, build, and lint commands to `spotexec run -- '<cmd>'` automatically (optionally only when laptop load exceeds `when_load_above = 0.6`), while keeping `git`, `rg`, and `localhost` dev servers local.
- **Zero cloud control plane.** Nothing runs in your cloud account except the ephemeral VMs themselves. Coordination uses local `flock` state and EC2 tags, and each VM terminates itself via a cloud-init `systemd` timer even if you close your laptop lid.

## Requirements

**On your local machine:**

- `git` and OpenSSH (`ssh`, used with `ControlMaster` connection multiplexing) on `PATH`.
- AWS CLI v2 is **optional**: `spotexec` signs AWS API requests directly with no AWS SDK in the binary. Install `aws` only if you sign in via SSO, `aws login`, `assume-role`, or want `spotexec setup` to provision IAM users automatically.

**On a remote runner:**

- An SSH server, `bash`, and `gzip` on Linux or macOS (`amd64` or `arm64`).
- Nothing of `spotexec` needs to be preinstalled: the `spotexec` binary embeds cross-compiled `spotexec-helper` binaries for all four OS/architecture targets and uploads the matching helper to `~/.cache/spotexec/bin/` on first contact.
- Docker is required only if you opt into [project containers](/docs/environments#project-containers) (installed automatically on AWS Spot runners).

## Installation

### macOS via Homebrew (Recommended)

```sh
brew install daintreehq/tap/spotexec     # installs spotexec, the sx alias, and shell completions
spotexec version
```

Homebrew is the recommended way to install and upgrade `spotexec` on macOS (Apple Silicon and Intel), and also works with Homebrew on Linux.

### Linux Packages (`.deb`, `.rpm`, `.apk`)

Download the package for your architecture (`amd64` or `arm64`) from the [latest GitHub release](https://github.com/daintreehq/spotexec/releases/latest):

```sh
# Debian / Ubuntu
sudo apt install ./spotexec_*_linux_amd64.deb

# Fedora / RHEL
sudo dnf install ./spotexec_*_linux_amd64.rpm

# Alpine
sudo apk add --allow-untrusted ./spotexec_*_linux_amd64.apk
```

### Standalone Archive or Source Build

Plain `.tar.gz` archives for macOS and Linux (`amd64` and `arm64`) are attached to every release—place `spotexec` on your `PATH` and symlink `sx` to it.

To build from source (requires Go 1.26+):

```sh
git clone https://github.com/daintreehq/spotexec.git && cd spotexec
mkdir -p "$(go env GOPATH)/bin"
make install        # cross-compiles helpers, builds bin/spotexec + bin/sx, installs to GOPATH/bin
```

> [!NOTE]
> Plain `go install` is not supported because `make build` cross-compiles the remote `spotexec-helper` for `linux/amd64`, `linux/arm64`, `darwin/amd64`, and `darwin/arm64` before embedding them into the single `spotexec` binary.

## Quickstart

### 1. Set up your machine (once per machine)

```sh
aws login                  # or aws sso login / aws configure, with IAM admin rights
spotexec setup             # writes ~/.config/spotexec/config.toml and creates a scoped IAM key
```

`spotexec setup` writes `~/.config/spotexec/config.toml`, creates a dedicated least-privilege IAM user (`spotexec-<hostname>`) and non-expiring access key under profile `spotexec`, verifies VPC and Spot vCPU quotas, and optionally sets up additional machines on your LAN over SSH:

```sh
spotexec setup studio-01.local studio-02.local
```

Using a homelab Linux server or spare Mac instead of AWS? Configure it as an `ssh` runner in `~/.config/spotexec/config.toml`—see [Setting Up Machines](/docs/setup#choosing-runners).

### 2. Initialize your repository (once per repo)

```sh
cd ~/code/my-app
spotexec init               # detects lockfiles, package manager, and common test/lint tasks
spotexec container init     # optional: scaffold .spotexec/Dockerfile for reproducible toolchains
spotexec doctor             # verify config, SSH ControlMaster, helper, AWS credentials, and quota
```

### 3. Run commands with `sx`

```sh
sx pnpm test                     # delta sync → run in remote slot → write back mutations
sx pnpm prettier --write .       # formatted files land safely back in your worktree
sx test --reporter=dot           # run a named task from .spotexec.toml with extra args
sx verify                        # fan out parallel tasks (e.g. typecheck + test) across slots
spotexec status                  # inspect active slots, runners, and idle timers
spotexec monitor                 # live TUI dashboard of slot load and Spot $/hr burn rate
```

> [!TIP]
> The very first AWS Spot run boots a fresh VM (~60s plus your bootstrap script). Subsequent runs reuse the warm VM instantly until it has been idle for `idle_timeout` (default `10m`). Run `spotexec image bake` to snapshot your bootstrap into a private AMI so future cold boots finish in bare EC2 boot time.

---

# Setting Up Machines

Every developer machine where you run `sx` needs three things:

1. `spotexec` installed on `PATH` (along with `git` and `ssh`);
2. A global runner configuration at `~/.config/spotexec/config.toml` defining where commands execute;
3. For `aws-spot` fleets, scoped AWS credentials.

`spotexec setup` automates all three on your current workstation and on any other machines you pass over SSH. It is idempotent and safe to re-run at any time.

## Setting Up This Machine

```sh
brew install daintreehq/tap/spotexec
brew install awscli            # recommended for initial AWS setup
aws login                      # or aws sso login / aws configure (with IAM user management access)
spotexec setup
```

When you run `spotexec setup`:

1. **Runner config.** If `~/.config/spotexec/config.toml` does not exist, setup writes a starter configuration with an `aws` Spot fleet in your AWS CLI's default region. Existing configs are preserved.
2. **Scoped AWS credentials.** For each `aws-spot` runner, setup previews every change and asks for confirmation before touching your account:
   - Creates a dedicated IAM user for this machine named `spotexec-<hostname>` (path `/spotexec/`, tagged `spotexec-managed`) with an inline least-privilege policy (`spotexec setup --print-policy`).
   - Creates an access key and writes it to `~/.aws/credentials` (mode `0600`) under profile `spotexec`. After verifying EC2 connectivity and confirming no environment overrides conflict, it cleans up any older keys on that user.
   - Sets `profile = "spotexec"` on your AWS runners in `config.toml`.
   - Ensures the `AWSServiceRoleForEC2Spot` service-linked role exists and creates a free default VPC if the region lacks one (`--no-vpc` skips this).
   - Verifies your regional Spot vCPU quota against your configured fleet ceiling and prints the exact `aws service-quotas` command if an increase is needed.
3. **Other machines.** In an interactive terminal, setup discovers other Macs with Remote Login via Bonjour (or Avahi on Linux) plus `Host` entries in `~/.ssh/config`, and offers to provision them over SSH.

```console
$ spotexec setup
✓ runner config /Users/me/.config/spotexec/config.toml
Setting up AWS access for aws (region us-east-1)…

Signed in to AWS account 123456789012 as arn:aws:iam::123456789012:user/me. This will:
  • create (or update) IAM user spotexec-studio-02 with only spotexec's permissions (see `spotexec setup --print-policy`)
  • save an access key for it as AWS profile "spotexec" in ~/.aws/credentials (if it hasn't one already)
  • set profile = "spotexec" on runner(s) aws
Continue? [Y/n]
  created IAM user spotexec-studio-02
  attached the spotexec policy
  created access key AKIA…
  saved it as AWS profile spotexec (/Users/me/.aws/credentials)
  verified: the key signs in and can reach EC2 in us-east-1
✓ AWS profile spotexec → arn:aws:iam::123456789012:user/spotexec/spotexec-studio-02 (key AKIA…; it doesn't expire)
✓ set profile = "spotexec" on aws in /Users/me/.config/spotexec/config.toml
✓ Spot vCPU quota in us-east-1: 640 (runners need up to 32)
```

Once `spotexec setup` finishes, your interactive `aws login` or SSO session can expire without interrupting `sx`—you only need an admin session again when rotating keys or changing IAM setup.

## Choosing Runners

A **runner** is either a static **SSH host** (`type = "ssh"`) or an elastic **Spot fleet** (`type = "aws-spot"`). A **pool** is the ordered list of runners a command may use, and each machine provides `max_slots` isolated execution **slots** (one concurrent job per slot).

### Option A: An SSH Box You Already Own

Non-interactive SSH key login must work (`ssh -o BatchMode=yes me@192.168.1.50 true`), and the host key must be present in `~/.ssh/known_hosts`. Commands execute under `bash` after sourcing `/etc/profile` and `~/.bash_profile` (falling back to `~/.profile`), plus `remote_path_setup` if configured.

```toml
version = 1
default_pool = ["homelab"]

[runners.homelab]
type = "ssh"
hosts = ["me@192.168.1.50", "me@homelab.tailnet.ts.net"]   # LAN first, Tailscale fallback
max_slots = 4
```

> [!TIP]
> Passing multiple addresses in `hosts` dials all routes in parallel and prefers an earlier entry if it connects within 500ms of the fastest response—ideal for switching seamlessly between home LAN Ethernet and Tailscale on the road.

### Option B: Scale-to-Zero AWS Spot Fleet

Spot VMs boot clean Ubuntu 24.04 images on demand and power themselves off after `idle_timeout` without activity. Install your global toolchain via `bootstrap_script` / `user_bootstrap_script`, or define reproducible per-repo environments with [project containers](/docs/environments#project-containers).

```toml
version = 1
default_pool = ["aws"]

[runners.aws]
type = "aws-spot"
profile = "spotexec"
region = "us-east-2"                             # always set explicitly
instance_types = ["c7a.xlarge", "c6a.xlarge"]    # tried in order on capacity errors
max_slots = 2                                    # concurrent jobs per VM
max_instances = 4                                # fleet ceiling (8 total slots)
idle_timeout = "10m"                             # self-terminates after 10m idle
```

### Option C: Hybrid Pool (Homelab First, Spot Overflow)

List both runners in `default_pool` and assign lower `priority` numbers to hardware you already pay for. `sx` fills all 4 `homelab` slots first and boots AWS Spot instances only when concurrent agent demand overflows:

```toml
version = 1
default_pool = ["homelab", "aws"]

[runners.homelab]
type = "ssh"
hosts = ["me@192.168.1.50", "me@homelab.tailnet.ts.net"]
max_slots = 4
priority = 10                  # lower runs first ($0 marginal cost)

[runners.aws]
type = "aws-spot"
profile = "spotexec"
region = "us-east-2"
instance_types = ["c7a.2xlarge", "c6a.2xlarge", "m7a.2xlarge"]
max_slots = 4
max_instances = 4
priority = 20                  # takes overflow only when homelab is full
idle_timeout = "10m"
hourly_cost = 0.14
```

## Provisioning Additional Machines Over SSH

Pass hostnames or SSH aliases to `spotexec setup` to configure laptops, Mac Studios, or remote dev boxes in one shot:

```sh
spotexec setup studio-01.local studio-02.local me@devbox
```

Over a single SSH connection per host, `spotexec setup`:

1. Installs or upgrades `spotexec` (via Homebrew if installed, otherwise downloading the latest release into `~/.local/bin`; override with `--install brew|release|copy|none`).
2. Copies `~/.config/spotexec/config.toml` (backing up any differing file to `config.toml.bak-<timestamp>`).
3. Uses your local admin AWS session—streamed over SSH `stdin`, never exposed in process args or written to remote disk—to create **that machine's own** IAM user (`spotexec-<remote-hostname>`), access key, SSH keypair, and `owner-id`.

Each machine keeps isolated credentials and its own `owner-id` tag, so one machine never terminates another machine's active Spot VMs, and a lost laptop can be revoked individually.

## Key Rotation, Revocation, and Keyless Mode

- **Rotate this machine's key:** `spotexec setup --rotate` creates a new access key, verifies EC2 access, updates `~/.aws/credentials`, and deletes the previous key.
- **Audit all provisioned machines:** `aws iam list-users --path-prefix /spotexec/` lists every machine user.
- **Keep zero long-lived keys on disk:** Run `spotexec setup --skip-aws` and point `profile` at an IAM Identity Center (`aws sso login`) profile carrying the `spotexec` policy.

To revoke a retired or lost machine immediately:

```sh
U=spotexec-studio-01
for k in $(aws iam list-access-keys --user-name "$U" --query 'AccessKeyMetadata[].AccessKeyId' --output text); do
  aws iam delete-access-key --user-name "$U" --access-key-id "$k"
done
aws iam delete-user-policy --user-name "$U" --policy-name spotexec
aws iam delete-user --user-name "$U"
```

## `spotexec setup` Flags

| Flag | Effect |
| --- | --- |
| `[host...]` | Also install and configure these machines over SSH. |
| `--yes`, `-y` | Skip confirmation prompts and skip interactive LAN host discovery. |
| `--install auto\|brew\|release\|copy\|none` | Installation method for remote machines (default `auto`). |
| `--admin-profile NAME` | AWS profile used to manage IAM users (default: `AWS_PROFILE`, then `default`). |
| `--profile NAME` | Local AWS profile name to write credentials into (default `spotexec`). |
| `--user NAME` | IAM user name for this machine (default `spotexec-<hostname>`). |
| `--rotate` | Issue a fresh IAM access key even if the existing one works. |
| `--no-vpc` | Do not create a default VPC when the configured region lacks one. |
| `--skip-aws` | Write/copy `config.toml` only without provisioning IAM credentials. |
| `--print-policy` | Print the least-privilege IAM policy JSON to stdout and exit. |

---

# Running Commands & Write-back

```sh
sx [flags] <command> [args...]        # sync → run remotely → write back changes
sx [flags] -- <command> [args...]     # explicit separator if command matches a flag/subcommand
sx <task> [args...]                   # run a named or parallel task from .spotexec.toml
spotexec exec -- <command>            # run inside the last-used slot without syncing
```

`sx` stops parsing its own flags at the first non-flag token, so `sx pnpm vitest run --watch=false` forwards `--watch=false` straight to `pnpm`. Multiple arguments are shell-quoted individually; a single string argument runs as a shell pipeline (`sx 'pnpm lint && pnpm test'`). Absolute local paths inside your worktree are automatically rewritten to the remote workspace path, and remote stack trace paths are rewritten back on output.

> [!IMPORTANT]
> `spotexec` only syncs an enclosing Git worktree or a directory containing `.spotexec.toml`. Running `sx` in an untracked parent directory like `$HOME` is refused so `~/.ssh` or `~/.aws` can never be uploaded by mistake.

## Run Flags

| Flag | Meaning |
| --- | --- |
| `--timeout D` | Wall-clock limit for the command (default `5m`, or project `timeout`). Exits `124` if exceeded. |
| `--queue-timeout D` | Maximum wait for a free slot when all runners are saturated (default `10m`). |
| `--sync auto\|in\|none` | `auto` syncs in and writes mutations back; `in` syncs in and discards remote changes; `none` skips sync. |
| `--writeback always\|on-success\|never` | Controls when remote file changes return (`always` is default so `eslint --fix` exiting `1` still writes fixes back). |
| `-r, --runner NAME` / `--pool A,B` | Pin execution to a specific runner or comma-separated pool. |
| `--exclusive` | Claim all slots on a machine so a heavy build or benchmark gets 100% of its vCPUs and RAM. |
| `--priority N` | Queue priority when waiting for a busy pool (higher numbers dequeue first). |
| `--no-boot` | Use only currently running machines; never launch new Spot VMs for this command. |
| `--pull GLOB` | Pull matching gitignored artifacts back after the run (repeatable, e.g. `--pull coverage/`). |
| `--no-deps` | Skip `[deps]` dependency cache hydration. |
| `--no-retry` | Disable automatic replay if an ephemeral Spot VM is interrupted mid-run. |
| `--fallback-local` | Run locally if no remote runner can be reached (never triggers once remote execution has started). |
| `--force-large-sync` | Override the safety cap of 100,000 files or 5 GiB. |
| `--json` / `--pretty` / `-q, --quiet` | Telemetry output format on stderr: NDJSON, erasable spinner + footer, or silent. |
| `--events-file PATH` | Append NDJSON lifecycle telemetry to `PATH` alongside normal terminal output. |
| `-C, --dir DIR` | Execute as if invoked from `DIR`. |

## Environment on the Runner

Every remote command receives these environment variables automatically, in addition to keys listed in `env_allow` (forwarded from your local shell) and `env` (static map in `.spotexec.toml`):

| Variable | Value |
| --- | --- |
| `SPOTEXEC=1`, `CI=true` | Indicates non-interactive remote execution. |
| `SPOTEXEC_RUNNER` | Name of the active runner (`homelab`, `aws`, etc.). |
| `SPOTEXEC_WORKSPACE` | Absolute path to the slot's workspace directory on the runner. |
| `SPOTEXEC_SLOT` | Host-wide slot number (`1..max_slots`), unique across all repos on that host. |
| `SPOTEXEC_PORT_BASE` | `20000 + slot × 1000` — bind ephemeral test servers/databases here so parallel slots never collide on ports. |
| `SPOTEXEC_CPUS` | Fair CPU share for this slot (`vCPUs ÷ max_slots`, or all vCPUs under `--exclusive`). |
| `SPOTEXEC_SLOTS` | Total configured slots on the machine. |
| `SPOTEXEC_RUN_ID`, `SPOTEXEC_ATTEMPT` | Stable run UUID across Spot replays (`SPOTEXEC_ATTEMPT=2` on replay). |
| `COMPOSE_PROJECT_NAME` | `sx_<repo>_sNN`, isolating concurrent Docker Compose test stacks per slot. |
| `FORCE_COLOR`, `CLICOLOR_FORCE` | Enabled automatically when your local `stdout` is an interactive TTY. |

## Exit Codes

| Exit Code | Meaning |
| --- | --- |
| *Command's exit status* | Passed through verbatim when the remote command runs and exits normally. |
| `124` | Remote command exceeded `--timeout` (sent `SIGTERM`, then `SIGKILL` after `grace_period`). |
| `125` | `spotexec` infrastructure error (unreachable SSH host, bootstrap failure, quota error). Outranks test failures in parallel tasks so coding agents never confuse "couldn't run tests" with "tests failed". |
| `128 + N` | Remote command terminated by signal `N`. |
| `130` | Interrupted locally (`Ctrl-C`, `SIGTERM`, or `SIGHUP`). |
| `1`, `2` | CLI usage or configuration validation error. |

## 3-Way Content-ID Write-back

When a remote command formats code (`prettier --write`), applies lint fixes (`eslint --fix`), updates test snapshots (`vitest -u`), or runs codegen, `spotexec` brings those edits back without risking concurrent local work.

Before syncing, `sx` records the exact Git SHA-1 blob ID of every synced file—the **baseline**. After execution, `spotexec-helper` hashes the slot workspace by content (never trusting mtimes) and returns mutations. Each modified or deleted file is checked against your local worktree **right now**:

```
Remote file mutated (new SHA-1 != baseline SHA-1)
  │
  ├─ Local file now == Remote result  ──▶ No-op (already identical)
  │
  ├─ Local file now == Baseline SHA-1 ──▶ Safe to apply: stage to temp file + atomic rename
  │
  └─ Local file now != Baseline SHA-1 ──▶ Edited locally mid-run!
                                          • Keep your local file untouched
                                          • Save remote bytes to <git-dir>/spotexec/conflicts/<run-id>/<path>
                                          • Emit SYNC_CONFLICT warning
```

| Your Local File Now | Write-back Outcome |
| --- | --- |
| Equals remote output | Nothing to write. |
| Equals synced baseline | Applied atomically via temp file + rename in the target directory. |
| Changed since sync started | **Your local file is preserved.** Remote version is parked in `<git-dir>/spotexec/conflicts/<run-id>/<path>` and a `SYNC_CONFLICT` warning is emitted. |

Additional write-back invariants:

- Deletions apply deepest-first before file writes.
- Newly created remote files are written back **only** if they are not ignored by `.gitignore` / `.spotexecignore` and do not already exist locally—so `dist/`, `target/`, and `node_modules/` never pollute your laptop.
- Paths containing `.git` (in any case spelling) or traversing symlinks are rejected unconditionally.

## Pulling Gitignored Artifacts

When you *do* want gitignored outputs back locally (such as HTML coverage reports, Playwright traces, or compiled binaries), pull them explicitly:

```sh
# Pull after a single run
sx --pull coverage/ --pull 'packages/*/coverage' pnpm vitest run --coverage

# Or pull on demand from the most recently used slot
spotexec pull coverage/ playwright-report/ --dest ./artifacts
```

To pull artifacts automatically on every run, set `pull_artifacts = ["coverage/"]` under `[sync]` in `.spotexec.toml`. Pulled artifacts never overwrite tracked or non-ignored source files.

---

# Coding Agents & Hooks

Running multiple coding agents (Claude Code, Codex, or custom harnesses) across parallel Git worktrees is where laptops hit a wall: the agents themselves are lightweight HTTP loops, while the **verification tools they spawn** (`vitest`, `playwright`, `tsc --noEmit`, `eslint`, `cargo test`, `go test`) consume gigabytes of RAM per worktree.

`spotexec` integrates directly with coding agents through two mechanisms:

1. **Transparent command hooks** that rewrite heavy verification commands to `spotexec run -- '<cmd>'` automatically while leaving fast local commands untouched.
2. **Structured NDJSON telemetry** on non-TTY `stderr`, keeping the command's `stdout` and `stderr` clean for agent parsing.

## Installing the Claude Code Hook

```sh
# Install into .claude/settings.local.json (per-worktree / gitignored)
spotexec hook install --claude

# Or commit into .claude/settings.json (--shared) or install globally in ~/.claude (--global)
spotexec hook install --claude --shared
spotexec hook install --claude --global

# Dry-run any command string to verify whether the hook offloads or keeps it local
spotexec hook check 'pnpm vitest run'
spotexec hook check 'git status'
```

### What Gets Offloaded vs. Kept Local

The PreToolUse Bash hook inspects the command AST before execution:

```
Agent invokes Bash("<command>")
  │
  ├─ Project has .spotexec.toml? ── No ──▶ Run locally (no-op)
  │
  ├─ Prefixed with SPOTEXEC_LOCAL=1, or uses pipes / redirects / background `&` / `cd ..`?
  │   └─ Yes ──▶ Run locally
  │
  ├─ Matches local-only tools (git, rg, ls, dev servers, --watch)?
  │   └─ Yes ──▶ Run locally
  │
  ├─ Matches heavy verification (test suites, tsc, linters, formatters, builds, gradle/mvn verify)
  │  AND laptop normalized load >= when_load_above?
  │   └─ Yes ──▶ Rewrite to: spotexec run -- '<command>'
  │
  └─ Otherwise ──▶ Run locally
```

- **Offloaded to remote runners:** Unit & E2E test suites (`vitest`, `jest`, `playwright`, `pytest`, `go test`, `cargo test`), typecheckers (`tsc`), linters & formatters (`eslint`, `prettier`, `biome`, `ruff`, `clippy`), compilers/bundlers, and Maven/Gradle `verify` goals.
- **Kept local automatically:** `git`, `ls`, `rg`, `fd`, interactive dev servers (`next dev`, `vite dev`), watch modes (`--watch`), shell pipelines, file redirects, background jobs, commands that `cd` outside the repository, and any command prefixed with `SPOTEXEC_LOCAL=1`.

> [!TIP]
> If an integration test requires a service running on your laptop (such as `localhost:5432`), prefix that command with `SPOTEXEC_LOCAL=1 pnpm test:db` so the hook leaves it local.

### Permission Rules in Claude Code

The hook **never auto-approves commands**—it only rewrites the command string before Claude Code evaluates permissions. If you pre-approve test commands in Claude Code, add `Bash(spotexec run:*)` to your allowed tool list so rewritten commands run without manual prompts.

### Load-Based Offloading & `[hook]` Tuning

Configure `[hook]` in `.spotexec.toml` to offload commands only when your laptop is actually under load:

```toml
[hook]
when_load_above = 0.6          # offload only when normalized CPU load exceeds 60%
patterns = ["^pnpm build:"]    # additional regexes to offload
exclude = ["test:unit:fast"]   # regexes to always keep local
# mode = "block"               # deny direct execution with instructions instead of rewriting
# disabled = false             # disable the hook in this repository
```

## Codex and Other Agent Harnesses

For harnesses without a PreToolUse command-rewriting hook (such as Codex CLI), `spotexec` generates a concise, machine-targeted section in `AGENTS.md` instructing the agent when to prefix commands with `sx` and how to interpret exit code `125` vs test failures:

```sh
spotexec hook install --codex      # appends spotexec instructions to AGENTS.md
spotexec hook install --uninstall  # cleanly removes Claude or Codex hook configuration
```

## Machine-Readable NDJSON Telemetry

Whenever `stderr` is not a TTY—or when `--json` or `SPOTEXEC_AGENT=1` is set—`spotexec` replaces the interactive terminal spinner with compact, deterministic NDJSON lifecycle events on `stderr`. The remote command's own output streams untouched:

```json
{"event":"slot_acquired","connect_ms":13,"runner":"aws","slot":3,"waited_ms":0}
{"event":"sync_in","bytes":0,"duration_ms":0,"files":0,"mode":"fingerprint_hit"}
{"event":"sync_out","conflicts":[],"duration_ms":31,"mutated_files":["src/index.ts"]}
{"event":"result","duration_ms":4120,"exit_code":0,"timed_out":false}
```

Full lifecycle event types emitted for agents and log pipelines (`--events-file`):

- **Queueing & placement:** `queued`, `queue_position`, `placed`, `slot_acquired`
- **Elastic scaling:** `scale_out`, `provision`, `instance_running`, `instance_ready`
- **Execution pipeline:** `manifest`, `sync_in`, `deps`, `image`, `exec`, `sync_out`, `pull`, `cancel`, `result`
- **Spot resilience:** `spot_interruption`, `spot_rebalance`, `retry`
- **Diagnostics:** `warning` (with stable codes: `SYNC_CONFLICT`, `RUNNER_LOST`, `SPOT_INTERRUPTION`, `TIMEOUT`, `LOCAL_FALLBACK`) and `error` (`SPOTEXEC_ERROR`, paired with exit code `125`).

---

# Containers, Caches & Baked Images

Fast remote execution comes down to eliminating repeated setup work across slots and ephemeral VMs. `spotexec` provides three composable layers:

1. **Project containers (`[container]`)** — pin exact Node, Python, Go, Rust, Bun, or JVM toolchains in a committed Dockerfile built once and cached in ECR.
2. **Lockfile-keyed dependency caches (`[deps]`)** — restore `node_modules`, `.venv`, or build caches instantly via copy-on-write `reflink` locally and private S3 archives across Spot VMs.
3. **Baked AMIs (`spotexec image bake`)** — snapshot your host bootstrap so fresh Spot VMs come online in bare EC2 boot time.

## Project Containers

With `[container]` configured in `.spotexec.toml`, both dependency installation and every `sx` command run inside a Docker container on the runner. The runner host itself only needs Docker (installed automatically on AWS Spot VMs).

### Scaffold, Customize, and Build

```sh
spotexec container init              # detects stack, writes .spotexec/Dockerfile, updates .spotexec.toml
$EDITOR .spotexec/Dockerfile         # add system libraries (e.g. libvips, playwright deps, protobuf)
spotexec container build -r aws      # build remotely now and push to private ECR on AWS
sx pnpm test                         # runs inside the pinned image
```

`spotexec container init` inspects your repository manifests and writes a pinned Dockerfile:

| Detected Stack | Generated Base Image |
| --- | --- |
| Node.js (`npm`, `pnpm`, `yarn`) | `node:<major>-bookworm` (version from `.nvmrc`, `.node-version`, or `engines.node`; pins `pnpm` via `packageManager`) |
| Bun | `oven/bun:1` |
| Python with `uv` | `ghcr.io/astral-sh/uv:python<x.y>-bookworm` (version from `.python-version`) |
| Python | `python:<x.y>-bookworm` |
| Go | `golang:<x.y>-bookworm` (version from `go.mod`) |
| Rust | `rust:<version>-bookworm` (from `rust-toolchain.toml`, plus `clippy` and `rustfmt`) |
| Maven / Gradle | `eclipse-temurin:21-jdk` (adds `maven` if `./mvnw` is absent) |
| Fallback | `ubuntu:24.04` with `build-essential` |

> [!IMPORTANT]
> `spotexec container init` stages `.spotexec/Dockerfile` with `git add` so it is included in the Git worktree manifest. Commit `.spotexec/Dockerfile` alongside `.spotexec.toml` so every worktree and runner receives it.

### Dockerfile Authoring Rules

- **Install tools, never source code.** Your worktree is bind-mounted at runtime, and dependencies (`node_modules`, `.venv`) are hydrated from `[deps]`. Do not `COPY . .` in `.spotexec/Dockerfile`.
- **Install system-wide (e.g. `/usr/local`).** Commands execute as the non-root runner user with `HOME=/home/spotexec` mounted per slot so tool caches (`~/.cache`, Go build cache, Cargo registry) persist across runs. Anything installed into `/root` or baked into `/home/spotexec` inside the image will be hidden by the slot mount.
- **Content-addressed rebuilds.** `spotexec` hashes the Dockerfile and its `.dockerignore`-filtered build context. The image is rebuilt only when that hash changes, pushed to a private `spotexec/<repo-id>` ECR repository (retaining the 10 newest tags), and pulled by future Spot VMs without rebuilding.

```toml
[container]
dockerfile = ".spotexec/Dockerfile"   # or pull an existing image directly with: image = "node:22-bookworm"
# context = "."                       # defaults to the Dockerfile's directory
# options = ["--shm-size=1g"]         # extra docker run flags (useful for Chromium / Playwright)
# user = "runner"                     # "runner" (default; preserves file ownership) or "root"
# timeout = "20m"                     # image build/pull timeout
```

## Lockfile-Keyed Dependency Cache

Configuring `[deps]` in `.spotexec.toml` avoids running `pnpm install`, `uv sync`, or `npm ci` from scratch on every slot or fresh Spot instance:

```toml
[deps]
lockfiles = ["pnpm-lock.yaml"]
install = "pnpm install --frozen-lockfile"
cache_dirs = ["node_modules", "packages/*/node_modules"]
hydrate = "auto"               # auto (reflink, else copy) | reflink | copy | hardlink
```

### How Cache Keys & Hydration Work

Each dependency archive is keyed by a cryptographic digest of:

- Lockfile contents + the `install` command + `cache_dirs` list;
- Runner OS and CPU architecture (`linux/amd64`, `linux/arm64`, etc.);
- Inside a `[container]`: the container image's **immutable image digest** (so native C++/Rust addons never cross Node/Python versions);
- Outside a container: the runner's AMI, bootstrap scripts, and `remote_path_setup`.

Hydration operates in two tiers:

1. **On the same runner (cross-slot):** Cached directories are cloned into each slot using filesystem copy-on-write `reflink` where supported (XFS, Btrfs, macOS APFS), falling back to a local copy. (`hardlink` is opt-in only because shared inodes allow a mutating test in Slot 1 to corrupt Slot 2's `node_modules`).
2. **Across ephemeral AWS Spot VMs:** The first VM to build a cache key compresses and uploads it to a private S3 bucket (`spotexec-cache-<account>-<region>`, Block Public Access enabled, 14-day lifecycle expiry). Subsequent VMs download and unpack the archive directly instead of running `install`.

> [!NOTE]
> Runner VMs hold **zero AWS credentials**. Whenever a slot needs to download or upload a dependency archive, the local `sx` CLI generates short-lived S3 presigned URLs scoped to that exact object key.

## Baked AMIs (`spotexec image bake`)

```sh
spotexec image bake aws      # launch temporary on-demand builder, run bootstraps, snapshot AMI
spotexec image list aws      # inspect active baked AMIs and bootstrap hashes
spotexec image clear aws     # deregister baked AMIs and delete backing EBS snapshots
```

`spotexec image bake` boots a temporary builder instance, runs `bootstrap_script`, installs Docker + Buildx, pre-pulls your project's public container image (plus any extra `--pull IMAGE` flags), runs `user_bootstrap_script`, strips per-host state (SSH host keys, cloud-init state, logs, watchdog timers), and registers a private AMI.

With `ami = "auto"` (the default), `spotexec` automatically selects the newest baked AMI matching your runner's current bootstrap hash and skips first-boot package installation—cutting cold Spot VM startup close to bare EC2 boot time. Changing `bootstrap_script`, `user_bootstrap_script`, `arch`, or `user` invalidates old hashes cleanly; `spotexec` retains only the 2 newest bakes per runner and garbage-collects older snapshots.

---

# AWS Spot Fleet Guide

`spotexec` drives EC2 Spot fleets directly from your workstation using hand-signed AWS SigV4 requests—there is no cloud control plane, no CloudFormation stack to maintain, and no IAM role attached to runner VMs.

**Fast path:** Sign in once with an administrative AWS session and run [`spotexec setup`](/docs/setup). It creates a dedicated IAM user (`spotexec-<hostname>`), attaches the least-privilege policy below, writes a non-expiring profile (`spotexec`), ensures the Spot service-linked role and default VPC exist, and verifies your regional Spot vCPU quota.

This guide documents every underlying AWS detail if you prefer to configure IAM, networking, or multi-size fleets manually.

## 1. AWS Sign-In Methods

`spotexec` resolves credentials in the following order:

1. `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN` (only when `profile` is unset on the runner).
2. `aws configure export-credentials --profile <profile>` when AWS CLI v2 is installed—supporting **IAM Identity Center (SSO)**, **`aws login`** (CLI 2.32+), **static IAM keys**, **`assume-role`**, and **`credential_process`**. If an SSO session has expired, `spotexec` reports `run aws sso login` clearly instead of silently falling back to another credential source.
3. Without the AWS CLI installed, `spotexec` reads `~/.aws` directly for static keys, `credential_process`, cached IAM Identity Center tokens (including token refresh), and active `aws login` sessions.

## 2. Least-Privilege IAM Policy

Run `spotexec setup --print-policy > spotexec-policy.json` at any time to export the exact policy. Statements 1–3 are required; statements 4–8 enable the S3 dependency cache, ECR project containers, `spotexec doctor` quota checks, and `spotexec image bake`:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": [
        "ec2:RunInstances", "ec2:CreateTags", "ec2:DescribeInstances", "ec2:TerminateInstances",
        "ec2:DescribeImages", "ec2:DescribeVpcs", "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups", "ec2:CreateSecurityGroup", "ec2:AuthorizeSecurityGroupIngress"
      ], "Resource": "*" },
    { "Effect": "Allow", "Action": "ssm:GetParameter",
      "Resource": "arn:aws:ssm:*::parameter/aws/service/canonical/*" },
    { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*",
      "Condition": { "StringEquals": { "iam:AWSServiceName": "spot.amazonaws.com" } } },
    { "Effect": "Allow", "Action": [
        "s3:CreateBucket", "s3:PutLifecycleConfiguration", "s3:ListBucket",
        "s3:GetObject", "s3:PutObject"
      ], "Resource": ["arn:aws:s3:::spotexec-cache-*", "arn:aws:s3:::spotexec-cache-*/*"] },
    { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" },
    { "Effect": "Allow", "Action": [
        "ecr:DescribeRepositories", "ecr:CreateRepository", "ecr:TagResource", "ecr:PutLifecyclePolicy",
        "ecr:BatchCheckLayerAvailability", "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer",
        "ecr:InitiateLayerUpload", "ecr:UploadLayerPart", "ecr:CompleteLayerUpload", "ecr:PutImage"
      ], "Resource": "arn:aws:ecr:*:*:repository/spotexec/*" },
    { "Effect": "Allow", "Action": "servicequotas:GetServiceQuota", "Resource": "*" },
    { "Effect": "Allow", "Action": [
        "ec2:StopInstances", "ec2:CreateImage", "ec2:DeregisterImage", "ec2:DeleteSnapshot"
      ], "Resource": "*" }
  ]
}
```

If your AWS account has never launched a Spot instance before and your identity cannot create service-linked roles, ask an administrator to run once:

```sh
aws iam create-service-linked-role --aws-service-name spot.amazonaws.com
```

## 3. Region Selection & Spot vCPU Quotas

Always set `region` explicitly on each `[runners.<name>]` block (`~/.aws/config` profile regions are intentionally ignored so runner behavior is deterministic across machines). Pick the region physically closest to you to keep SSH ControlMaster round-trips sub-second, and list 2–3 `instance_types` of the same CPU architecture (`c7a.xlarge`, `c6a.xlarge`, `m7a.xlarge`) so capacity shortages in one pool fall over immediately.

### Checking Spot vCPU Quotas (`L-34B43A08`)

New AWS accounts often start with a very low *All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests* quota (sometimes 5 vCPUs). Ensure your quota covers `max_instances × vCPUs` across your fleets:

```sh
# Inspect current Spot vCPU quota
aws service-quotas get-service-quota --profile spotexec --region us-east-2 \
  --service-code ec2 --quota-code L-34B43A08 --query Quota.Value

# Request an increase (requires an admin session)
aws service-quotas request-service-quota-increase --region us-east-2 \
  --service-code ec2 --quota-code L-34B43A08 --desired-value 64
```

## 4. Networking & Security Groups

By default, `spotexec` launches instances into the region's **default VPC**, assigns a public IPv4 address, and manages a `spotexec-ssh` security group allowing inbound TCP `22` from `ssh_cidr` (`0.0.0.0/0` by default, using Ed25519 key-only authentication generated at `~/.config/spotexec/keys/id_ed25519`).

- **Locking down SSH ingress:** Set `ssh_cidr = "203.0.113.4/32"` if you have a static office/home IP.
- **Custom VPCs:** Set `subnet_id` to a public subnet (routing `0.0.0.0/0` to an Internet Gateway) and optionally `security_group_ids` allowing inbound TCP 22 from your workstation and outbound HTTPS for package registries, S3, and ECR.

## 5. What Gets Created in Your Account

| AWS Resource | Created When | Lifecycle & Cleanup |
| --- | --- | --- |
| IAM user `spotexec-<hostname>` (`/spotexec/`) | `spotexec setup` | Delete user in IAM to revoke that machine. |
| Spot EC2 instances (`spotexec-runner`, `spotexec-owner` tags) | On demand via `sx` or `spotexec up` | Self-terminate on idle (`idle_timeout`) or `max_lifetime`; `spotexec down` terminates immediately. |
| Security group `spotexec-ssh` | First launch without `security_group_ids` | Reusable across runs; delete manually if uninstalling. |
| S3 bucket `spotexec-cache-<account>-<region>` | First run with `[deps]` | Private bucket; objects auto-expire after 14 days. |
| ECR repository `spotexec/<repo-id>` | First `[container]` Dockerfile build | Lifecycle policy retains only the 10 newest images. |
| Baked AMIs & EBS snapshots | `spotexec image bake` | Keeps 2 newest per runner; `spotexec image clear` deletes all. |

## 6. The 6-Stage Life of a Spot VM

1. **Launch.** `sx` submits a one-time Spot `RunInstances` call capped at `max_price` (default: on-demand price) with IMDSv2 required and `InstanceInitiatedShutdownBehavior=terminate`. Capacity errors immediately step through `instance_types`.
2. **Arm Watchdog First.** The very first lines of `cloud-init` user-data (`spotexec cloud-init <runner>`) install and start the `systemd` idle timer and `max_lifetime` hard timer—*before* SSH keys or bootstrap scripts run. Even a hung or broken bootstrap cannot leak an immortal VM.
3. **Bootstrap.** `bootstrap_script` (as `root`) and `user_bootstrap_script` (as `ubuntu`) run while holding the lifecycle `flock`. If either exits non-zero, the VM marks itself failed, returns `BOOTSTRAP_FAILED` with the tail of `/var/log/spotexec-bootstrap.log`, and pauses fleet launches for 1 hour so a broken script cannot burn money in a retry loop.
4. **Serve.** Concurrent `sx` sessions multiplex over SSH `ControlMaster`, acquire slot locks, and refresh the activity timestamp.
5. **Spot Interruption Handling.** `spotexec-helper` polls IMDSv2 every 5s. On a 2-minute `spot/instance-action` notice, it marks the VM `draining` (finishing in-flight jobs while refusing new ones) and notifies `sx`, which immediately acquires or launches replacement capacity in the background. Any job cut off before completion is **replayed once** automatically (`--no-retry` disables replay for non-idempotent deploy scripts). Advisory `rebalance` notices are logged without draining to avoid unnecessary fleet churn.
6. **Self-Termination.** Once idle for `idle_timeout` (default `10m`), or immediately once a `draining` VM finishes its active slots, or at `max_lifetime` (`12h`), the VM powers itself off—which EC2 immediately transitions to `terminated`, stopping EBS and compute billing.

> [!TIP]
> As an account-level backstop against a rare Linux kernel panic that freezes `systemd` timers, add an **AWS Budgets** alert or a simple **EventBridge** rule terminating `spotexec-runner` instances older than 24 hours.

---

# Scaling, Slots & Pool Manager

Every `sx` invocation schedules cooperatively against shared local lockfiles (`~/.local/state/spotexec/`) and cloud instance tags—requiring zero central server or cloud coordinator.

```
  sx (worktree 1)    sx (worktree 2)    sx (worktree 3)    sx (worktree 4)
         │                  │                  │                  │
         └──────────── pool.json + pool.lock (local flock) ───────┘
                                       │
   ┌───────────────────────────────────┴───────────────────────────────────┐
   ▼                                                                       ▼
homelab (ssh · priority 10)                                 aws (aws-spot · priority 20)
 └─ [slot 1 ●][slot 2 ●][slot 3 ●][slot 4 ●]                 ├─ i-04a1 (oldest) [slot 1 ●][slot 2 ●] ◀─ bin-packed first
                                                             └─ i-08f2 (newest) [slot 1 ○][slot 2 ○] ──▶ idles out → $0/hr
```

## Client-Side Scheduling & Bin-Packing

1. **Static runners first.** `ssh` runners in the active pool are probed first in ascending `priority` order without blocking. Your local build server or Mac Studio absorbs work at zero cloud cost until all its slots are occupied.
2. **Oldest-first Spot bin-packing.** When jobs spill onto an `aws-spot` fleet, `sx` reads running instances (refreshing EC2 state when older than 8s or on error) and tries them **oldest-first** (ties broken by instance ID). Concentrating new jobs onto older VMs leaves newer VMs empty so their `idle_timeout` clocks run uninterrupted and scale the fleet back in automatically.
3. **Fair local queueing.** When every slot across the pool is busy, `sx` takes a ticket in a fair local queue ordered by `(priority, arrival_time)`. Interactive terminals display `queued #3 of 9`, while coding agents receive `queue_position` NDJSON events.
4. **Demand-driven scale-out.** Queued jobs register waiter files locally. Under `pool.lock`, `sx` launches new Spot instances only when waiting slot demand exceeds capacity already booting in the launch ledger, up to `max_instances`. Lower-priority fleets launch only for demand that higher-priority fleets cannot cover.
5. **Autonomous scale-in.** Each Spot VM powers itself off after `idle_timeout` (default `10m`) once no `spotexec-helper` session holds `/var/tmp/spotexec/lifecycle.lock`.

## Sizing Test Workers with `$SPOTEXEC_CPUS`

When a 16-vCPU Spot VM is configured with `max_slots = 4`, each slot's fair share is 4 vCPUs—exposed automatically via `$SPOTEXEC_CPUS`. Point your test runners at `$SPOTEXEC_CPUS` so four concurrent agents don't each spawn 16 workers (64 threads total) and thrash the runner:

```sh
# Vitest (Vitest 2+ needs --minWorkers=1 on small shares)
vitest run --minWorkers=1 --maxWorkers=$SPOTEXEC_CPUS

# Jest
jest --maxWorkers=$SPOTEXEC_CPUS

# Pytest (pytest-xdist)
pytest -n $SPOTEXEC_CPUS

# Go
go test -p $SPOTEXEC_CPUS ./...
```

When a single heavy build or benchmark needs the entire machine, pass `sx --exclusive <cmd>` (or set `exclusive = true` on a task in `.spotexec.toml`) to claim every slot on a host and set `SPOTEXEC_CPUS` to the full machine vCPU count.

## Mixing Small and Large Fleets

Define multiple fleets in `~/.config/spotexec/config.toml` to match instance shapes to workloads—for example, fast 2-vCPU instances for linters/unit tests and 8-vCPU instances for overflow or E2E suites:

```toml
default_pool = ["aws-fast", "aws-heavy"]

[runners.aws-fast]
type = "aws-spot"
region = "us-east-2"
profile = "spotexec"
instance_types = ["c7a.large", "c6a.large"]
max_slots = 1
max_instances = 4
priority = 10

[runners.aws-heavy]
type = "aws-spot"
region = "us-east-2"
profile = "spotexec"
instance_types = ["c7a.2xlarge", "m7a.2xlarge"]
max_slots = 4
max_instances = 2
priority = 20
```

## Optional Local Pool Manager (`spotexec daemon`)

`sx` scales from zero to `max_instances` and back to zero completely on its own. `spotexec daemon` is an **optional** background process running on **your laptop** (never in the cloud) that shares the exact same `pool.json` lockfiles as `sx`:

```sh
spotexec daemon start          # run in background; logs to ~/.local/state/spotexec/daemon/daemon.log
spotexec daemon status         # view warm instances, health probes, and recent scaling actions (--json)
spotexec daemon stop           # stop daemon; any running VMs simply idle out after idle_timeout
spotexec daemon run            # run in foreground (for launchd / systemd user services)
```

What `spotexec daemon` adds when running:

- **Warm capacity (`min_instances > 0`).** Keeps `min_instances` booted and refreshes their idle timers so the first `sx` command of the morning starts in milliseconds instead of waiting for EC2 boot. Fifteen minutes before a warm VM hits `max_lifetime`, the daemon drains it and boots a fresh replacement seamlessly.
- **Proactive Spot interruption replacement.** Probes fleet VMs every 30s; if a warm VM receives a 2-minute Spot notice while idle between `sx` runs, the daemon drains it and boots a replacement immediately.
- **Unhealthy VM replacement & circuit breaking.** Terminates and replaces VMs that fail to finish first-boot setup or stop responding for 8 minutes (throttled to 1 replacement per 30m if *all* VMs are unreachable due to local Wi-Fi outage, and pausing a fleet for 1 hour if 3 consecutive boots fail).
- **Safety reaping.** Reaps VMs past `max_lifetime` (if a watchdog ever failed), orphaned instances without IP addresses, and leftover `image bake` builders.

---

# Architecture & Security Model

`spotexec` is built around a deliberate systems constraint: **no SaaS backend, no cloud control plane, and no long-running cloud agent.** Everything required to sync worktrees, multiplex sessions, manage Spot fleets, and guarantee self-termination lives inside a single Go binary, an embedded helper, OpenSSH, and a 30-line `cloud-init` watchdog.

## System Topology

```
Local Workstation                                  Remote Runner (SSH Box or Spot VM)
─────────────────                                  ──────────────────────────────────
sx ─┬─ 1. Manifest: git ls-files + index SHA-1     spotexec-helper (uploaded on first contact)
    ├─ 2. OpenSSH ControlMaster ─────── SSH ────▶   ├─ flock concurrency tokens + isolated slots
    └─ 3. Framed stdio RPC ─────────────────────▶   ├─ applies missing-blob gzip tar delta
                                                    ├─ hydrates [deps] (reflink / presigned S3)
EC2 / S3 / ECR / SSM APIs ◀── Hand-signed SigV4     ├─ runs command in dedicated process group
  RunInstances / Describe / Terminate               └─ hashes workspace post-run → mutation list
  tags: spotexec-runner, spotexec-owner
                                                   cloud-init systemd timer ──▶ poweroff (terminate)
```

### The 6 Steps of an `sx` Invocation

1. **Build worktree manifest.** Enumerates files Git tracks or would track (honoring `.gitignore`, `.spotexecignore`, and `sync.exclude`) alongside their Git SHA-1 blob IDs. Clean tracked files reuse `.git/index` blob IDs directly without reading file contents from disk.
2. **Acquire an isolated slot.** Tries configured runners in pool priority order with non-blocking `flock` token acquisition. If every slot is occupied, joins the fair local queue and registers demand so `aws-spot` fleets scale out up to `max_instances`.
3. **Stream missing-blob delta.** `spotexec-helper` compares the manifest against the slot's current files and requests only missing SHA-1 blobs, streamed as a single gzip'd tar archive over the multiplexed SSH channel. An unchanged worktree hits a manifest fingerprint match and transfers **zero bytes**.
4. **Hydrate dependencies.** Restores `cache_dirs` (`node_modules`, `.venv`, etc.) from the lockfile-keyed cache via copy-on-write `reflink` (or S3 presigned download across VMs), running your `install` command only on a cache miss.
5. **Execute with bidirectional heartbeats.** Runs the command inside its own process group (or `[container]` Docker image), streaming `stdout`/`stderr` back live while exchanging RPC heartbeats in both directions.
6. **3-way content-ID write-back.** Hashes the workspace after exit and returns created, modified, and deleted paths. `sx` applies mutations atomically only if your local file still matches the synced baseline SHA-1—saving concurrent local edits from ever being clobbered.

## Failure Modes & Responsibility Matrix

| Scenario / Failure Mode | How It Is Handled |
| --- | --- |
| **Launching & discovering VMs** | Local `sx` calls EC2 directly with your credentials; instances are discovered via `spotexec-runner` and `spotexec-owner` tags (`pool.json` is only an 8s address cache). |
| **Laptop lid closed / Wi-Fi drops mid-run** | After **45s** of missing RPC heartbeats, `spotexec-helper` kills the command process group and releases its shared `flock` on `/var/tmp/spotexec/lifecycle.lock`, restarting the VM's idle countdown. |
| **Preventing shutdown mid-job** | Active helper sessions hold a shared `flock` on `/var/tmp/spotexec/lifecycle.lock`. The `systemd` idle watchdog requires an **exclusive** lock before shutting down. |
| **Scaling back to $0.00/hr** | Installed by `cloud-init` *before* bootstrap scripts run, the `systemd` timer powers off the VM after `idle_timeout` or unconditionally at `max_lifetime`. `InstanceInitiatedShutdownBehavior=terminate` turns power-off into immediate EC2 termination. |
| **AWS 2-minute Spot notice** | Helper polls IMDSv2 every 5s, marks the VM `draining`, and alerts `sx`, which immediately acquires/boots replacement capacity and replays the run once if the VM terminates first. |
| **Concurrent agents booting a fleet** | Local `flock`-protected launch ledger in `pool.json` paired with idempotent EC2 `ClientToken` parameters prevents duplicate Spot launches. |
| **Hung first-boot script** | Bounded by `boot_timeout`; VMs still unready after `2 × boot_timeout + 2m` are reaped by ID or killed by their own `max_lifetime` timer. |

## Security Model & Trust Boundaries

- **Zero AWS credentials on VMs.** Spot VMs launch with **no IAM instance profile** and require IMDSv2. Access to S3 dependency archives and ECR layers uses short-lived presigned URLs scoped to exact objects in your account.
- **Cloned `.spotexec.toml` files cannot hijack connections.** Repository configs are forbidden from setting `type`, `host`, `hosts`, `ssh_options`, `identity_file`, `work_root`, `profile`, `region`, `subnet_id`, `security_group_ids`, `ssh_cidr`, `user`, or `cache_bucket`. Project fleets must `extends` a global runner you defined.
- **Strict write-back path jailing.** Every path returned by a runner is validated locally:
  - Must be clean and relative, with **no `.git` path segment** under any case folding;
  - Must not traverse any symlink;
  - Modified/deleted paths must exist in the synced manifest (a runner cannot request or overwrite `~/.aws/credentials`);
  - Newly created paths must not be gitignored, excluded, or already present locally.
- **Secrets stay off command lines.** Forwarded `env_allow` variables are written to a mode `0600` file on the runner rather than passed in `ps`-visible process arguments.
- **Worktree trust note.** Because write-back exists to let remote formatters and codegen modify tracked files, treat running untrusted repositories on a **shared `ssh` runner** with the same care as CI configs, and inspect `git status` when working against untrusted hosts.

## Key Design Decisions & Known Limits

- **Why no `rsync`?** macOS ships `openrsync` while Linux distributions ship GNU `rsync 3.x` with subtle flag incompatibilities. Streaming a missing-blob tar archive over framed stdio RPC avoids external tool skew and guarantees byte-exact baseline tracking for 3-way write-back.
- **Why Git SHA-1 blob IDs instead of BLAKE3/xxHash?** Clean tracked files already have precomputed SHA-1 blob IDs inside `.git/index`, letting `sx` build manifests for large monorepos without reading unchanged files from disk.
- **Known limits:**
  - Process cleanup uses process groups (plus working-directory sweeps on Linux) rather than cgroups; a background daemon that `setsid`s *and* `chdir`s outside the slot workspace can survive until VM shutdown.
  - Each workstation generates its own `owner-id` and manages its own Spot VMs independently (copy `~/.config/spotexec/owner-id` across machines only if you intentionally want shared EC2 tag visibility).
  - Case-only directory renames on case-insensitive macOS filesystems update files inside the existing directory casing; case-only *file* renames apply cleanly.

---

# Configuration Reference

`spotexec` splits configuration into two TOML files so credentials and hardware stay private while project tasks and caches travel with Git:

| File | Purpose | Committed to Git? |
| --- | --- | --- |
| `~/.config/spotexec/config.toml` | Your runners, SSH hosts, AWS accounts, regions, VPCs, and cost caps | **Never** |
| `.spotexec.toml` (repo root) | How this project syncs, caches dependencies, builds containers, and runs tasks | **Yes** |

### Precedence Order (Highest First)

1. CLI flags (`--timeout`, `-r`, `--sync`, `--pool`, etc.)
2. `SPOTEXEC_*` environment variables (`SPOTEXEC_TIMEOUT`, `SPOTEXEC_SYNC`, `SPOTEXEC_RUNNER`, `SPOTEXEC_POOL`, `SPOTEXEC_PRIORITY`, `SPOTEXEC_JSON=1`, `SPOTEXEC_AGENT=1`)
3. `.spotexec.toml` in the repository root
4. `~/.config/spotexec/config.toml`
5. Built-in defaults

*(Exceptions: `-r` / `SPOTEXEC_RUNNER` overrides any pool; `--json` / `SPOTEXEC_JSON=1` overrides `--pretty`; a task's explicit `pool` overrides `SPOTEXEC_POOL`.)*

---

## Global Config: `~/.config/spotexec/config.toml`

```toml
version = 1
default_pool = ["homelab", "aws"]   # runners tried in priority order
# local_fallback = false            # run locally when no runner is reachable

[runners.homelab]
type = "ssh"
hosts = ["me@192.168.1.50", "me@homelab.tailnet.ts.net"]
max_slots = 4
priority = 10

[runners.aws]
type = "aws-spot"
region = "us-east-2"
profile = "spotexec"
instance_types = ["c7a.4xlarge", "c6a.4xlarge", "m7a.4xlarge"]
max_slots = 8
max_instances = 4
min_instances = 0                 # kept warm only while `spotexec daemon` runs
priority = 20
idle_timeout = "10m"
max_lifetime = "12h"
volume_gb = 64
hourly_cost = 0.25                # used by `spotexec monitor` for $/hr burn rate
bootstrap_script = """
apt-get update -o DPkg::Lock::Timeout=300 && apt-get install -y build-essential
"""
user_bootstrap_script = """
curl -fsSL https://mise.run | sh
"""
remote_path_setup = 'export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"'
```

### Common Runner Keys

| Key | Default | Description |
| --- | --- | --- |
| `type` | *(required)* | `"ssh"`, `"aws-spot"`, or `"sim"` (local Docker simulator). |
| `max_slots` | `vCPUs ÷ 4` (min `1`) | Concurrent isolated slots per machine (`1`–`44`). |
| `priority` | `0` | Lower numbers run first; higher numbers receive overflow demand. |
| `work_root` | `~/.cache/spotexec` | Remote directory storing the helper binary, slots, and dependency caches. |
| `remote_path_setup` | *(none)* | Shell snippet sourced before commands outside `[container]` (e.g. `mise` or `nvm` PATH). |
| `ssh_options` | *(none)* | Extra `-o Key=Value` options passed to OpenSSH. |
| `hourly_cost` | *(none)* | Estimated USD/hour per instance displayed in `spotexec monitor`. |

### `ssh` Runner Keys

| Key | Description |
| --- | --- |
| `host` / `hosts` | Single SSH destination or list of routes to the same machine (`user@host`, hostname, or `~/.ssh/config` alias), dialed in parallel with a 500ms preference window for earlier entries. |
| `user` | SSH login username when omitted from `host`. |
| `identity_file` | Path to private key (defaults to `~/.ssh/config` resolution). |

### `aws-spot` Runner Keys

| Key | Default | Description |
| --- | --- | --- |
| `region` | `AWS_REGION` / `us-east-1` | Target EC2 region (always set explicitly). |
| `profile` | *(env / default)* | AWS credentials/config profile name (`spotexec` after `spotexec setup`). |
| `instance_types` | `c7a.4xlarge`, `c6a.4xlarge`, `m7a.4xlarge` | Instance types tried in order on Spot capacity errors (must share one CPU architecture). |
| `arch` | Derived from first type | `"amd64"` or `"arm64"` (Graviton). |
| `max_instances` | `4` | Maximum concurrent VMs in this fleet. |
| `min_instances` | `0` | Warm VM floor maintained while `spotexec daemon` is running. |
| `idle_timeout` | `"10m"` | Duration without active slots before the VM powers itself off. |
| `max_lifetime` | `"12h"` | Unconditional VM lifetime cap (must exceed `boot_timeout`). |
| `boot_timeout` | `"15m"` | Maximum time allowed for first-boot scripts before the VM is marked failed. |
| `bootstrap_script` | *(none)* | First-boot root `bash -e` script run after the watchdog is armed. |
| `user_bootstrap_script` | *(none)* | First-boot login-shell script run as `user` after `bootstrap_script`. |
| `max_price` | On-demand price | Maximum Spot bid in USD/hour as a string (e.g. `"0.40"`). |
| `on_demand_fallback` | `false` | Fall back to On-Demand if all Spot pools return insufficient capacity. |
| `volume_gb` | `64` | Root `gp3` EBS volume size in GiB (minimum `8`). |
| `ami` | `"auto"` | `"auto"` (newest matching baked AMI, else Ubuntu 24.04), `"base"`, or explicit `"ami-..."`. |
| `user` | `"ubuntu"` | SSH login user for the AMI. |
| `subnet_id`, `security_group_ids` | Default VPC + `spotexec-ssh` | Custom public subnet and security groups. |
| `ssh_cidr` | `"0.0.0.0/0"` | Allowed source CIDR for the managed `spotexec-ssh` group (Ed25519 key auth only). |
| `cache_bucket` | `"auto"` | `"auto"` (`spotexec-cache-<acct>-<region>`), custom S3 bucket name, or `"off"`. |

---

## Project Config: `.spotexec.toml`

Run `spotexec init` at the root of a repository to auto-generate `.spotexec.toml` for npm, pnpm, Yarn, Bun, Go, Rust, or Python projects:

```toml
version = 1
timeout = "5m"                 # per-command timeout (exits 124)
grace_period = "5s"            # delay between SIGTERM and SIGKILL
queue_timeout = "10m"          # max wait for a busy pool
env_allow = ["NODE_OPTIONS"]   # local env vars forwarded via 0600 file
env = { TZ = "UTC" }           # static env vars set on every remote run
pool = ["aws"]                 # optional override of global default_pool

[sync]
mode = "auto"                  # auto (write back) | in (discard remote edits) | none
writeback = "always"           # always | on-success | never
exclude = [".env*", "*.pem"]   # extra excludes on top of .gitignore & .spotexecignore
pull_artifacts = ["coverage/"] # gitignored outputs copied back automatically

[deps]
lockfiles = ["pnpm-lock.yaml"]
install = "pnpm install --frozen-lockfile"
cache_dirs = ["node_modules", "packages/*/node_modules"]
hydrate = "auto"               # auto (reflink, else copy) | reflink | copy | hardlink

[container]
dockerfile = ".spotexec/Dockerfile"

[tasks.test]
run = "pnpm vitest run --minWorkers=1 --maxWorkers=$SPOTEXEC_CPUS {args}"

[tasks.typecheck]
run = "pnpm tsc --noEmit"

[tasks.verify]
parallel = ["typecheck", "test"]
fail_fast = false

[hook]
when_load_above = 0.6          # Claude Code hook offloads only when laptop is busy
```

### `[tasks.<name>]` Keys

| Key | Description |
| --- | --- |
| `run` | Shell command string. `{args}` expands positional arguments passed after `sx <task>`; if omitted, extra args are appended. |
| `parallel` | List of non-parallel task names to fan out concurrently across separate slots (`sx verify`). |
| `fail_fast` | Cancel sibling parallel tasks immediately when one fails. |
| `timeout`, `sync`, `exclusive` | Per-task execution overrides. |
| `pool`, `priority` | Route specific tasks (e.g. `e2e`) to larger fleets or higher queue priority. |

### Sandboxed Project Fleets (`extends`)

A repository can customize VM sizing or bootstrap scripts by extending one of your global runners—without ever being allowed to alter SSH targets, credentials, regions, or VPCs:

```toml
pool = ["ci-big"]

[runners.ci-big]
extends = "aws"                # must reference a runner in ~/.config/spotexec/config.toml
instance_types = ["c7a.2xlarge", "m7a.2xlarge"]
max_slots = 2
max_instances = 3
```

---

# CLI & File Reference

## Command Cheat Sheet

```
sx [flags] <command...>                      run remotely (shorthand for `spotexec run --`)
sx <task> [args...]                          run a named task; parallel tasks fan out across slots
spotexec run [flags] -- <cmd>                sync worktree, execute remotely, write back changes
spotexec exec [flags] -- <cmd>               execute in the last-used slot without syncing
spotexec tasks                               list tasks defined in .spotexec.toml
spotexec sync [flags]                        pre-sync a slot (plus container build & [deps] install)
spotexec sync-plan [--json]                  dry-run manifest: file count, bytes, largest files, excludes
spotexec pull <glob>... [--dest DIR]         pull gitignored artifacts from the last-used slot
spotexec status [runner...] [--json]         inspect runners, Spot VMs, active slot holders, idle timers
spotexec monitor [runner...] [--interval 2s] live terminal dashboard of slot load and $/hr burn rate
spotexec up [runner] [--count N]             pre-warm Spot VMs now and wait until ready
spotexec down [runner...] [--all]            terminate fleet VMs immediately (and leftover bake builders)
spotexec daemon start|stop|run               manage optional local pool manager process
spotexec daemon status [--json]              inspect warm pool health, probes, and scaling history
spotexec container init [--path P] [--force] scaffold .spotexec/Dockerfile and wire [container] config
spotexec container build [run flags]         build project container on a runner now (pushes to ECR on AWS)
spotexec image bake [runner] [--pull IMG]    bake a fast-boot AMI from the runner's bootstrap scripts
spotexec image list|clear [runner]           list or deregister baked AMIs and snapshots
spotexec setup [host...] [flags]             configure this machine (and others over SSH) + scoped IAM keys
spotexec doctor [runner...]                  end-to-end check of SSH, helper, config, AWS auth, VPC, quota
spotexec unlock [runner[/id]] (--slot N | --all)
                                             kill a wedged command and reclaim its slot lock
spotexec prune [runner[/id]...] [--older-than 7d] [--dry-run]
                                             clean up unused dependency caches and idle slot workspaces
spotexec init [--global] [--force]           generate starter .spotexec.toml (or global config.toml)
spotexec hook install [--claude] [--codex] [--shared|--global] [--uninstall]
                                             install or remove coding agent command hooks
spotexec hook check '<command>'              test whether the agent hook offloads or keeps a command local
spotexec cloud-init [runner]                 print the exact cloud-init user-data script for an AWS runner
spotexec completion bash|zsh|fish|powershell generate shell tab-completion script
spotexec version                             print binary and embedded helper version
```

## Files & State Paths

| Path | Purpose |
| --- | --- |
| `~/.config/spotexec/config.toml` | Global runner configuration (overridable via `SPOTEXEC_CONFIG`, `SPOTEXEC_CONFIG_DIR`, or `XDG_CONFIG_HOME`). |
| `~/.config/spotexec/keys/` | Auto-generated Ed25519 SSH keypair (`id_ed25519`) injected into ephemeral Spot VMs via user-data. |
| `~/.config/spotexec/owner-id` | Per-machine random identifier applied via the `spotexec-owner` EC2 tag so workstations isolate their VMs. |
| `~/.aws/credentials` `[spotexec]`, `~/.aws/config` `[profile spotexec]` | Scoped IAM user access key (`0600`) and region written by `spotexec setup`. |
| `~/.local/state/spotexec/` | Local `flock`-protected pool ledgers (`pool.json`), fair queue tickets (`waiters/`), per-instance `known_hosts`, and `daemon` logs (`SPOTEXEC_STATE_DIR` / `XDG_STATE_HOME`). |
| `~/.cache/spotexec/` | Local OpenSSH `ControlMaster` sockets (`cm/`) and helper upload markers. |
| `.spotexec.toml`, `.spotexecignore` | Committed per-repository configuration and extra sync exclusion patterns. |
| `.spotexec/Dockerfile` | Committed project toolchain Dockerfile created by `spotexec container init`. |
| `<git-dir>/spotexec/conflicts/<run-id>/` | Saved remote versions of files that conflicted with concurrent local edits during 3-way write-back (`.spotexec/conflicts/` outside Git). |
| **Runner:** `~/.cache/spotexec/` | Remote `work_root` holding `bin/spotexec-helper`, per-slot workspaces, and lockfile dependency caches. |
| **Runner:** `/var/tmp/spotexec/`, `/var/log/spotexec-bootstrap.log` | Ephemeral VM `lifecycle.lock`, activity clock, Spot drain markers, and first-boot log. |

---

# Operations & Troubleshooting

## Day-2 Diagnostic & Operational Commands

```sh
spotexec doctor                      # check config, SSH ControlMaster, embedded helpers, IAM, AMI, VPC, quota
spotexec status [--json]             # list runners, running Spot VMs, slot lock holders, and idle countdowns
spotexec monitor                     # live 2s-refresh dashboard of active slots, CPU/RAM free, and $/hr burn
spotexec up aws --count 2            # pre-warm 2 Spot VMs immediately (respects max_instances)
spotexec down aws                    # terminate all VMs in a fleet now (or --all across every fleet)
spotexec unlock aws/i-0abc --slot 3  # terminate a wedged process and free slot 3 (--all reclaims all slots)
spotexec prune --older-than 7d       # prune stale dependency archives and idle slot workspaces on runners
spotexec sync-plan                   # dry-run manifest: inspect total files, bytes, top 10 largest files
spotexec cloud-init aws              # inspect the exact cloud-init user-data script sent to EC2
```

## Common Symptoms & Fixes

| Symptom / Code | Cause & Resolution |
| --- | --- |
| `BOOTSTRAP_FAILED` | Your `bootstrap_script` or `user_bootstrap_script` exited non-zero or exceeded `boot_timeout`. Read the printed tail of `/var/log/spotexec-bootstrap.log`, fix the script in `config.toml`, and run `spotexec down <runner>` (which terminates the failed VM and clears the 1-hour fleet launch hold). |
| Jobs stay `queued` without launching Spot VMs | Check `max_instances` in `config.toml`, run `spotexec daemon status` to see if the fleet is paused after repeated boot failures, and run `spotexec doctor` to verify your regional Spot vCPU quota (`L-34B43A08`). Failed launches back off for 20s. |
| `InsufficientInstanceCapacity` | EC2 is temporarily out of Spot capacity for that shape in that AZ/region. Add 2–3 fallback shapes to `instance_types` (e.g. `["c7a.xlarge", "c6a.xlarge", "m7a.xlarge"]`), switch regions, or set `on_demand_fallback = true`. |
| Sync is slow or rejected as too large | Run `spotexec sync-plan` to list the 10 largest files being included. Add large fixtures, datasets, or untracked build directories to `.spotexecignore` or `[sync].exclude`. |
| `SYNC_CONFLICT` warning after a run | You or an agent modified a file locally while the remote command also modified it. Your local file was preserved; diff or merge against `.git/spotexec/conflicts/<run-id>/<path>`. |
| Integration test fails trying to reach `localhost` | The command needs a local database or dev server on your laptop. Prefix with `SPOTEXEC_LOCAL=1 <cmd>` (when using the Claude hook) or invoke the command directly without `sx`. |
| Expired AWS SSO / `aws login` session | Run `aws sso login` (or `aws login`), or run `spotexec setup` once to provision a non-expiring scoped `spotexec` access key so sessions never expire mid-workflow. |

## Testing Without AWS Using the Local Spot Simulator

`spotexec` ships a built-in Docker Spot fleet simulator (`type = "sim"`) that executes the exact same `cloud-init` user-data, `sshd` transport, `flock` slot tokens, idle self-termination watchdog, and 2-minute Spot interruption flow locally in containers:

```toml
# Example simulator fleet in ~/.config/spotexec/config.toml
[runners.sim]
type = "sim"
max_slots = 2
max_instances = 3
idle_timeout = "1m"
```

```sh
spotexec sim build                   # build the local sshd + cloud-init simulator image
sx -r sim uname -a                   # scales out a container instance over real SSH
spotexec sim interrupt sim           # inject a synthetic 2-minute Spot interruption notice
spotexec sim rebalance sim           # inject an advisory EC2 rebalance recommendation
spotexec down sim                    # tear down simulated fleet containers
```