# spotexec (`sx`)

**Offload tests, typechecks, linters, E2E suites, and builds from local git worktrees to an SSH box or scale-to-zero AWS Spot fleet.**

Prefix any command with `sx` (shorthand for `spotexec run --`):

```console
$ sx pnpm vitest run -u src/auth.test.ts
 ✓ src/auth.test.ts (12 tests) 340ms
● 4.1s on aws#3 · synced 4 files (18 KB) · pulled 2 modified files
```

Files modified remotely (`prettier --write`, `eslint --fix`, `vitest -u`, codegen) return to your local worktree through a **3-way content-ID check**, so edits you or an agent make during the run are never overwritten.

---

## Why `spotexec` Exists: Coding Agents Drove Up CPU & Memory

Modern coding agents (Claude Code, Codex, and parallel worktree workflows) write code across 4–8 branches simultaneously—and then **verify** every edit by running unit tests, Playwright E2E browsers, `tsc --noEmit`, ESLint, Prettier, and builds in every worktree at once:

- Node / Vitest / Jest worker pools (`2–6 GB` RAM per worktree)
- Headless Chromium / Playwright E2E browsers (`3–8 GB` RAM per run)
- `tsc --noEmit`, ESLint, Prettier, Rust `cargo test`, Go `go test -race`, Gradle/Maven verify

Running four or five concurrent agent verification loops on a 32 GB laptop pushes memory into swap, spikes CPU fans, and stalls your editor.

### Two Core Benefits

1. **Low cost (~65–70% Spot savings):** Typical 2 vCPU / 4–8 GB AWS Spot workers (`t3a.large` ~$0.024/hr, `c6i.large` ~$0.028/hr, `c7a.large` ~$0.035/hr, `m7a.large` ~$0.038/hr) cost 65–70% less than On-Demand and scale back to `$0.00/hr` when idle.
2. **Run far more on your local hardware:** Offloading heavy verification lets you run 4–8+ parallel coding agents, editors, and local dev servers on your Mac without thermal throttling or swap thrashing.

### Why Spot Works Here: Offload the Heavy Task, Not the Whole Agent

Running an entire agent workflow on Spot instances is fragile—if AWS reclaims the VM mid-session, agent state and progress are lost. `spotexec` keeps your agents, Git worktrees, and editor on your Mac and only dispatches individual stateless verification tasks. If a Spot VM is reclaimed mid-check, nothing is lost: `sx` replays that single task on another slot.

`spotexec` moves those heavy verification commands off your laptop while keeping `git`, `rg`, file edits, and `localhost` dev servers local:

```
Local Laptop (4+ Agent Worktrees)              Remote Capacity (SSH or AWS Spot)
─────────────────────────────────              ─────────────────────────────────
wt/auth   ─┐                                   homelab (ssh, priority 10)
wt/api    ─┼─ sx / Claude Bash Hook ── SSH ──▶  └─ [slot 1 ●][slot 2 ●][slot 3 ●][slot 4 ●]
wt/e2e    ─┤  (Git-blob delta sync)
wt/lint   ─┘                                   aws-spot (overflow, priority 20)
                                                ├─ i-04a1 [slot 1 ●][slot 2 ●]
                                                └─ scales to $0.00/hr after idle_timeout
```

---

## Core Properties

- **Drop-in two-letter prefix.** `sx <command>` streams stdout/stderr live, preserves exit codes, and rewrites remote paths in stack traces back to local paths.
- **Scale-to-zero AWS Spot or static SSH.** Fill a homelab box first (`priority = 10`), spill overflow to AWS Spot VMs (`priority = 20`), bin-pack jobs onto the oldest VMs, and let idle VMs self-terminate via a cloud-init `systemd` watchdog.
- **3-way safe write-back.** Compares local files against the exact Git blob SHA-1 baseline that was synced. If you edited a file mid-run, your local edit is kept and the remote output is saved under `.git/spotexec/conflicts/<run-id>/`.
- **Delta sync & lockfile caches.** Clean tracked files reuse `.git/index` blob IDs without reading disk; only missing blobs travel over SSH ControlMaster. Dependencies hydrate via `reflink` (CoW) locally and private S3 archives across Spot VMs.
- **Zero cloud control plane.** Single Go binary with embedded Linux/macOS (`amd64`/`arm64`) helpers and hand-signed AWS SigV4 requests. No SaaS account, no daemon in AWS, and **zero IAM credentials on runner VMs**.
- **Built for coding agents.** Emits structured NDJSON events on non-TTY stderr, and includes `spotexec hook install --claude` to rewrite heavy Bash commands automatically when laptop load exceeds `when_load_above`.

---

## Install on macOS (Recommended)

```sh
brew install daintreehq/tap/spotexec
```

Installs `spotexec`, the `sx` shorthand alias, shell completions, and embedded cross-compiled remote helpers.

### Linux & Source Builds

```sh
# Homebrew on Linux
brew install daintreehq/tap/spotexec

# Debian / Ubuntu, Fedora / RHEL, Alpine (from GitHub Releases)
sudo apt install ./spotexec_*_linux_amd64.deb
sudo dnf install ./spotexec_*_linux_amd64.rpm
sudo apk add --allow-untrusted ./spotexec_*_linux_amd64.apk

# From source (Go 1.26+; cross-compiles and embeds remote helpers)
git clone https://github.com/daintreehq/spotexec.git && cd spotexec
make install
```

## 3-Step Quickstart

```sh
# 1. Provision runner config and dedicated per-machine IAM key (once per machine)
aws login
spotexec setup

# 2. Initialize project config in your repository (once per repo)
cd ~/code/my-app
spotexec init
spotexec doctor

# 3. Prefix heavy commands with sx (or install the Claude Code hook)
sx pnpm test
sx pnpm prettier --write .
spotexec hook install --claude
```

## Documentation Index

- [Introduction & Quickstart](/docs.md) — Overview, requirements, install, and first run
- [Setting Up Machines](/docs/setup.md) — `spotexec setup`, SSH vs. AWS Spot runners, multi-machine provisioning, key rotation
- [Running Commands & Write-back](/docs/running.md) — Flags, runner environment variables, exit codes, 3-way write-back, artifacts
- [Coding Agents & Hooks](/docs/agents.md) — Claude Code & Codex hooks, load-aware offloading, NDJSON event stream
- [Containers, Caches & Baked Images](/docs/environments.md) — Project Dockerfiles, `reflink`/S3 dependency cache, `image bake`
- [AWS Spot Fleet Guide](/docs/aws.md) — IAM policy, regions, Spot vCPU quotas, VPC networking, Spot VM lifecycle
- [Scaling, Slots & Pool Manager](/docs/scaling.md) — Slot bin-packing, queueing, `$SPOTEXEC_CPUS`, optional local `daemon`
- [Architecture & Security Model](/docs/architecture.md) — RPC lifecycle, cloud-init watchdog, security boundaries, known limits
- [Configuration Reference](/docs/configuration.md) — Complete `config.toml` and `.spotexec.toml` reference
- [CLI & File Reference](/docs/reference.md) — Every command and local/remote state path
- [Operations & Troubleshooting](/docs/troubleshooting.md) — Day-2 commands, common error codes, Docker Spot simulator