# Architecture & Security Model

`spotexec` is built around a deliberate systems constraint: **no SaaS backend, no cloud control plane, and no long-running cloud agent.** Everything required to sync worktrees, multiplex sessions, manage Spot fleets, and guarantee self-termination lives inside a single Go binary, an embedded helper, OpenSSH, and a 30-line `cloud-init` watchdog.

## System Topology

```
Local Workstation                                  Remote Runner (SSH Box or Spot VM)
─────────────────                                  ──────────────────────────────────
sx ─┬─ 1. Manifest: git ls-files + index SHA-1     spotexec-helper (uploaded on first contact)
    ├─ 2. OpenSSH ControlMaster ─────── SSH ────▶   ├─ flock concurrency tokens + isolated slots
    └─ 3. Framed stdio RPC ─────────────────────▶   ├─ applies missing-blob gzip tar delta
                                                    ├─ hydrates [deps] (reflink / presigned S3)
EC2 / S3 / ECR / SSM APIs ◀── Hand-signed SigV4     ├─ runs command in dedicated process group
  RunInstances / Describe / Terminate               └─ hashes workspace post-run → mutation list
  tags: spotexec-runner, spotexec-owner
                                                   cloud-init systemd timer ──▶ poweroff (terminate)
```

### The 6 Steps of an `sx` Invocation

1. **Build worktree manifest.** Enumerates files Git tracks or would track (honoring `.gitignore`, `.spotexecignore`, and `sync.exclude`) alongside their Git SHA-1 blob IDs. Clean tracked files reuse `.git/index` blob IDs directly without reading file contents from disk.
2. **Acquire an isolated slot.** Tries configured runners in pool priority order with non-blocking `flock` token acquisition. If every slot is occupied, joins the fair local queue and registers demand so `aws-spot` fleets scale out up to `max_instances`.
3. **Stream missing-blob delta.** `spotexec-helper` compares the manifest against the slot's current files and requests only missing SHA-1 blobs, streamed as a single gzip'd tar archive over the multiplexed SSH channel. An unchanged worktree hits a manifest fingerprint match and transfers **zero bytes**.
4. **Hydrate dependencies.** Restores `cache_dirs` (`node_modules`, `.venv`, etc.) from the lockfile-keyed cache via copy-on-write `reflink` (or S3 presigned download across VMs), running your `install` command only on a cache miss.
5. **Execute with bidirectional heartbeats.** Runs the command inside its own process group (or `[container]` Docker image), streaming `stdout`/`stderr` back live while exchanging RPC heartbeats in both directions.
6. **3-way content-ID write-back.** Hashes the workspace after exit and returns created, modified, and deleted paths. `sx` applies mutations atomically only if your local file still matches the synced baseline SHA-1—saving concurrent local edits from ever being clobbered.

## Failure Modes & Responsibility Matrix

| Scenario / Failure Mode | How It Is Handled |
| --- | --- |
| **Launching & discovering VMs** | Local `sx` calls EC2 directly with your credentials; instances are discovered via `spotexec-runner` and `spotexec-owner` tags (`pool.json` is only an 8s address cache). |
| **Laptop lid closed / Wi-Fi drops mid-run** | After **45s** of missing RPC heartbeats, `spotexec-helper` kills the command process group and releases its shared `flock` on `/var/tmp/spotexec/lifecycle.lock`, restarting the VM's idle countdown. |
| **Preventing shutdown mid-job** | Active helper sessions hold a shared `flock` on `/var/tmp/spotexec/lifecycle.lock`. The `systemd` idle watchdog requires an **exclusive** lock before shutting down. |
| **Scaling back to $0.00/hr** | Installed by `cloud-init` *before* bootstrap scripts run, the `systemd` timer powers off the VM after `idle_timeout` or unconditionally at `max_lifetime`. `InstanceInitiatedShutdownBehavior=terminate` turns power-off into immediate EC2 termination. |
| **AWS 2-minute Spot notice** | Helper polls IMDSv2 every 5s, marks the VM `draining`, and alerts `sx`, which immediately acquires/boots replacement capacity and replays the run once if the VM terminates first. |
| **Concurrent agents booting a fleet** | Local `flock`-protected launch ledger in `pool.json` paired with idempotent EC2 `ClientToken` parameters prevents duplicate Spot launches. |
| **Hung first-boot script** | Bounded by `boot_timeout`; VMs still unready after `2 × boot_timeout + 2m` are reaped by ID or killed by their own `max_lifetime` timer. |

## Security Model & Trust Boundaries

- **Zero AWS credentials on VMs.** Spot VMs launch with **no IAM instance profile** and require IMDSv2. Access to S3 dependency archives and ECR layers uses short-lived presigned URLs scoped to exact objects in your account.
- **Cloned `.spotexec.toml` files cannot hijack connections.** Repository configs are forbidden from setting `type`, `host`, `hosts`, `ssh_options`, `identity_file`, `work_root`, `profile`, `region`, `subnet_id`, `security_group_ids`, `ssh_cidr`, `user`, or `cache_bucket`. Project fleets must `extends` a global runner you defined.
- **Strict write-back path jailing.** Every path returned by a runner is validated locally:
  - Must be clean and relative, with **no `.git` path segment** under any case folding;
  - Must not traverse any symlink;
  - Modified/deleted paths must exist in the synced manifest (a runner cannot request or overwrite `~/.aws/credentials`);
  - Newly created paths must not be gitignored, excluded, or already present locally.
- **Secrets stay off command lines.** Forwarded `env_allow` variables are written to a mode `0600` file on the runner rather than passed in `ps`-visible process arguments.
- **Worktree trust note.** Because write-back exists to let remote formatters and codegen modify tracked files, treat running untrusted repositories on a **shared `ssh` runner** with the same care as CI configs, and inspect `git status` when working against untrusted hosts.

## Key Design Decisions & Known Limits

- **Why no `rsync`?** macOS ships `openrsync` while Linux distributions ship GNU `rsync 3.x` with subtle flag incompatibilities. Streaming a missing-blob tar archive over framed stdio RPC avoids external tool skew and guarantees byte-exact baseline tracking for 3-way write-back.
- **Why Git SHA-1 blob IDs instead of BLAKE3/xxHash?** Clean tracked files already have precomputed SHA-1 blob IDs inside `.git/index`, letting `sx` build manifests for large monorepos without reading unchanged files from disk.
- **Known limits:**
  - Process cleanup uses process groups (plus working-directory sweeps on Linux) rather than cgroups; a background daemon that `setsid`s *and* `chdir`s outside the slot workspace can survive until VM shutdown.
  - Each workstation generates its own `owner-id` and manages its own Spot VMs independently (copy `~/.config/spotexec/owner-id` across machines only if you intentionally want shared EC2 tag visibility).
  - Case-only directory renames on case-insensitive macOS filesystems update files inside the existing directory casing; case-only *file* renames apply cleanly.