# Containers, Caches & Baked Images

Fast remote execution comes down to eliminating repeated setup work across slots and ephemeral VMs. `spotexec` provides three composable layers:

1. **Project containers (`[container]`)** — pin exact Node, Python, Go, Rust, Bun, or JVM toolchains in a committed Dockerfile built once and cached in ECR.
2. **Lockfile-keyed dependency caches (`[deps]`)** — restore `node_modules`, `.venv`, or build caches instantly via copy-on-write `reflink` locally and private S3 archives across Spot VMs.
3. **Baked AMIs (`spotexec image bake`)** — snapshot your host bootstrap so fresh Spot VMs come online in bare EC2 boot time.

## Project Containers

With `[container]` configured in `.spotexec.toml`, both dependency installation and every `sx` command run inside a Docker container on the runner. The runner host itself only needs Docker (installed automatically on AWS Spot VMs).

### Scaffold, Customize, and Build

```sh
spotexec container init              # detects stack, writes .spotexec/Dockerfile, updates .spotexec.toml
$EDITOR .spotexec/Dockerfile         # add system libraries (e.g. libvips, playwright deps, protobuf)
spotexec container build -r aws      # build remotely now and push to private ECR on AWS
sx pnpm test                         # runs inside the pinned image
```

`spotexec container init` inspects your repository manifests and writes a pinned Dockerfile:

| Detected Stack | Generated Base Image |
| --- | --- |
| Node.js (`npm`, `pnpm`, `yarn`) | `node:<major>-bookworm` (version from `.nvmrc`, `.node-version`, or `engines.node`; pins `pnpm` via `packageManager`) |
| Bun | `oven/bun:1` |
| Python with `uv` | `ghcr.io/astral-sh/uv:python<x.y>-bookworm` (version from `.python-version`) |
| Python | `python:<x.y>-bookworm` |
| Go | `golang:<x.y>-bookworm` (version from `go.mod`) |
| Rust | `rust:<version>-bookworm` (from `rust-toolchain.toml`, plus `clippy` and `rustfmt`) |
| Maven / Gradle | `eclipse-temurin:21-jdk` (adds `maven` if `./mvnw` is absent) |
| Fallback | `ubuntu:24.04` with `build-essential` |

> [!IMPORTANT]
> `spotexec container init` stages `.spotexec/Dockerfile` with `git add` so it is included in the Git worktree manifest. Commit `.spotexec/Dockerfile` alongside `.spotexec.toml` so every worktree and runner receives it.

### Dockerfile Authoring Rules

- **Install tools, never source code.** Your worktree is bind-mounted at runtime, and dependencies (`node_modules`, `.venv`) are hydrated from `[deps]`. Do not `COPY . .` in `.spotexec/Dockerfile`.
- **Install system-wide (e.g. `/usr/local`).** Commands execute as the non-root runner user with `HOME=/home/spotexec` mounted per slot so tool caches (`~/.cache`, Go build cache, Cargo registry) persist across runs. Anything installed into `/root` or baked into `/home/spotexec` inside the image will be hidden by the slot mount.
- **Content-addressed rebuilds.** `spotexec` hashes the Dockerfile and its `.dockerignore`-filtered build context. The image is rebuilt only when that hash changes, pushed to a private `spotexec/<repo-id>` ECR repository (retaining the 10 newest tags), and pulled by future Spot VMs without rebuilding.

```toml
[container]
dockerfile = ".spotexec/Dockerfile"   # or pull an existing image directly with: image = "node:22-bookworm"
# context = "."                       # defaults to the Dockerfile's directory
# options = ["--shm-size=1g"]         # extra docker run flags (useful for Chromium / Playwright)
# user = "runner"                     # "runner" (default; preserves file ownership) or "root"
# timeout = "20m"                     # image build/pull timeout
```

## Lockfile-Keyed Dependency Cache

Configuring `[deps]` in `.spotexec.toml` avoids running `pnpm install`, `uv sync`, or `npm ci` from scratch on every slot or fresh Spot instance:

```toml
[deps]
lockfiles = ["pnpm-lock.yaml"]
install = "pnpm install --frozen-lockfile"
cache_dirs = ["node_modules", "packages/*/node_modules"]
hydrate = "auto"               # auto (reflink, else copy) | reflink | copy | hardlink
```

### How Cache Keys & Hydration Work

Each dependency archive is keyed by a cryptographic digest of:

- Lockfile contents + the `install` command + `cache_dirs` list;
- Runner OS and CPU architecture (`linux/amd64`, `linux/arm64`, etc.);
- Inside a `[container]`: the container image's **immutable image digest** (so native C++/Rust addons never cross Node/Python versions);
- Outside a container: the runner's AMI, bootstrap scripts, and `remote_path_setup`.

Hydration operates in two tiers:

1. **On the same runner (cross-slot):** Cached directories are cloned into each slot using filesystem copy-on-write `reflink` where supported (XFS, Btrfs, macOS APFS), falling back to a local copy. (`hardlink` is opt-in only because shared inodes allow a mutating test in Slot 1 to corrupt Slot 2's `node_modules`).
2. **Across ephemeral AWS Spot VMs:** The first VM to build a cache key compresses and uploads it to a private S3 bucket (`spotexec-cache-<account>-<region>`, Block Public Access enabled, 14-day lifecycle expiry). Subsequent VMs download and unpack the archive directly instead of running `install`.

> [!NOTE]
> Runner VMs hold **zero AWS credentials**. Whenever a slot needs to download or upload a dependency archive, the local `sx` CLI generates short-lived S3 presigned URLs scoped to that exact object key.

## Baked AMIs (`spotexec image bake`)

```sh
spotexec image bake aws      # launch temporary on-demand builder, run bootstraps, snapshot AMI
spotexec image list aws      # inspect active baked AMIs and bootstrap hashes
spotexec image clear aws     # deregister baked AMIs and delete backing EBS snapshots
```

`spotexec image bake` boots a temporary builder instance, runs `bootstrap_script`, installs Docker + Buildx, pre-pulls your project's public container image (plus any extra `--pull IMAGE` flags), runs `user_bootstrap_script`, strips per-host state (SSH host keys, cloud-init state, logs, watchdog timers), and registers a private AMI.

With `ami = "auto"` (the default), `spotexec` automatically selects the newest baked AMI matching your runner's current bootstrap hash and skips first-boot package installation—cutting cold Spot VM startup close to bare EC2 boot time. Changing `bootstrap_script`, `user_bootstrap_script`, `arch`, or `user` invalidates old hashes cleanly; `spotexec` retains only the 2 newest bakes per runner and garbage-collects older snapshots.