Containers, Caches & Baked Images

Fast remote execution comes down to eliminating repeated setup work across slots and ephemeral VMs. spotexec provides three composable layers:

1. Project containers ([container]) — pin exact Node, Python, Go, Rust, Bun, or JVM toolchains in a committed Dockerfile built once and cached in ECR.
2. Lockfile-keyed dependency caches ([deps]) — restore node_modules, .venv, or build caches instantly via copy-on-write reflink locally and private S3 archives across Spot VMs.
3. Baked AMIs (spotexec image bake) — snapshot your host bootstrap so fresh Spot VMs come online in bare EC2 boot time.

Project Containers#

With [container] configured in .spotexec.toml, both dependency installation and every sx command run inside a Docker container on the runner. The runner host itself only needs Docker (installed automatically on AWS Spot VMs).

Scaffold, Customize, and Build#

spotexec container init              # detects stack, writes .spotexec/Dockerfile, updates .spotexec.toml
$EDITOR .spotexec/Dockerfile         # add system libraries (e.g. libvips, playwright deps, protobuf)
spotexec container build -r aws      # build remotely now and push to private ECR on AWS
sx pnpm test                         # runs inside the pinned image

spotexec container init inspects your repository manifests and writes a pinned Dockerfile:

Dockerfile Authoring Rules#

• Install tools, never source code. Your worktree is bind-mounted at runtime, and dependencies (node_modules, .venv) are hydrated from [deps]. Do not COPY . . in .spotexec/Dockerfile.
• Install system-wide (e.g. /usr/local). Commands execute as the non-root runner user with HOME=/home/spotexec mounted per slot so tool caches (~/.cache, Go build cache, Cargo registry) persist across runs. Anything installed into /root or baked into /home/spotexec inside the image will be hidden by the slot mount.
• Content-addressed rebuilds. spotexec hashes the Dockerfile and its .dockerignore-filtered build context. The image is rebuilt only when that hash changes, pushed to a private spotexec/<repo-id> ECR repository (retaining the 10 newest tags), and pulled by future Spot VMs without rebuilding.

[container]
dockerfile = ".spotexec/Dockerfile"   # or pull an existing image directly with: image = "node:22-bookworm"
# context = "."                       # defaults to the Dockerfile's directory
# options = ["--shm-size=1g"]         # extra docker run flags (useful for Chromium / Playwright)
# user = "runner"                     # "runner" (default; preserves file ownership) or "root"
# timeout = "20m"                     # image build/pull timeout

Lockfile-Keyed Dependency Cache#

Configuring [deps] in .spotexec.toml avoids running pnpm install, uv sync, or npm ci from scratch on every slot or fresh Spot instance:

[deps]
lockfiles = ["pnpm-lock.yaml"]
install = "pnpm install --frozen-lockfile"
cache_dirs = ["node_modules", "packages/*/node_modules"]
hydrate = "auto"               # auto (reflink, else copy) | reflink | copy | hardlink

How Cache Keys & Hydration Work#

Each dependency archive is keyed by a cryptographic digest of:

• Lockfile contents + the install command + cache_dirs list;
• Runner OS and CPU architecture (linux/amd64, linux/arm64, etc.);
• Inside a [container]: the container image's immutable image digest (so native C++/Rust addons never cross Node/Python versions);
• Outside a container: the runner's AMI, bootstrap scripts, and remote_path_setup.

Hydration operates in two tiers:

1. On the same runner (cross-slot): Cached directories are cloned into each slot using filesystem copy-on-write reflink where supported (XFS, Btrfs, macOS APFS), falling back to a local copy. (hardlink is opt-in only because shared inodes allow a mutating test in Slot 1 to corrupt Slot 2's node_modules).
2. Across ephemeral AWS Spot VMs: The first VM to build a cache key compresses and uploads it to a private S3 bucket (spotexec-cache-<account>-<region>, Block Public Access enabled, 14-day lifecycle expiry). Subsequent VMs download and unpack the archive directly instead of running install.

Baked AMIs (spotexec image bake)#

spotexec image bake aws      # launch temporary on-demand builder, run bootstraps, snapshot AMI
spotexec image list aws      # inspect active baked AMIs and bootstrap hashes
spotexec image clear aws     # deregister baked AMIs and delete backing EBS snapshots

spotexec image bake boots a temporary builder instance, runs bootstrap_script, installs Docker + Buildx, pre-pulls your project's public container image (plus any extra --pull IMAGE flags), runs user_bootstrap_script, strips per-host state (SSH host keys, cloud-init state, logs, watchdog timers), and registers a private AMI.

With ami = "auto" (the default), spotexec automatically selects the newest baked AMI matching your runner's current bootstrap hash and skips first-boot package installation—cutting cold Spot VM startup close to bare EC2 boot time. Changing bootstrap_script, user_bootstrap_script, arch, or user invalidates old hashes cleanly; spotexec retains only the 2 newest bakes per runner and garbage-collects older snapshots.