Architecture & Security Model

spotexec is built around a deliberate systems constraint: no SaaS backend, no cloud control plane, and no long-running cloud agent. Everything required to sync worktrees, multiplex sessions, manage Spot fleets, and guarantee self-termination lives inside a single Go binary, an embedded helper, OpenSSH, and a 30-line cloud-init watchdog.

System Topology#

Local Workstation                                  Remote Runner (SSH Box or Spot VM)
─────────────────                                  ──────────────────────────────────
sx ─┬─ 1. Manifest: git ls-files + index SHA-1     spotexec-helper (uploaded on first contact)
    ├─ 2. OpenSSH ControlMaster ─────── SSH ────▶   ├─ flock concurrency tokens + isolated slots
    └─ 3. Framed stdio RPC ─────────────────────▶   ├─ applies missing-blob gzip tar delta
                                                    ├─ hydrates [deps] (reflink / presigned S3)
EC2 / S3 / ECR / SSM APIs ◀── Hand-signed SigV4     ├─ runs command in dedicated process group
  RunInstances / Describe / Terminate               └─ hashes workspace post-run → mutation list
  tags: spotexec-runner, spotexec-owner
                                                   cloud-init systemd timer ──▶ poweroff (terminate)

The 6 Steps of an sx Invocation#

1. Build worktree manifest. Enumerates files Git tracks or would track (honoring .gitignore, .spotexecignore, and sync.exclude) alongside their Git SHA-1 blob IDs. Clean tracked files reuse .git/index blob IDs directly without reading file contents from disk.
2. Acquire an isolated slot. Tries configured runners in pool priority order with non-blocking flock token acquisition. If every slot is occupied, joins the fair local queue and registers demand so aws-spot fleets scale out up to max_instances.
3. Stream missing-blob delta. spotexec-helper compares the manifest against the slot's current files and requests only missing SHA-1 blobs, streamed as a single gzip'd tar archive over the multiplexed SSH channel. An unchanged worktree hits a manifest fingerprint match and transfers zero bytes.
4. Hydrate dependencies. Restores cache_dirs (node_modules, .venv, etc.) from the lockfile-keyed cache via copy-on-write reflink (or S3 presigned download across VMs), running your install command only on a cache miss.
5. Execute with bidirectional heartbeats. Runs the command inside its own process group (or [container] Docker image), streaming stdout/stderr back live while exchanging RPC heartbeats in both directions.
6. 3-way content-ID write-back. Hashes the workspace after exit and returns created, modified, and deleted paths. sx applies mutations atomically only if your local file still matches the synced baseline SHA-1—saving concurrent local edits from ever being clobbered.

Failure Modes & Responsibility Matrix#

Security Model & Trust Boundaries#

• Zero AWS credentials on VMs. Spot VMs launch with no IAM instance profile and require IMDSv2. Access to S3 dependency archives and ECR layers uses short-lived presigned URLs scoped to exact objects in your account.
• Cloned .spotexec.toml files cannot hijack connections. Repository configs are forbidden from setting type, host, hosts, ssh_options, identity_file, work_root, profile, region, subnet_id, security_group_ids, ssh_cidr, user, or cache_bucket. Project fleets must extends a global runner you defined.
• Strict write-back path jailing. Every path returned by a runner is validated locally:
  • Must be clean and relative, with no .git path segment under any case folding;
  • Must not traverse any symlink;
  • Modified/deleted paths must exist in the synced manifest (a runner cannot request or overwrite ~/.aws/credentials);
  • Newly created paths must not be gitignored, excluded, or already present locally.
• Secrets stay off command lines. Forwarded env_allow variables are written to a mode 0600 file on the runner rather than passed in ps-visible process arguments.
• Worktree trust note. Because write-back exists to let remote formatters and codegen modify tracked files, treat running untrusted repositories on a shared ssh runner with the same care as CI configs, and inspect git status when working against untrusted hosts.

Key Design Decisions & Known Limits#

• Why no rsync? macOS ships openrsync while Linux distributions ship GNU rsync 3.x with subtle flag incompatibilities. Streaming a missing-blob tar archive over framed stdio RPC avoids external tool skew and guarantees byte-exact baseline tracking for 3-way write-back.
• Why Git SHA-1 blob IDs instead of BLAKE3/xxHash? Clean tracked files already have precomputed SHA-1 blob IDs inside .git/index, letting sx build manifests for large monorepos without reading unchanged files from disk.
• Known limits:
  • Process cleanup uses process groups (plus working-directory sweeps on Linux) rather than cgroups; a background daemon that setsids and chdirs outside the slot workspace can survive until VM shutdown.
  • Each workstation generates its own owner-id and manages its own Spot VMs independently (copy ~/.config/spotexec/owner-id across machines only if you intentionally want shared EC2 tag visibility).
  • Case-only directory renames on case-insensitive macOS filesystems update files inside the existing directory casing; case-only file renames apply cleanly.