Setting Up Machines

Every developer machine where you run sx needs three things:

1. spotexec installed on PATH (along with git and ssh);
2. A global runner configuration at ~/.config/spotexec/config.toml defining where commands execute;
3. For aws-spot fleets, scoped AWS credentials.

spotexec setup automates all three on your current workstation and on any other machines you pass over SSH. It is idempotent and safe to re-run at any time.

Setting Up This Machine#

brew install daintreehq/tap/spotexec
brew install awscli            # recommended for initial AWS setup
aws login                      # or aws sso login / aws configure (with IAM user management access)
spotexec setup

When you run spotexec setup:

1. Runner config. If ~/.config/spotexec/config.toml does not exist, setup writes a starter configuration with an aws Spot fleet in your AWS CLI's default region. Existing configs are preserved.
2. Scoped AWS credentials. For each aws-spot runner, setup previews every change and asks for confirmation before touching your account:
   • Creates a dedicated IAM user for this machine named spotexec-<hostname> (path /spotexec/, tagged spotexec-managed) with an inline least-privilege policy (spotexec setup --print-policy).
   • Creates an access key and writes it to ~/.aws/credentials (mode 0600) under profile spotexec. After verifying EC2 connectivity and confirming no environment overrides conflict, it cleans up any older keys on that user.
   • Sets profile = "spotexec" on your AWS runners in config.toml.
   • Ensures the AWSServiceRoleForEC2Spot service-linked role exists and creates a free default VPC if the region lacks one (--no-vpc skips this).
   • Verifies your regional Spot vCPU quota against your configured fleet ceiling and prints the exact aws service-quotas command if an increase is needed.
3. Other machines. In an interactive terminal, setup discovers other Macs with Remote Login via Bonjour (or Avahi on Linux) plus Host entries in ~/.ssh/config, and offers to provision them over SSH.

$ spotexec setup
✓ runner config /Users/me/.config/spotexec/config.toml
Setting up AWS access for aws (region us-east-1)…

Signed in to AWS account 123456789012 as arn:aws:iam::123456789012:user/me. This will:
  • create (or update) IAM user spotexec-studio-02 with only spotexec's permissions (see `spotexec setup --print-policy`)
  • save an access key for it as AWS profile "spotexec" in ~/.aws/credentials (if it hasn't one already)
  • set profile = "spotexec" on runner(s) aws
Continue? [Y/n]
  created IAM user spotexec-studio-02
  attached the spotexec policy
  created access key AKIA…
  saved it as AWS profile spotexec (/Users/me/.aws/credentials)
  verified: the key signs in and can reach EC2 in us-east-1
✓ AWS profile spotexec → arn:aws:iam::123456789012:user/spotexec/spotexec-studio-02 (key AKIA…; it doesn't expire)
✓ set profile = "spotexec" on aws in /Users/me/.config/spotexec/config.toml
✓ Spot vCPU quota in us-east-1: 640 (runners need up to 32)

Once spotexec setup finishes, your interactive aws login or SSO session can expire without interrupting sx—you only need an admin session again when rotating keys or changing IAM setup.

Choosing Runners#

A runner is either a static SSH host (type = "ssh") or an elastic Spot fleet (type = "aws-spot"). A pool is the ordered list of runners a command may use, and each machine provides max_slots isolated execution slots (one concurrent job per slot).

Option A: An SSH Box You Already Own#

Non-interactive SSH key login must work (ssh -o BatchMode=yes me@192.168.1.50 true), and the host key must be present in ~/.ssh/known_hosts. Commands execute under bash after sourcing /etc/profile and ~/.bash_profile (falling back to ~/.profile), plus remote_path_setup if configured.

version = 1
default_pool = ["homelab"]

[runners.homelab]
type = "ssh"
hosts = ["me@192.168.1.50", "me@homelab.tailnet.ts.net"]   # LAN first, Tailscale fallback
max_slots = 4

Option B: Scale-to-Zero AWS Spot Fleet#

Spot VMs boot clean Ubuntu 24.04 images on demand and power themselves off after idle_timeout without activity. Install your global toolchain via bootstrap_script / user_bootstrap_script, or define reproducible per-repo environments with project containers.

version = 1
default_pool = ["aws"]

[runners.aws]
type = "aws-spot"
profile = "spotexec"
region = "us-east-2"                             # always set explicitly
instance_types = ["c7a.xlarge", "c6a.xlarge"]    # tried in order on capacity errors
max_slots = 2                                    # concurrent jobs per VM
max_instances = 4                                # fleet ceiling (8 total slots)
idle_timeout = "10m"                             # self-terminates after 10m idle

Option C: Hybrid Pool (Homelab First, Spot Overflow)#

List both runners in default_pool and assign lower priority numbers to hardware you already pay for. sx fills all 4 homelab slots first and boots AWS Spot instances only when concurrent agent demand overflows:

version = 1
default_pool = ["homelab", "aws"]

[runners.homelab]
type = "ssh"
hosts = ["me@192.168.1.50", "me@homelab.tailnet.ts.net"]
max_slots = 4
priority = 10                  # lower runs first ($0 marginal cost)

[runners.aws]
type = "aws-spot"
profile = "spotexec"
region = "us-east-2"
instance_types = ["c7a.2xlarge", "c6a.2xlarge", "m7a.2xlarge"]
max_slots = 4
max_instances = 4
priority = 20                  # takes overflow only when homelab is full
idle_timeout = "10m"
hourly_cost = 0.14

Provisioning Additional Machines Over SSH#

Pass hostnames or SSH aliases to spotexec setup to configure laptops, Mac Studios, or remote dev boxes in one shot:

spotexec setup studio-01.local studio-02.local me@devbox

Over a single SSH connection per host, spotexec setup:

1. Installs or upgrades spotexec (via Homebrew if installed, otherwise downloading the latest release into ~/.local/bin; override with --install brew|release|copy|none).
2. Copies ~/.config/spotexec/config.toml (backing up any differing file to config.toml.bak-<timestamp>).
3. Uses your local admin AWS session—streamed over SSH stdin, never exposed in process args or written to remote disk—to create that machine's own IAM user (spotexec-<remote-hostname>), access key, SSH keypair, and owner-id.

Each machine keeps isolated credentials and its own owner-id tag, so one machine never terminates another machine's active Spot VMs, and a lost laptop can be revoked individually.

Key Rotation, Revocation, and Keyless Mode#

• Rotate this machine's key: spotexec setup --rotate creates a new access key, verifies EC2 access, updates ~/.aws/credentials, and deletes the previous key.
• Audit all provisioned machines: aws iam list-users --path-prefix /spotexec/ lists every machine user.
• Keep zero long-lived keys on disk: Run spotexec setup --skip-aws and point profile at an IAM Identity Center (aws sso login) profile carrying the spotexec policy.

To revoke a retired or lost machine immediately:

U=spotexec-studio-01
for k in $(aws iam list-access-keys --user-name "$U" --query 'AccessKeyMetadata[].AccessKeyId' --output text); do
  aws iam delete-access-key --user-name "$U" --access-key-id "$k"
done
aws iam delete-user-policy --user-name "$U" --policy-name spotexec
aws iam delete-user --user-name "$U"

spotexec setup Flags#