# Setting Up Machines

Every developer machine where you run `sx` needs three things:

1. `spotexec` installed on `PATH` (along with `git` and `ssh`);
2. A global runner configuration at `~/.config/spotexec/config.toml` defining where commands execute;
3. For `aws-spot` fleets, scoped AWS credentials.

`spotexec setup` automates all three on your current workstation and on any other machines you pass over SSH. It is idempotent and safe to re-run at any time.

## Setting Up This Machine

```sh
brew install daintreehq/tap/spotexec
brew install awscli            # recommended for initial AWS setup
aws login                      # or aws sso login / aws configure (with IAM user management access)
spotexec setup
```

When you run `spotexec setup`:

1. **Runner config.** If `~/.config/spotexec/config.toml` does not exist, setup writes a starter configuration with an `aws` Spot fleet in your AWS CLI's default region. Existing configs are preserved.
2. **Scoped AWS credentials.** For each `aws-spot` runner, setup previews every change and asks for confirmation before touching your account:
   - Creates a dedicated IAM user for this machine named `spotexec-<hostname>` (path `/spotexec/`, tagged `spotexec-managed`) with an inline least-privilege policy (`spotexec setup --print-policy`).
   - Creates an access key and writes it to `~/.aws/credentials` (mode `0600`) under profile `spotexec`. After verifying EC2 connectivity and confirming no environment overrides conflict, it cleans up any older keys on that user.
   - Sets `profile = "spotexec"` on your AWS runners in `config.toml`.
   - Ensures the `AWSServiceRoleForEC2Spot` service-linked role exists and creates a free default VPC if the region lacks one (`--no-vpc` skips this).
   - Verifies your regional Spot vCPU quota against your configured fleet ceiling and prints the exact `aws service-quotas` command if an increase is needed.
3. **Other machines.** In an interactive terminal, setup discovers other Macs with Remote Login via Bonjour (or Avahi on Linux) plus `Host` entries in `~/.ssh/config`, and offers to provision them over SSH.

```console
$ spotexec setup
✓ runner config /Users/me/.config/spotexec/config.toml
Setting up AWS access for aws (region us-east-1)…

Signed in to AWS account 123456789012 as arn:aws:iam::123456789012:user/me. This will:
  • create (or update) IAM user spotexec-studio-02 with only spotexec's permissions (see `spotexec setup --print-policy`)
  • save an access key for it as AWS profile "spotexec" in ~/.aws/credentials (if it hasn't one already)
  • set profile = "spotexec" on runner(s) aws
Continue? [Y/n]
  created IAM user spotexec-studio-02
  attached the spotexec policy
  created access key AKIA…
  saved it as AWS profile spotexec (/Users/me/.aws/credentials)
  verified: the key signs in and can reach EC2 in us-east-1
✓ AWS profile spotexec → arn:aws:iam::123456789012:user/spotexec/spotexec-studio-02 (key AKIA…; it doesn't expire)
✓ set profile = "spotexec" on aws in /Users/me/.config/spotexec/config.toml
✓ Spot vCPU quota in us-east-1: 640 (runners need up to 32)
```

Once `spotexec setup` finishes, your interactive `aws login` or SSO session can expire without interrupting `sx`—you only need an admin session again when rotating keys or changing IAM setup.

## Choosing Runners

A **runner** is either a static **SSH host** (`type = "ssh"`) or an elastic **Spot fleet** (`type = "aws-spot"`). A **pool** is the ordered list of runners a command may use, and each machine provides `max_slots` isolated execution **slots** (one concurrent job per slot).

### Option A: An SSH Box You Already Own

Non-interactive SSH key login must work (`ssh -o BatchMode=yes me@192.168.1.50 true`), and the host key must be present in `~/.ssh/known_hosts`. Commands execute under `bash` after sourcing `/etc/profile` and `~/.bash_profile` (falling back to `~/.profile`), plus `remote_path_setup` if configured.

```toml
version = 1
default_pool = ["homelab"]

[runners.homelab]
type = "ssh"
hosts = ["me@192.168.1.50", "me@homelab.tailnet.ts.net"]   # LAN first, Tailscale fallback
max_slots = 4
```

> [!TIP]
> Passing multiple addresses in `hosts` dials all routes in parallel and prefers an earlier entry if it connects within 500ms of the fastest response—ideal for switching seamlessly between home LAN Ethernet and Tailscale on the road.

### Option B: Scale-to-Zero AWS Spot Fleet

Spot VMs boot clean Ubuntu 24.04 images on demand and power themselves off after `idle_timeout` without activity. Install your global toolchain via `bootstrap_script` / `user_bootstrap_script`, or define reproducible per-repo environments with [project containers](/docs/environments#project-containers).

```toml
version = 1
default_pool = ["aws"]

[runners.aws]
type = "aws-spot"
profile = "spotexec"
region = "us-east-2"                             # always set explicitly
instance_types = ["c7a.xlarge", "c6a.xlarge"]    # tried in order on capacity errors
max_slots = 2                                    # concurrent jobs per VM
max_instances = 4                                # fleet ceiling (8 total slots)
idle_timeout = "10m"                             # self-terminates after 10m idle
```

### Option C: Hybrid Pool (Homelab First, Spot Overflow)

List both runners in `default_pool` and assign lower `priority` numbers to hardware you already pay for. `sx` fills all 4 `homelab` slots first and boots AWS Spot instances only when concurrent agent demand overflows:

```toml
version = 1
default_pool = ["homelab", "aws"]

[runners.homelab]
type = "ssh"
hosts = ["me@192.168.1.50", "me@homelab.tailnet.ts.net"]
max_slots = 4
priority = 10                  # lower runs first ($0 marginal cost)

[runners.aws]
type = "aws-spot"
profile = "spotexec"
region = "us-east-2"
instance_types = ["c7a.2xlarge", "c6a.2xlarge", "m7a.2xlarge"]
max_slots = 4
max_instances = 4
priority = 20                  # takes overflow only when homelab is full
idle_timeout = "10m"
hourly_cost = 0.14
```

## Provisioning Additional Machines Over SSH

Pass hostnames or SSH aliases to `spotexec setup` to configure laptops, Mac Studios, or remote dev boxes in one shot:

```sh
spotexec setup studio-01.local studio-02.local me@devbox
```

Over a single SSH connection per host, `spotexec setup`:

1. Installs or upgrades `spotexec` (via Homebrew if installed, otherwise downloading the latest release into `~/.local/bin`; override with `--install brew|release|copy|none`).
2. Copies `~/.config/spotexec/config.toml` (backing up any differing file to `config.toml.bak-<timestamp>`).
3. Uses your local admin AWS session—streamed over SSH `stdin`, never exposed in process args or written to remote disk—to create **that machine's own** IAM user (`spotexec-<remote-hostname>`), access key, SSH keypair, and `owner-id`.

Each machine keeps isolated credentials and its own `owner-id` tag, so one machine never terminates another machine's active Spot VMs, and a lost laptop can be revoked individually.

## Key Rotation, Revocation, and Keyless Mode

- **Rotate this machine's key:** `spotexec setup --rotate` creates a new access key, verifies EC2 access, updates `~/.aws/credentials`, and deletes the previous key.
- **Audit all provisioned machines:** `aws iam list-users --path-prefix /spotexec/` lists every machine user.
- **Keep zero long-lived keys on disk:** Run `spotexec setup --skip-aws` and point `profile` at an IAM Identity Center (`aws sso login`) profile carrying the `spotexec` policy.

To revoke a retired or lost machine immediately:

```sh
U=spotexec-studio-01
for k in $(aws iam list-access-keys --user-name "$U" --query 'AccessKeyMetadata[].AccessKeyId' --output text); do
  aws iam delete-access-key --user-name "$U" --access-key-id "$k"
done
aws iam delete-user-policy --user-name "$U" --policy-name spotexec
aws iam delete-user --user-name "$U"
```

## `spotexec setup` Flags

| Flag | Effect |
| --- | --- |
| `[host...]` | Also install and configure these machines over SSH. |
| `--yes`, `-y` | Skip confirmation prompts and skip interactive LAN host discovery. |
| `--install auto\|brew\|release\|copy\|none` | Installation method for remote machines (default `auto`). |
| `--admin-profile NAME` | AWS profile used to manage IAM users (default: `AWS_PROFILE`, then `default`). |
| `--profile NAME` | Local AWS profile name to write credentials into (default `spotexec`). |
| `--user NAME` | IAM user name for this machine (default `spotexec-<hostname>`). |
| `--rotate` | Issue a fresh IAM access key even if the existing one works. |
| `--no-vpc` | Do not create a default VPC when the configured region lacks one. |
| `--skip-aws` | Write/copy `config.toml` only without provisioning IAM credentials. |
| `--print-policy` | Print the least-privilege IAM policy JSON to stdout and exit. |