# AWS Spot Fleet Guide

`spotexec` drives EC2 Spot fleets directly from your workstation using hand-signed AWS SigV4 requests—there is no cloud control plane, no CloudFormation stack to maintain, and no IAM role attached to runner VMs.

**Fast path:** Sign in once with an administrative AWS session and run [`spotexec setup`](/docs/setup). It creates a dedicated IAM user (`spotexec-<hostname>`), attaches the least-privilege policy below, writes a non-expiring profile (`spotexec`), ensures the Spot service-linked role and default VPC exist, and verifies your regional Spot vCPU quota.

This guide documents every underlying AWS detail if you prefer to configure IAM, networking, or multi-size fleets manually.

## 1. AWS Sign-In Methods

`spotexec` resolves credentials in the following order:

1. `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN` (only when `profile` is unset on the runner).
2. `aws configure export-credentials --profile <profile>` when AWS CLI v2 is installed—supporting **IAM Identity Center (SSO)**, **`aws login`** (CLI 2.32+), **static IAM keys**, **`assume-role`**, and **`credential_process`**. If an SSO session has expired, `spotexec` reports `run aws sso login` clearly instead of silently falling back to another credential source.
3. Without the AWS CLI installed, `spotexec` reads `~/.aws` directly for static keys, `credential_process`, cached IAM Identity Center tokens (including token refresh), and active `aws login` sessions.

## 2. Least-Privilege IAM Policy

Run `spotexec setup --print-policy > spotexec-policy.json` at any time to export the exact policy. Statements 1–3 are required; statements 4–8 enable the S3 dependency cache, ECR project containers, `spotexec doctor` quota checks, and `spotexec image bake`:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": [
        "ec2:RunInstances", "ec2:CreateTags", "ec2:DescribeInstances", "ec2:TerminateInstances",
        "ec2:DescribeImages", "ec2:DescribeVpcs", "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups", "ec2:CreateSecurityGroup", "ec2:AuthorizeSecurityGroupIngress"
      ], "Resource": "*" },
    { "Effect": "Allow", "Action": "ssm:GetParameter",
      "Resource": "arn:aws:ssm:*::parameter/aws/service/canonical/*" },
    { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*",
      "Condition": { "StringEquals": { "iam:AWSServiceName": "spot.amazonaws.com" } } },
    { "Effect": "Allow", "Action": [
        "s3:CreateBucket", "s3:PutLifecycleConfiguration", "s3:ListBucket",
        "s3:GetObject", "s3:PutObject"
      ], "Resource": ["arn:aws:s3:::spotexec-cache-*", "arn:aws:s3:::spotexec-cache-*/*"] },
    { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" },
    { "Effect": "Allow", "Action": [
        "ecr:DescribeRepositories", "ecr:CreateRepository", "ecr:TagResource", "ecr:PutLifecyclePolicy",
        "ecr:BatchCheckLayerAvailability", "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer",
        "ecr:InitiateLayerUpload", "ecr:UploadLayerPart", "ecr:CompleteLayerUpload", "ecr:PutImage"
      ], "Resource": "arn:aws:ecr:*:*:repository/spotexec/*" },
    { "Effect": "Allow", "Action": "servicequotas:GetServiceQuota", "Resource": "*" },
    { "Effect": "Allow", "Action": [
        "ec2:StopInstances", "ec2:CreateImage", "ec2:DeregisterImage", "ec2:DeleteSnapshot"
      ], "Resource": "*" }
  ]
}
```

If your AWS account has never launched a Spot instance before and your identity cannot create service-linked roles, ask an administrator to run once:

```sh
aws iam create-service-linked-role --aws-service-name spot.amazonaws.com
```

## 3. Region Selection & Spot vCPU Quotas

Always set `region` explicitly on each `[runners.<name>]` block (`~/.aws/config` profile regions are intentionally ignored so runner behavior is deterministic across machines). Pick the region physically closest to you to keep SSH ControlMaster round-trips sub-second, and list 2–3 `instance_types` of the same CPU architecture (`c7a.xlarge`, `c6a.xlarge`, `m7a.xlarge`) so capacity shortages in one pool fall over immediately.

### Checking Spot vCPU Quotas (`L-34B43A08`)

New AWS accounts often start with a very low *All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests* quota (sometimes 5 vCPUs). Ensure your quota covers `max_instances × vCPUs` across your fleets:

```sh
# Inspect current Spot vCPU quota
aws service-quotas get-service-quota --profile spotexec --region us-east-2 \
  --service-code ec2 --quota-code L-34B43A08 --query Quota.Value

# Request an increase (requires an admin session)
aws service-quotas request-service-quota-increase --region us-east-2 \
  --service-code ec2 --quota-code L-34B43A08 --desired-value 64
```

## 4. Networking & Security Groups

By default, `spotexec` launches instances into the region's **default VPC**, assigns a public IPv4 address, and manages a `spotexec-ssh` security group allowing inbound TCP `22` from `ssh_cidr` (`0.0.0.0/0` by default, using Ed25519 key-only authentication generated at `~/.config/spotexec/keys/id_ed25519`).

- **Locking down SSH ingress:** Set `ssh_cidr = "203.0.113.4/32"` if you have a static office/home IP.
- **Custom VPCs:** Set `subnet_id` to a public subnet (routing `0.0.0.0/0` to an Internet Gateway) and optionally `security_group_ids` allowing inbound TCP 22 from your workstation and outbound HTTPS for package registries, S3, and ECR.

## 5. What Gets Created in Your Account

| AWS Resource | Created When | Lifecycle & Cleanup |
| --- | --- | --- |
| IAM user `spotexec-<hostname>` (`/spotexec/`) | `spotexec setup` | Delete user in IAM to revoke that machine. |
| Spot EC2 instances (`spotexec-runner`, `spotexec-owner` tags) | On demand via `sx` or `spotexec up` | Self-terminate on idle (`idle_timeout`) or `max_lifetime`; `spotexec down` terminates immediately. |
| Security group `spotexec-ssh` | First launch without `security_group_ids` | Reusable across runs; delete manually if uninstalling. |
| S3 bucket `spotexec-cache-<account>-<region>` | First run with `[deps]` | Private bucket; objects auto-expire after 14 days. |
| ECR repository `spotexec/<repo-id>` | First `[container]` Dockerfile build | Lifecycle policy retains only the 10 newest images. |
| Baked AMIs & EBS snapshots | `spotexec image bake` | Keeps 2 newest per runner; `spotexec image clear` deletes all. |

## 6. The 6-Stage Life of a Spot VM

1. **Launch.** `sx` submits a one-time Spot `RunInstances` call capped at `max_price` (default: on-demand price) with IMDSv2 required and `InstanceInitiatedShutdownBehavior=terminate`. Capacity errors immediately step through `instance_types`.
2. **Arm Watchdog First.** The very first lines of `cloud-init` user-data (`spotexec cloud-init <runner>`) install and start the `systemd` idle timer and `max_lifetime` hard timer—*before* SSH keys or bootstrap scripts run. Even a hung or broken bootstrap cannot leak an immortal VM.
3. **Bootstrap.** `bootstrap_script` (as `root`) and `user_bootstrap_script` (as `ubuntu`) run while holding the lifecycle `flock`. If either exits non-zero, the VM marks itself failed, returns `BOOTSTRAP_FAILED` with the tail of `/var/log/spotexec-bootstrap.log`, and pauses fleet launches for 1 hour so a broken script cannot burn money in a retry loop.
4. **Serve.** Concurrent `sx` sessions multiplex over SSH `ControlMaster`, acquire slot locks, and refresh the activity timestamp.
5. **Spot Interruption Handling.** `spotexec-helper` polls IMDSv2 every 5s. On a 2-minute `spot/instance-action` notice, it marks the VM `draining` (finishing in-flight jobs while refusing new ones) and notifies `sx`, which immediately acquires or launches replacement capacity in the background. Any job cut off before completion is **replayed once** automatically (`--no-retry` disables replay for non-idempotent deploy scripts). Advisory `rebalance` notices are logged without draining to avoid unnecessary fleet churn.
6. **Self-Termination.** Once idle for `idle_timeout` (default `10m`), or immediately once a `draining` VM finishes its active slots, or at `max_lifetime` (`12h`), the VM powers itself off—which EC2 immediately transitions to `terminated`, stopping EBS and compute billing.

> [!TIP]
> As an account-level backstop against a rare Linux kernel panic that freezes `systemd` timers, add an **AWS Budgets** alert or a simple **EventBridge** rule terminating `spotexec-runner` instances older than 24 hours.