AWS Spot Fleet Guide

spotexec drives EC2 Spot fleets directly from your workstation using hand-signed AWS SigV4 requests—there is no cloud control plane, no CloudFormation stack to maintain, and no IAM role attached to runner VMs.

Fast path: Sign in once with an administrative AWS session and run spotexec setup. It creates a dedicated IAM user (spotexec-<hostname>), attaches the least-privilege policy below, writes a non-expiring profile (spotexec), ensures the Spot service-linked role and default VPC exist, and verifies your regional Spot vCPU quota.

This guide documents every underlying AWS detail if you prefer to configure IAM, networking, or multi-size fleets manually.

1. AWS Sign-In Methods#

spotexec resolves credentials in the following order:

1. AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN (only when profile is unset on the runner).
2. aws configure export-credentials --profile <profile> when AWS CLI v2 is installed—supporting IAM Identity Center (SSO), aws login (CLI 2.32+), static IAM keys, assume-role, and credential_process. If an SSO session has expired, spotexec reports run aws sso login clearly instead of silently falling back to another credential source.
3. Without the AWS CLI installed, spotexec reads ~/.aws directly for static keys, credential_process, cached IAM Identity Center tokens (including token refresh), and active aws login sessions.

2. Least-Privilege IAM Policy#

Run spotexec setup --print-policy > spotexec-policy.json at any time to export the exact policy. Statements 1–3 are required; statements 4–8 enable the S3 dependency cache, ECR project containers, spotexec doctor quota checks, and spotexec image bake:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": [
        "ec2:RunInstances", "ec2:CreateTags", "ec2:DescribeInstances", "ec2:TerminateInstances",
        "ec2:DescribeImages", "ec2:DescribeVpcs", "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups", "ec2:CreateSecurityGroup", "ec2:AuthorizeSecurityGroupIngress"
      ], "Resource": "*" },
    { "Effect": "Allow", "Action": "ssm:GetParameter",
      "Resource": "arn:aws:ssm:*::parameter/aws/service/canonical/*" },
    { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*",
      "Condition": { "StringEquals": { "iam:AWSServiceName": "spot.amazonaws.com" } } },
    { "Effect": "Allow", "Action": [
        "s3:CreateBucket", "s3:PutLifecycleConfiguration", "s3:ListBucket",
        "s3:GetObject", "s3:PutObject"
      ], "Resource": ["arn:aws:s3:::spotexec-cache-*", "arn:aws:s3:::spotexec-cache-*/*"] },
    { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" },
    { "Effect": "Allow", "Action": [
        "ecr:DescribeRepositories", "ecr:CreateRepository", "ecr:TagResource", "ecr:PutLifecyclePolicy",
        "ecr:BatchCheckLayerAvailability", "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer",
        "ecr:InitiateLayerUpload", "ecr:UploadLayerPart", "ecr:CompleteLayerUpload", "ecr:PutImage"
      ], "Resource": "arn:aws:ecr:*:*:repository/spotexec/*" },
    { "Effect": "Allow", "Action": "servicequotas:GetServiceQuota", "Resource": "*" },
    { "Effect": "Allow", "Action": [
        "ec2:StopInstances", "ec2:CreateImage", "ec2:DeregisterImage", "ec2:DeleteSnapshot"
      ], "Resource": "*" }
  ]
}

If your AWS account has never launched a Spot instance before and your identity cannot create service-linked roles, ask an administrator to run once:

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

3. Region Selection & Spot vCPU Quotas#

Always set region explicitly on each [runners.<name>] block (~/.aws/config profile regions are intentionally ignored so runner behavior is deterministic across machines). Pick the region physically closest to you to keep SSH ControlMaster round-trips sub-second, and list 2–3 instance_types of the same CPU architecture (c7a.xlarge, c6a.xlarge, m7a.xlarge) so capacity shortages in one pool fall over immediately.

Checking Spot vCPU Quotas (L-34B43A08)#

New AWS accounts often start with a very low All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests quota (sometimes 5 vCPUs). Ensure your quota covers max_instances × vCPUs across your fleets:

# Inspect current Spot vCPU quota
aws service-quotas get-service-quota --profile spotexec --region us-east-2 \
  --service-code ec2 --quota-code L-34B43A08 --query Quota.Value

# Request an increase (requires an admin session)
aws service-quotas request-service-quota-increase --region us-east-2 \
  --service-code ec2 --quota-code L-34B43A08 --desired-value 64

4. Networking & Security Groups#

By default, spotexec launches instances into the region's default VPC, assigns a public IPv4 address, and manages a spotexec-ssh security group allowing inbound TCP 22 from ssh_cidr (0.0.0.0/0 by default, using Ed25519 key-only authentication generated at ~/.config/spotexec/keys/id_ed25519).

• Locking down SSH ingress: Set ssh_cidr = "203.0.113.4/32" if you have a static office/home IP.
• Custom VPCs: Set subnet_id to a public subnet (routing 0.0.0.0/0 to an Internet Gateway) and optionally security_group_ids allowing inbound TCP 22 from your workstation and outbound HTTPS for package registries, S3, and ECR.

5. What Gets Created in Your Account#

6. The 6-Stage Life of a Spot VM#

1. Launch. sx submits a one-time Spot RunInstances call capped at max_price (default: on-demand price) with IMDSv2 required and InstanceInitiatedShutdownBehavior=terminate. Capacity errors immediately step through instance_types.
2. Arm Watchdog First. The very first lines of cloud-init user-data (spotexec cloud-init <runner>) install and start the systemd idle timer and max_lifetime hard timer—before SSH keys or bootstrap scripts run. Even a hung or broken bootstrap cannot leak an immortal VM.
3. Bootstrap. bootstrap_script (as root) and user_bootstrap_script (as ubuntu) run while holding the lifecycle flock. If either exits non-zero, the VM marks itself failed, returns BOOTSTRAP_FAILED with the tail of /var/log/spotexec-bootstrap.log, and pauses fleet launches for 1 hour so a broken script cannot burn money in a retry loop.
4. Serve. Concurrent sx sessions multiplex over SSH ControlMaster, acquire slot locks, and refresh the activity timestamp.
5. Spot Interruption Handling. spotexec-helper polls IMDSv2 every 5s. On a 2-minute spot/instance-action notice, it marks the VM draining (finishing in-flight jobs while refusing new ones) and notifies sx, which immediately acquires or launches replacement capacity in the background. Any job cut off before completion is replayed once automatically (--no-retry disables replay for non-idempotent deploy scripts). Advisory rebalance notices are logged without draining to avoid unnecessary fleet churn.
6. Self-Termination. Once idle for idle_timeout (default 10m), or immediately once a draining VM finishes its active slots, or at max_lifetime (12h), the VM powers itself off—which EC2 immediately transitions to terminated, stopping EBS and compute billing.