AWS Spot instance limits

If you plan to run a lot of parallel jobs on AWS Spot, your default account quota is almost certainly too low. A brand-new AWS account starts at 5 standard Spot vCPUs per region, which is barely one c7i.xlarge. Raising that quota is one of the first things to do if you want to burst to 10 or 20+ workers.

spotexec setup and spotexec doctor check your regional Spot vCPU quota (L-34B43A08) against your configured max_instances, but this guide covers how AWS Spot quotas work, how to check them across regions, and how to get an increase approved.

How spot limits work#

Spot limits cap the total vCPUs your spot instances use, per AWS account, per region, per instance-family group. They do not cap the number of instances.

• vCPUs, not instances. With a 256 vCPU standard quota you could run 32 m5.2xlarge (8 vCPUs each) or 16 c5.4xlarge (16 vCPUs each), or any mix that adds up to 256.
• Pending requests count too. Running spot instances and open spot requests that are not yet fulfilled both use up the quota. A request for a terminated instance keeps counting until EC2 notices the termination and closes it.
• Per region. A quota raised in us-east-1 does nothing in eu-west-1. Every region you launch in has its own values.
• Per family group. Most general-purpose families (A, C, D, H, I, M, R, T, Z) share one "Standard" quota. GPU and accelerator families each have their own, usually starting at 0.
• Spot and on-demand are separate. Raising your on-demand quota does not raise your spot quota, and the reverse.
• AWS adjusts them over time. EC2 raises spot quotas automatically based on usage. You can also request an increase at any time.

Quota codes#

Every spot quota has a code under the ec2 service, and the CLI uses that code. Default values are what a brand-new account gets in each region. Standard starts at 5 vCPUs and every other group starts at 0.

On-demand also has High Memory (L-43DA4232) and HPC (L-F7808C92) quotas, which have no spot equivalent. Codes and defaults were read from aws service-quotas list-aws-default-service-quotas --service-code ec2 in us-east-1 and match the Spot Instance quotas page. Some third-party guides list wrong codes for the X, F, and DL groups. Treat this table, or that command, as the reference.

Checking your current limits and usage#

The fastest check is one CLI call per region. It returns your account's applied value, which may be well above the default.

Console: open the Service Quotas console for EC2, pick the region in the top bar, and filter by "Spot". Open a quota to see its applied value and, if you are using it, current utilization.

CLI, one quota:

aws service-quotas get-service-quota \
  --service-code ec2 --quota-code L-34B43A08 \
  --region us-east-1 --query 'Quota.Value'

CLI, all spot quotas in a region:

aws service-quotas list-service-quotas \
  --service-code ec2 --region us-east-1 \
  --query "Quotas[?contains(QuotaName, 'Spot Instance Requests')].[QuotaCode,QuotaName,Value]" \
  --output table

CLI, across every region you use:

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  printf '%s\t' "$r"
  aws service-quotas get-service-quota --service-code ec2 \
    --quota-code L-34B43A08 --region "$r" --query 'Quota.Value' --output text
done

Current usage: EC2 publishes vCPU usage to CloudWatch. The metric is ResourceCount in the AWS/Usage namespace, with dimensions Service=EC2, Type=Resource, Resource=vCPU and Class=Standard/Spot (other groups use their own class, such as G/Spot). Use the Maximum statistic. In CloudWatch metric math, SERVICE_QUOTA(m1) returns the quota for that metric, so m1 / SERVICE_QUOTA(m1) * 100 gives percent used. The Service Quotas console can create an alarm on this for you; see Service Quotas and Amazon CloudWatch alarms.

For a quick count without CloudWatch, list running spot instances and add up their vCPUs:

aws ec2 describe-instances --region us-east-1 \
  --filters Name=instance-lifecycle,Values=spot Name=instance-state-name,Values=pending,running \
  --query 'Reservations[].Instances[].[InstanceType, CpuOptions.CoreCount, CpuOptions.ThreadsPerCore]' \
  --output text

vCPUs per instance = cores x threads per core. Open spot requests also count; list them with aws ec2 describe-spot-instance-requests --filters Name=state,Values=open.

New accounts#

A new account can run 5 standard spot vCPUs per region and no GPU or accelerator spot at all. That is one c7i.xlarge (4 vCPUs) or two t3.medium, nothing more.

• Why limits are low. AWS sets low starting quotas so new accounts ramp up gradually. This limits surprise bills and abuse such as stolen-card crypto mining.
• They grow on their own. EC2 "automatically adjusts your Spot Instance quotas based on your usage" (Spot Instance quotas). AWS does not publish the thresholds or timing. In practice, steady use and a few paid invoices raise the limits without a request.
• Some accounts start below the default. Questions on AWS re:Post report new accounts with an applied vCPU quota of 1, below the published default of 5. Always check the applied value with get-service-quota, not the documented default.
• 0-vCPU groups need a request. A quota of 0 never grows from usage, because you cannot use it. GPU spot (G, P) and the other specialized groups need an explicit request before the first launch.

Requesting an increase#

Request the new total vCPU count, once per region, through Service Quotas. The console and CLI do the same thing; the Support Center is the fallback.

Work out the number first. Add the vCPUs you run now, the vCPUs you plan to add, and a 10% to 30% buffer, as AWS recommends in How do I request a vCPU service quota increase. For example, 20 c7i.2xlarge workers at 8 vCPUs each is 160 vCPUs; with a 20% buffer, ask for 192.

Console#

1. Open the Service Quotas console for EC2 and select the region in the top bar.
2. Filter by "Spot" and select the quota, such as All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests.
3. Choose Request increase at account level.
4. Enter the new total, not the amount to add, and choose Request.
5. Track it under Dashboard or the quota's Request history tab. Once it reaches Quota requested, it shows the linked support case number.

CLI#

aws service-quotas request-service-quota-increase \
  --service-code ec2 --quota-code L-34B43A08 \
  --desired-value 192 --region us-east-1

The response includes an Id. Check on it with:

aws service-quotas get-requested-service-quota-change --request-id <Id> --region us-east-1
# or everything recent for EC2:
aws service-quotas list-requested-service-quota-change-history \
  --service-code ec2 --region us-east-1

Permissions#

The caller needs servicequotas:RequestServiceQuotaIncrease plus the read actions to check status. The first request also needs iam:CreateServiceLinkedRole, because Service Quotas opens support cases through a service-linked role. The AWS managed policy ServiceQuotasFullAccess covers all of it (Requesting a quota increase).

Support Center fallback#

If Service Quotas rejects the request or you need to add context, open a case in the AWS Support Center. Choose Service limit increase, then EC2 Spot Instances, the region, and the new vCPU limit. Every support plan, including Basic, can file these cases.

Automating it#

• Automatic Management. Service Quotas can watch usage and alert you at 80% and 95% of a quota. In Notify and Auto-Adjust mode it also files the increase request for you, without a support case. Approval is not guaranteed; if it fails, file a manual request (Automatic Management).
• Quota request templates (AWS Organizations). The management account can list up to 10 quota values that every new member account requests automatically. It needs Organizations with all features enabled, and it does not change existing accounts (Using request templates).

Approval: what to expect#

Small increases are usually approved automatically; larger ones go to AWS Support, which can approve, deny, or approve part of the request (Requesting a quota increase). AWS does not publish the size cut-off or a turnaround time, and quota requests do not get priority support.

Request new total
  │
  ├─ Small increase ──▶ Automatic approval ──▶ New quota applied
  │
  └─ Larger increase ─▶ Support case opened ─▶ AWS Support reviews
                                                  │
                                                  ├─ Approved ──▶ New quota applied
                                                  └─ Partial or denied ──▶ Add detail, ask again

A partial approval or denial loops back: add detail on the case, or run at the granted level and ask again later.

Typical timing, from community reports, not AWS commitments:

• Automatic approvals: minutes to a few hours.
• Support review: usually 1 to 3 business days, sometimes up to a week on Basic support.
• New accounts asking for large jumps, or any GPU group from 0, are the most likely to get a partial approval or a denial.

How to get approved:

• Ask in steps. Use most of your current quota, then ask for the next level. A jump from 5 to 1,000 vCPUs on a week-old account is likely to be cut down.
• Name the region and instance types. For example: "us-east-1, up to 24 c7i.2xlarge spot instances for batch jobs." AWS's own guidance asks for this.
• Say how they run. State whether instances run in parallel or one at a time, how long jobs last, and whether it is development or production.
• Request the peak you will test. One quota covers a whole family group, so ask for the largest concurrent run you plan. You can then swap instance types freely within it.
• Answer the support case quickly. If a case opens, AWS may ask for a use-case description. Requests often wait on that reply.
• On a denial or partial approval, reply on the case with more detail, or run at the granted level for a few weeks and ask again.

Hitting the limit#

MaxSpotInstanceCountExceeded is the only one of these errors that a quota increase fixes for spot. The others mean you are on the wrong quota or AWS is short of hardware.

Descriptions follow the EC2 API error codes reference.

If you hit the quota unexpectedly:

1. Check the region. The CLI's default region may not be the one you raised.
2. Check the family group. A g5 launch uses the G and VT quota, not Standard.
3. Look for open spot requests with aws ec2 describe-spot-instance-requests --filters Name=state,Values=open. Cancel stale ones with aws ec2 cancel-spot-instance-requests; they count against the quota until closed.
4. Add up running spot vCPUs (see Checking your current limits and usage). Instances you have terminated can count briefly until EC2 closes their requests.

Sources#

• Spot Instance quotas (Amazon EC2 User Guide)
• Amazon EC2 service quotas (Amazon EC2 User Guide)
• Requesting a quota increase (Service Quotas User Guide)
• RequestedServiceQuotaChange (Service Quotas API Reference, request statuses)
• Service Quotas and Amazon CloudWatch alarms
• Service Quotas Automatic Management
• Using Service Quotas request templates
• EC2 API error codes
• How do I request a vCPU service quota increase for my EC2 On-Demand Instance? (AWS re:Post Knowledge Center)
• How do I increase the service quota of my Amazon EC2 resources? (AWS re:Post Knowledge Center)
• Quota codes, default values, and CloudWatch metric dimensions were read from the Service Quotas API (list-aws-default-service-quotas, get-service-quota) in us-east-1.
• Approval timings and partial-approval patterns come from community reports on AWS re:Post, not from AWS documentation.