vpc cni warm ip target

Overview

Guide to prevent subnet IP exhaustion on EKS by tuning WARM_IP_TARGET and MINIMUM_IP_TARGET in VPC CNI.

By default, VPC CNI keeps a spare ENI (WARM_ENI_TARGET=1) per node. A single ENI can hold dozens of IPs, so unused IPs pile up and exhaust the subnet. Setting WARM_IP_TARGET and MINIMUM_IP_TARGET switches to IP-level control, which is more efficient.

VariableDescriptionDefaultNote
WARM_IP_TARGETUnused IPs to keep ready per nodeNot set
MINIMUM_IP_TARGETMinimum total IPs per nodeNot set
WARM_ENI_TARGETUnused ENIs to keep ready per node1⚠️ Ignored when WARM_IP_TARGET or MINIMUM_IP_TARGET is set

How it works

When WARM_IP_TARGET and MINIMUM_IP_TARGET are both set, whichever requires more IPs wins:

Total IPs on node = max( assigned IPs + WARM_IP_TARGET, MINIMUM_IP_TARGET )

Example with WARM_IP_TARGET=1, MINIMUM_IP_TARGET=8:

■ = assigned IP, □ = warm IP

3 pods:             15 pods:
┌─ node ───┐        ┌─ node ───────────┐
│ ■■■□□□□□ │        │ ■■■■■■■■■■■■■■■□ │
└──────────┘        └──────────────────┘
  Minimum IP 8        Assigned IP 15 + Warm IP 1
Assigned IPsWARM rule (assigned + 1)MINIMUM ruleActual IPs held
0188
3488
7888
8989
1516816
3031831

Configuration

I recommend using the Helm chart over the EKS managed add-on, because the add-on configuration is JSON-formatted and tightly coupled to the AWS console.

env:
  WARM_IP_TARGET: "1"
  MINIMUM_IP_TARGET: "8"

EKS managed add-on (Terraform)

cluster_addons = {
  vpc-cni = {
    most_recent          = true
    configuration_values = jsonencode({
      env = {
        WARM_IP_TARGET    = "1"
        MINIMUM_IP_TARGET = "8"
      }
    })
  }
}

Verify

Check the aws-node daemonset env vars to confirm WARM_IP_TARGET and MINIMUM_IP_TARGET are applied:

kubectl get daemonset aws-node \
  -n kube-system \
  -o jsonpath='{.spec.template.spec.containers[0].env}' \
  | jq '.[] | select(.name | test("WARM_IP|MINIMUM_IP"))'

Considerations

Prefix delegation

With ENABLE_PREFIX_DELEGATION=true, WARM_PREFIX_TARGET takes priority. ENABLE_PREFIX_DELEGATION defaults to "false". The WARM_IP_TARGET + MINIMUM_IP_TARGET combination works best with prefix delegation disabled.

Sandbox errors

Setting WARM_IP_TARGET=1 caused sandbox creation failures during initial node startup. Set MINIMUM_IP_TARGET to at least the expected initial pod count per node. Continuously tune both values as workload patterns and pod density change over time.

ScenarioWARM_IP_TARGETMINIMUM_IP_TARGET
Subnet has plenty of IPs18-16
Subnet is tight on IPs13-5
Heavy pod deployment2-316-32

References