We tried to roll out a multi-tenant “shared ALB” architecture for our production EKS cluster and the apply blew up with three apparently unrelated errors. Untangling them turned into a tour of EKS Auto Mode internals, the AWS Load Balancer Controller’s Ingress-group feature, and the seams that appear when one Kubernetes cluster is provisioned by two or more separate Terraform Cloud workspaces. This post writes down the findings so the next person doesn’t burn a day on it.<\/p>\n\n\n\n
The cluster is EKS 1.34 with Auto Mode enabled, two A single It would be easy to read these as three bugs in our code. They’re not. They’re three independent failure modes that surfaced together because we built one design on top of another stack we didn’t fully understand.<\/p>\n\n\n\n EKS Auto Mode bundles its own ALB controller as a managed capability \u2014 exposed via Separately, you can install the standalone AWS Load Balancer Controller (LBC) via the Helm chart These are not the same controller. They are not the same product. They are not interchangeable. They differ in their feature set, their CRD group, and the API contract for annotations.<\/p>\n\n\n\n Our cluster had both installed. We discovered this only when we noticed the live Why both? Git archaeology answered it. In March we shipped a staging-only design that used Auto Mode’s built-in controller and per-gateway ALBs (one ALB per gateway, no sharing). In May we shipped a different<\/em> production design that uses a shared ALB across teams via the The standalone LBC pods entered CrashLoopBackOff with:<\/p>\n\n\n\n Pods can’t reach IMDSv2 because the token PUT requires an extra network hop and the node enforces “EKS Auto Mode enforces IMDSv2 with a hop limit of 1 by default, adhering to AWS security best practices. This default configuration cannot be modified in Auto Mode.<\/strong>“ The same page also documents the official workaround:<\/p>\n\n\n\n “For add-ons that typically require IMDS access, supply parameters (such as AWS region) during installation to avoid IMDS lookups.”<\/p>\n<\/blockquote>\n\n\n\n And the AWS LBC Helm install guide is specific about which values to set when the nodes have IMDS restricted:<\/p>\n\n\n\n “If you’re deploying the controller to Amazon EC2 nodes that have restricted access to the Amazon EC2 instance metadata service (IMDS)<\/strong>, or if you’re deploying to Fargate or Amazon EKS Hybrid Nodes, then add the following flags to the \u2014 https:\/\/docs.aws.amazon.com\/eks\/latest\/userguide\/lbc-helm.html#lbc-helm-install<\/a><\/p>\n<\/blockquote>\n\n\n\n So the fix for LBC on Auto Mode is to inject both<\/strong> values it would have learned from IMDS as explicit Helm values:<\/p>\n\n\n\n The LBC chart renders these into Note: we initially set only This is not a hack. This is the AWS-documented pattern for any IMDS-using add-on running on Auto Mode (and for Fargate, and for EKS Hybrid Nodes \u2014 same restriction shape). Internalise it \u2014 every IRSA workload you add to an Auto Mode cluster needs to take its region\/VPC\/account as parameters, not from IMDS. The day you forget, you’ll spend an hour debugging “why does this pod hang on startup”.<\/p>\n\n\n\n The standalone LBC ships a When LBC pods crash, the webhook service has zero endpoints. Every Service create \u2014 including the Service that the ExternalDNS Helm chart wants to make in the team’s namespace \u2014 is rejected because the cluster can’t call the webhook. So a controller crash that has nothing to do with ExternalDNS still breaks our ExternalDNS rollout. The Helm releases retry, partially apply, and leave the cluster in a half-state.<\/p>\n\n\n\n If you ever see “webhook failed” or “no endpoints available for service” for Our An “anchor” Ingress holds listener-level config that can’t be expressed per-rule \u2014 listen ports, SSL policy, the cert ARN list for SNI, a catch-all 404 action for hosts no team rule matches. We use This pattern is documented for the standalone LBC:<\/p>\n\n\n\n “You can add an order number of your ingress resource. Auto Mode supports a similar concept, but the API is different:<\/p>\n\n\n\ngeneral-purpose<\/code> Auto Mode nodes, and four “team” namespaces (analytics<\/code>, design<\/code>, platform<\/code>, qa<\/code>) each getting their own ExternalDNS instance pointed at a delegated Route53 zone under tools.drivemode.com<\/code>. The intent is one shared internet-facing ALB serving <name>.<team>.tools.drivemode.com<\/code> for any team, with cert SNI across four wildcard ACM certs and team isolation enforced by a ValidatingAdmissionPolicy<\/code>.<\/p>\n\n\n\nThe three errors, and why they were misleading<\/h2>\n\n\n\n
terraform apply<\/code> produced:<\/p>\n\n\n\nError: 3 errors occurred:\n * clusterroles.rbac.authorization.k8s.io \"external-dns\" already exists\n * clusterrolebindings.rbac.authorization.k8s.io \"external-dns-viewer\" already exists\n * Internal error occurred: failed calling webhook \"mservice.elbv2.k8s.aws\":\n no endpoints available for service \"aws-load-balancer-webhook-service\"\n\nError: Cannot create resource that already exists\n resource \"\/alb\" already exists\n module.tools_dns.module.alb_ingressclass.kubernetes_manifest.ingressclass\n<\/code><\/pre>\n\n\n\nTwo different ALB controllers, one cluster<\/h2>\n\n\n\n
kubernetesNetworkConfig.elasticLoadBalancing.enabled<\/code>. There is no pod for it; the control plane runs it for you. Its IngressClass controller string is eks.amazonaws.com\/alb<\/code> and its IngressClassParams live in the eks.amazonaws.com\/v1<\/code> CRD group.<\/p>\n\n\n\naws-load-balancer-controller<\/code>. That ships its own pod, its own webhooks, its own CRDs in elbv2.k8s.aws<\/code>, and its IngressClass controller string is ingress.k8s.aws\/alb<\/code>.<\/p>\n\n\n\nIngressClass alb<\/code> had two distinct IngressClassParams CRDs in two different API groups, and the Helm-annotated IngressClass alb<\/code> was owned by the standalone LBC release \u2014 not by the EKS-managed controller our tools_dns<\/code> module was hard-coded against.<\/p>\n\n\n\nalb.ingress.kubernetes.io\/group.name<\/code> annotation \u2014 and that annotation is a feature of the standalone LBC, not Auto Mode. So the same PR that introduced the shared-ALB design also flipped on enable_aws_load_balancer_controller = true<\/code>. Both controllers ended up coexisting on the cluster.<\/p>\n\n\n\nThe IMDS hop-limit ambush<\/h2>\n\n\n\n
unable to initialize AWS cloud: failed to introspect vpcID from EC2Metadata\nor Node name, specify --aws-vpc-id instead if EC2Metadata is unavailable:\nEC2MetadataError: failed to make EC2Metadata request status code: 401\n<\/code><\/pre>\n\n\n\nHttpPutResponseHopLimit=1<\/code>. AWS documents this constraint explicitly for Auto Mode:<\/p>\n\n\n\n\n
\u2014 https:\/\/docs.aws.amazon.com\/eks\/latest\/userguide\/automode-learn-instances.html<\/a><\/p>\n<\/blockquote>\n\n\n\n\n
\n
helm<\/code> command that follows:<\/p>\n\n\n\n\n
--set region={{region-code}}<\/code><\/li>\n\n\n\n--set vpcId={{vpc-xxxxxxxx}}<\/code>“<\/li>\n<\/ul>\n\n\n\naws_load_balancer_controller = {\n set = [\n { name = \"vpcId\", value = module.network.vpc_id },\n { name = \"region\", value = data.aws_region.current.name },\n ]\n}\n<\/code><\/pre>\n\n\n\n--aws-vpc-id=<value><\/code> and --aws-region=<value><\/code> on the Deployment, the IMDS lookup is skipped, and pods come Ready.<\/p>\n\n\n\nvpcId<\/code>, reasoning that IRSA already injects AWS_REGION<\/code> env vars into the pod so the AWS SDK would pick up the region without IMDS. That reasoning may be true for SDK calls, but the AWS doc explicitly tells you to set both for this exact scenario. We followed the doc. Don’t second-guess AWS guidance with inferred reasoning \u2014 they wrote that line for a reason, and the cost of setting a redundant value is zero.<\/p>\n\n\n\nThe webhook cascade \u2014 why one crashing pod broke unrelated Helm releases<\/h2>\n\n\n\n
MutatingWebhookConfiguration<\/code> with three webhooks: mpod.elbv2.k8s.aws<\/code>, mservice.elbv2.k8s.aws<\/code>, mtargetgroupbinding.elbv2.k8s.aws<\/code>. All three are failurePolicy: Fail<\/code>. The mservice<\/code> webhook fires on every Service create, cluster-wide.<\/p>\n\n\n\naws-load-balancer-webhook-service<\/code>, don’t look at the resource you were trying to create. Look at the LBC pods.<\/p>\n\n\n\nThe
group.name<\/code> annotation is the whole reason we run standalone LBC<\/h2>\n\n\n\ntools_dns<\/code> design wants one ALB serving every team, with each team owning its own Ingresses. The mechanism that lets multiple Ingresses converge onto a single ALB is the standalone LBC’s IngressGroup<\/code> feature. You stamp every Ingress that should land on the same ALB with the same alb.ingress.kubernetes.io\/group.name<\/code> annotation and the controller merges them.<\/p>\n\n\n\ngroup.order = 1000<\/code> on the anchor so its catch-all 404 sits last in the merged rule list, after all team rules.<\/p>\n\n\n\n\n
alb.ingress.kubernetes.io\/group.order: '10'<\/code> … Duplicate rules with a higher number can overwrite rules with a lower number. By default, the rule order between ingresses within the same ingress group is determined lexicographically based namespace and name.”
\u2014 https:\/\/docs.aws.amazon.com\/eks\/latest\/userguide\/alb-ingress.html<\/a><\/p>\n<\/blockquote>\n\n\n\nEKS Auto Mode supports “shared ALB” \u2014 but not the same way<\/h2>\n\n\n\n
\n