Issue#

Pods stay in ImagePullBackOff or ErrImagePull, usually after a rollout, a node scale-up, or when several nodes pull the same images at once:

kubectl get pods NAME READY STATUS RESTARTS AGE web-7d9c6b8f5d-2xkqp 0/1 ImagePullBackOff 0 4m

kubectl describe pod shows:

Failed to pull image "nginx:1.27": failed to pull and unpack image "docker.io/library/nginx:1.27": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:...: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Cause#

Docker Hub limits anonymous pulls per source IP. Cluster nodes behind a single NAT gateway share one public IP, so they hit the limit quickly. imagePullPolicy: Always (the default for :latest tags) makes it worse, because every pod start contacts the registry.

Solution#

  1. Check the remaining anonymous quota from an affected node (requires curl and jq):
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token) curl -s --head -H "Authorization: Bearer $TOKEN" \ https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit

ratelimit-remaining: 0;w=21600 confirms that the node is being rate limited.

  1. Create a Docker Hub access token under Account settings > Personal access tokens. Then store it as a registry secret in the namespace:
export DOCKER_USER=myuser export DOCKER_TOKEN=dckr_pat_xxxxxxxx kubectl create secret docker-registry dockerhub-cred \ --namespace default \ --docker-server=https://index.docker.io/v1/ \ --docker-username="$DOCKER_USER" \ --docker-password="$DOCKER_TOKEN"
  1. Attach the secret to the namespace's service account so that every pod uses it automatically:
kubectl patch serviceaccount default -n default \ -p '{"imagePullSecrets":[{"name":"dockerhub-cred"}]}'

Or set it per workload:

spec: template: spec: imagePullSecrets: - name: dockerhub-cred containers: - name: web image: nginx:1.27 imagePullPolicy: IfNotPresent

  1. Use pinned tags with imagePullPolicy: IfNotPresent so that nodes reuse cached images instead of pulling on every restart.

  2. Restart the affected workload. Existing pods do not pick up a service account change:

kubectl rollout restart deployment/web -n default

Tip: For larger clusters, run a pull-through cache such as Harbor, or a registry:2 with proxy.remoteurl set. You can also mirror critical images into your own registry so that production does not depend on Docker Hub.

Verify#

kubectl get pods -n default kubectl get events -n default --field-selector reason=Pulled

Pods should reach Running, and the events should show Successfully pulled image. Run the check from step 1 again with credentials (curl -u "$DOCKER_USER:$DOCKER_TOKEN" on the token URL). It should now report the higher authenticated limit.