Issue#

Pods suddenly stop and get replaced, and kubectl get pods shows many of them in the Evicted (or Error/ContainerStatusUnknown) state:

kubectl get pods -A | grep -i evicted

kubectl describe pod shows the reason:

Status: Failed Reason: Evicted Message: The node was low on resource: ephemeral-storage. Threshold quantity: 4902142351, available: 3984236Ki. Container app was using 6582924Ki, request is 0, has larger consumption of ephemeral-storage.

The node also reports disk pressure:

DiskPressure True KubeletHasDiskPressure kubelet has disk pressure

Cause#

The kubelet watches free space on the node filesystem (/var/lib/kubelet, container logs) and the image filesystem (/var/lib/containerd). When free space drops below the eviction thresholds (by default nodefs.available<10% and imagefs.available<15%), it taints the node with node.kubernetes.io/disk-pressure and evicts pods. Pods using the most ephemeral storage relative to their request go first, and pods without an ephemeral-storage request are evicted earliest.

Typical culprits: large container logs, apps writing to their writable layer or emptyDir, and old unused images.

Solution#

  1. Find the affected node and confirm disk pressure:
kubectl get pods -A -o wide --field-selector=status.phase=Failed kubectl describe node <node-name> | grep -A8 Conditions
  1. On the node, check what is using the space:
df -h /var/lib/kubelet /var/lib/containerd sudo du -xh --max-depth=1 /var/lib/containerd /var/log/pods 2>/dev/null | sort -h | tail
  1. Free space by removing unused images and old journal logs:
sudo crictl rmi --prune sudo journalctl --vacuum-size=500M
  1. Clean up the evicted pods (they are not removed automatically until the pod GC threshold is reached):
kubectl delete pods -A --field-selector=status.phase=Failed
  1. Prevent it from happening again by setting ephemeral storage requests and limits on the workload, so the scheduler accounts for it and only the offending pod is evicted:
resources: requests: ephemeral-storage: "1Gi" limits: ephemeral-storage: "4Gi"
  1. Optionally cap container log size in the kubelet config (/var/lib/kubelet/config.yaml on kubeadm nodes), then restart the kubelet:
containerLogMaxSize: "50Mi" containerLogMaxFiles: 3
sudo systemctl restart kubelet

Tip: If the node disk is simply too small for its workload, grow the disk or move /var/lib/containerd to a dedicated volume instead of lowering the eviction thresholds.

Verify#

kubectl describe node <node-name> | grep DiskPressure kubectl get pods -A --field-selector=status.phase=Failed

DiskPressure should be False, the disk-pressure taint should be gone from kubectl describe node, and no new evicted pods should appear.