Issue#
A pod keeps restarting, and its status sometimes shows CrashLoopBackOff or OOMKilled. The application logs stop suddenly and show no error.
kubectl get pods
NAME READY STATUS RESTARTS AGE
api-6d8f7c9b5d-x2kqp 0/1 OOMKilled 5 (40s ago) 6m
kubectl describe pod shows the reason in the previous container state:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Cause#
The container used more memory than its resources.limits.memory. When that happens, the kernel's cgroup OOM killer sends SIGKILL to the process. Exit code 137 means 128 + 9 (SIGKILL). The process is killed immediately, so it can't log anything first.
Common reasons:
- The memory limit is set too low for the real workload.
- A runtime ignores the container limit and sizes its heap from the node's memory. Older JVMs and some Node.js setups do this.
- The application has a memory leak.
Solution#
- Confirm that the restarts are OOM kills and check the current limit:
POD=api-6d8f7c9b5d-x2kqp
kubectl get pod "$POD" \
-o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}'
kubectl get pod "$POD" \
-o jsonpath='{.spec.containers[*].resources}{"\n"}'
- Check how much memory the pod actually uses (this requires metrics-server):
kubectl top pod "$POD" --containers
- Raise the memory request and limit on the deployment:
kubectl set resources deployment/api -c api \
--requests=memory=256Mi --limits=memory=512Mi
Or set them in the manifest:
resources:
requests:
memory: "256Mi"
limits:
memory: "512Mi"
- For JVM applications, make the heap size follow the container limit instead of the node's memory:
env:
- name: JAVA_TOOL_OPTIONS
value: "-XX:MaxRAMPercentage=75.0"
For Node.js, set the heap size explicitly, below the container limit:
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=384"
Tip: If memory keeps growing until the pod is killed at any limit you set, the application has a leak. Raising the limit only delays the next restart.
Verify#
Watch the restart count. It should stop increasing:
kubectl get pods -w
Then confirm that the new limit is applied and that usage stays below it:
kubectl get pod -l app=api \
-o jsonpath='{.items[*].spec.containers[*].resources.limits.memory}{"\n"}'
kubectl top pod -l app=api --containers