Issue#

A pod keeps restarting, and its status sometimes shows CrashLoopBackOff or OOMKilled. The application logs stop suddenly and show no error.

kubectl get pods
NAME READY STATUS RESTARTS AGE api-6d8f7c9b5d-x2kqp 0/1 OOMKilled 5 (40s ago) 6m

kubectl describe pod shows the reason in the previous container state:

Last State: Terminated Reason: OOMKilled Exit Code: 137

Cause#

The container used more memory than its resources.limits.memory. When that happens, the kernel's cgroup OOM killer sends SIGKILL to the process. Exit code 137 means 128 + 9 (SIGKILL). The process is killed immediately, so it can't log anything first.

Common reasons:

  • The memory limit is set too low for the real workload.
  • A runtime ignores the container limit and sizes its heap from the node's memory. Older JVMs and some Node.js setups do this.
  • The application has a memory leak.

Solution#

  1. Confirm that the restarts are OOM kills and check the current limit:
POD=api-6d8f7c9b5d-x2kqp kubectl get pod "$POD" \ -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}' kubectl get pod "$POD" \ -o jsonpath='{.spec.containers[*].resources}{"\n"}'
  1. Check how much memory the pod actually uses (this requires metrics-server):
kubectl top pod "$POD" --containers
  1. Raise the memory request and limit on the deployment:
kubectl set resources deployment/api -c api \ --requests=memory=256Mi --limits=memory=512Mi

Or set them in the manifest:

resources: requests: memory: "256Mi" limits: memory: "512Mi"
  1. For JVM applications, make the heap size follow the container limit instead of the node's memory:
env: - name: JAVA_TOOL_OPTIONS value: "-XX:MaxRAMPercentage=75.0"

For Node.js, set the heap size explicitly, below the container limit:

env: - name: NODE_OPTIONS value: "--max-old-space-size=384"

Tip: If memory keeps growing until the pod is killed at any limit you set, the application has a leak. Raising the limit only delays the next restart.

Verify#

Watch the restart count. It should stop increasing:

kubectl get pods -w

Then confirm that the new limit is applied and that usage stays below it:

kubectl get pod -l app=api \ -o jsonpath='{.items[*].spec.containers[*].resources.limits.memory}{"\n"}' kubectl top pod -l app=api --containers