How to fix Evicted pods in Kubernetes

1. Find the evicted pods

kubectl get pods -A --field-selector=status.phase=Failed | grep Evicted

2. Read the reason

Under Message, describe names the resource that ran out, such as "The node was low on resource: memory" or "ephemeral-storage":

kubectl describe pod <pod> -n <namespace>

3. Check the pressure on the node

The node’s Conditions show MemoryPressure, DiskPressure or PIDPressure, and the events show when the kubelet started evicting:

kubectl describe node <node>
kubectl top nodes

Who goes first

• The kubelet first evicts pods using more than their requests, then lower-priority pods and, among them, those furthest over their request.
• Pods with no requests or limits (BestEffort QoS) are the first candidates; pods with requests equal to limits (Guaranteed QoS) are the last.

How to prevent it

• Memory: set requests.memory close to real usage, so the scheduler does not pack the node with pods that ask for little and use a lot.
• Disk: logs written inside the container and emptyDir volumes count as ephemeral-storage. Set ephemeral-storage requests and limits and send logs to stdout.
• Large images and many old versions also fill the node’s disk; the kubelet garbage-collects images, but a disk that is too small lives at the limit.
• Give a higher PriorityClass to workloads that must not go first.

Clean up Evicted pods

Evicted pods stay listed as Failed until the cluster’s garbage collector removes them. To clean up a namespace:

kubectl delete pods -n <namespace> --field-selector=status.phase=Failed

Without a terminal, in Kubepier

In Kubepier, the pod status shows Evicted, node pressure warning events rise to the top and CPU and memory usage shows per node and namespace, from any cluster, in the browser or on your phone.