Force removal of eviction/terminated Pods in Kubernetes

To
You can download this article in PDF format via the link below to support us.

Download the guide in PDF format

turn off
To

To
To

In this short tutorial, we will study how to delete pods that have been evicted or terminated in a Kubernetes cluster. There are many reasons why you can find the pods in the expelled and terminated state. For eviction, it is usually due to resource pressure or application errors in the working node. The termination may be the result of the termination of the old Pod due to the downsizing of the application or the deployment of a new version of the application.

The kubelet service running on each node in the cluster is responsible for Pod eviction. The order of pods ejection is:

Best effort-QoS level
The explosive pod uses more resources than it requests for hungry resources.
The explosive pod uses less resources than its request for hungry resources.

You can get the list of Pods in the terminated or expelled namespace by running the following command:

kubectl get pods -n namespace | egrep -i 'Terminated|Evicted'

Force removal of eviction/terminated Pods in Kubernetes

You can delete these panes in several ways.

Use kubectl and Bash native commands

These are bash commands with filtering capabilities, and you will run these commands to force deletion of Pods that are stuck in the “exit” or “terminated” namespace.

# Define namespace
namespace="mynamespace"

# Get all pods in Terminated / Evicted State
epods=$(kubectl get pods -n ${namespace} | egrep -i 'Terminated|Evicted' | awk '{print $1 }')

# Force deletion of the pods

for i in ${epods[@]}; do
  kubectl delete pod --force=true --wait=false --grace-period=0 $i -n ${namespace}
done

Check if there are any pods in this state.

kubectl get pods -n ${namespace} | egrep -i 'Terminated|Evicted'

Delete all evicted and terminated pods from all namespaces:

kubectl get pods --all-namespaces | egrep -i  'Evicted|Terminated' | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod --force=true --wait=false --grace-period=0

Remove all containers in the ImagePullBackOff state from all namespaces-bonus:

kubectl get pods --all-namespaces | grep 'ImagePullBackOff' | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod

Remove all containers in the ImagePullBackOff or ErrImagePull state from all namespaces-bonus:

kubectl get pods --all-namespaces | grep -E 'ImagePullBackOff|ErrImagePull|Evicted' | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod

Use kubectl filter and jq

You can also filter the kubectl command output and pipe it to jq to get specific columns.

First install the jq command:

--- Ubuntu / Debian ---
$ sudo apt update && sudo apt install jq

--- CentOS/Fedora ---
$ sudo yum -y install epel-release
$ sudo yum -y install jq

--- RHEL ---
wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 -O jq
chmod +x jq
sudo mv jq /usr/local/bin

Then use the following command to delete the eviction pod:

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | "kubectl delete pods (.metadata.name) -n (.metadata.namespace)"' | xargs -n 1 bash -c

Keep in touch for more interesting guides about containers. Please also check out other related articles on our website.

Install Istio Service Mesh in the EKS Kubernetes cluster

How to decode/decrypt Kubernetes secrets

How to copy Kubernetes secrets between namespaces

Kubernetes courses:

Kubernetes for absolute beginners-hands-on

★★★★★
(16527)

$ 11.81

$ 159.37

In stock

Buy now

Udemy.com