Force removal of eviction/terminated Pods in Kubernetes
To
You can download this article in PDF format via the link below to support us.
Download the guide in PDF format
turn off
To
To
To
In this short tutorial, we will study how to delete pods that have been evicted or terminated in a Kubernetes cluster. There are many reasons why you can find the pods in the expelled and terminated state. For eviction, it is usually due to resource pressure or application errors in the working node. The termination may be the result of the termination of the old Pod due to the downsizing of the application or the deployment of a new version of the application.
The kubelet service running on each node in the cluster is responsible for Pod eviction. The order of pods ejection is:
- Best effort-QoS level
- The explosive pod uses more resources than it requests for hungry resources.
- The explosive pod uses less resources than its request for hungry resources.
You can get the list of Pods in the terminated or expelled namespace by running the following command:
kubectl get pods -n namespace | egrep -i 'Terminated|Evicted'
Force removal of eviction/terminated Pods in Kubernetes
You can delete these panes in several ways.
Use kubectl and Bash native commands
These are bash commands with filtering capabilities, and you will run these commands to force deletion of Pods that are stuck in the “exit” or “terminated” namespace.
# Define namespace
namespace="mynamespace"
# Get all pods in Terminated / Evicted State
epods=$(kubectl get pods -n ${namespace} | egrep -i 'Terminated|Evicted' | awk '{print $1 }')
# Force deletion of the pods
for i in ${epods[@]}; do
kubectl delete pod --force=true --wait=false --grace-period=0 $i -n ${namespace}
done
Check if there are any pods in this state.
kubectl get pods -n ${namespace} | egrep -i 'Terminated|Evicted'
Delete all evicted and terminated pods from all namespaces:
kubectl get pods --all-namespaces | egrep -i 'Evicted|Terminated' | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod --force=true --wait=false --grace-period=0
Remove all containers in the ImagePullBackOff state from all namespaces-bonus:
kubectl get pods --all-namespaces | grep 'ImagePullBackOff' | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod
Remove all containers in the ImagePullBackOff or ErrImagePull state from all namespaces-bonus:
kubectl get pods --all-namespaces | grep -E 'ImagePullBackOff|ErrImagePull|Evicted' | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod
Use kubectl filter and jq
You can also filter the kubectl command output and pipe it to jq to get specific columns.
First install the jq command:
--- Ubuntu / Debian ---
$ sudo apt update && sudo apt install jq
--- CentOS/Fedora ---
$ sudo yum -y install epel-release
$ sudo yum -y install jq
--- RHEL ---
wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 -O jq
chmod +x jq
sudo mv jq /usr/local/bin
Then use the following command to delete the eviction pod:
kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | "kubectl delete pods (.metadata.name) -n (.metadata.namespace)"' | xargs -n 1 bash -c
Keep in touch for more interesting guides about containers. Please also check out other related articles on our website.
Install Istio Service Mesh in the EKS Kubernetes cluster
How to decode/decrypt Kubernetes secrets
How to copy Kubernetes secrets between namespaces
Kubernetes courses:
Kubernetes for absolute beginners-hands-on
★★★★★
(16527)
$ 11.81
$ 159.37
In stock
Udemy.com
Certified Kubernetes Administrator (CKA) and practice test
★★★★★
(14265)
$ 11.81
$ 159.37
In stock
Udemy.com
Kubernetes Certified Application Developer (CKAD) with testing capabilities
★★★★★
(6721)
$ 11.81
$ 159.37
In stock
Udemy.com
You can download this article in PDF format via the link below to support us.
Download the guide in PDF format
turn off
To
To
To