Deploy Prometheus on the EKS Kubernetes cluster

To
You can download this article in PDF format via the link below to support us.

Download the guide in PDF format

turn off
To

To
To

Prometheus is an open source monitoring system with dimensional data model, flexible query language, efficient time series database and modern alarm methods. Prometheus can be installed on Linux machines as a standalone service or deployed in a Kubernetes cluster. In this guide, we will guide you through the installation of Prometheus on an EKS cluster deployed in AWS Cloud.

The main features of Prometheus are:

  • Multidimensional Data model Has time series data identified by metric names and key/value pairs
  • PromQL, one Flexible query language Use this dimension
  • Does not rely on distributed storage; single server nodes are autonomous
  • Time series collection is performed through the pull model on HTTP
  • Push time series Support through intermediate gateway
  • Discover targets through service discovery or static configuration
  • Multiple graphics and dashboard support modes

Deploy Prometheus on the EKS Kubernetes cluster

You must run an EKS cluster before you can use this guide with the installed kubectl and be able to interact with the API server.

List the available clusters.

$ eksctl get cluster
NAME			REGION
prod-eks-cluster	eu-west-1

Step 1: Configure kubectl

To obtain cluster credentials from the EKS cluster deployed via eksctl at any time, run:

$ eksctl utils write-kubeconfig --cluster= [--kubeconfig=][--set-kubeconfig-context=]
See below example:
# Create default kubeconfig directory - Can be custom
--- Linux ---
$ mkdir /home/$USER/.kube

--- macOS ---
$ mkdir /Users/$USER/.kube

# Obtain cluster credentials
--- Linux ---
$ s write-kubeconfig --cluster=prod-eks-cluster --kubeconfig=/home/$USER/.kube/config

--- macOS ---
$ eksctl utils write-kubeconfig --cluster=prod-eks-cluster --kubeconfig=/Users/$USER/.kube/config

Expected command execution output:

[ℹ]  eksctl version 0.25.0
[ℹ]  using region eu-west-1
[✔]  saved kubeconfig as "/Users/jkmutai/.kube/config"

Confirm that the kubectl command works as expected.

$ kubectl get  nodes
NAME                                            STATUS   ROLES    AGE     VERSION
ip-192-168-138-244.eu-west-1.compute.internal   Ready       7d21h   v1.17.9-eks-4c6976
ip-192-168-176-247.eu-west-1.compute.internal   Ready       7d21h   v1.17.9-eks-4c6976

Step 2: Install Kubernetes Metrics Server

The Kubernetes API server exposes many indicators useful for monitoring and analysis. These indicators are disclosed internally through the indicator endpoint, which references /Index HTTP API.

Follow the guidelines below to install Metrics Server:

Install Kubernetes Metrics Server on Amazon EKS cluster

You can view the raw metrics of the cluster by running the following command:

$ kubectl get --raw /metrics

Step 3: Install the helmet

The Helm package manager for Kubernetes helps you install and manage applications on a Kubernetes cluster. We will help you deploy Prometheus on EKS.

These are the commands you use to install the helmet.

--- Linux ---
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

--- macOS ---
$ brew install helm

--- Windows ---
$ choco install kubernetes-helm

You can use the following command to query the installed version:

$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"dirty", GoVersion:"go1.14.3"}

Step 4: Deploy Prometheus on the EKS Kubernetes cluster

Prometheus can be installed on a Kubernetes cluster using Operator or helmet. We will use the Helm V3 we just installed to deploy Prometheus in the EKS cluster.

First create a monitoring namespace.

$ kubectl create namespace monitoring
namespace/monitoring created

Prometheus needed a way to retain indicator data for historical reference. We will use the EBS provided with the gp2 storage class.

$ kubectl get sc
kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
efs-sc          efs.csi.aws.com         Delete          Immediate              false                  2d23h
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  7d21h

Add a chart repository:

$ helm repo add stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositories

Use Helm to deploy Prometheus.

helm install prometheus stable/prometheus 
    --namespace monitoring 
    --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

Command execution output:

NAME: prometheus
LAST DEPLOYED: Wed Aug 19 22:01:16 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local

Confirm that PV and PVC have been created.

$ kubectl get pv -n monitoring
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS   REASON   AGE
pvc-7a1d891c-5b3f-4d28-91b5-5f1d1590daf9   8Gi        RWO            Delete           Bound    monitoring/prometheus-server         gp2                     11m
pvc-7d45951f-a8c5-4fb1-b149-747dccaceecc   2Gi        RWO            Delete           Bound    monitoring/prometheus-alertmanager   gp2                     11m

$ kubectl get pvc -n monitoring
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus-alertmanager   Bound    pvc-7d45951f-a8c5-4fb1-b149-747dccaceecc   2Gi        RWO            gp2            11m
prometheus-server         Bound    pvc-7a1d891c-5b3f-4d28-91b5-5f1d1590daf9   8Gi        RWO            gp2            11m

Step 4: Access Prometheus on the EKS Kubernetes cluster

After installation, query all resources in the monitoring namespace:

$ kubectl get all -n monitoring
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/prometheus-alertmanager-d47577c4b-kz9td          2/2     Running   0          98s
pod/prometheus-kube-state-metrics-6df5d44568-fzfwn   1/1     Running   0          98s
pod/prometheus-node-exporter-p6qk7                   1/1     Running   0          99s
pod/prometheus-node-exporter-stsjk                   1/1     Running   0          99s
pod/prometheus-pushgateway-57c97d878d-hmnwr          1/1     Running   0          98s
pod/prometheus-server-559c49b4ff-42k46               2/2     Running   0          98s

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/prometheus-alertmanager         ClusterIP   10.100.136.125           80/TCP     99s
service/prometheus-kube-state-metrics   ClusterIP   10.100.169.64            8080/TCP   99s
service/prometheus-node-exporter        ClusterIP   None                     9100/TCP   99s
service/prometheus-pushgateway          ClusterIP   10.100.71.173            9091/TCP   99s
service/prometheus-server               ClusterIP   10.100.138.66            80/TCP     99s

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   2         2         2       2            2                     99s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-alertmanager         1/1     1            1           99s
deployment.apps/prometheus-kube-state-metrics   1/1     1            1           99s
deployment.apps/prometheus-pushgateway          1/1     1            1           99s
deployment.apps/prometheus-server               1/1     1            1           99s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-alertmanager-d47577c4b          1         1         1       99s
replicaset.apps/prometheus-kube-state-metrics-6df5d44568   1         1         1       99s
replicaset.apps/prometheus-pushgateway-57c97d878d          1         1         1       99s
replicaset.apps/prometheus-server-559c49b4ff               1         1         1       99s

Obtain the Prometheus server URL by running the following command in the same shell:

export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")

Use Kubernetes port forwarding to access Prometheus Server.

$ kubectl --namespace monitoring port-forward $POD_NAME 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Access 127.0.0.1:9090 from the browser of the local computer.

Similar commands are used to access the Alertmanager UI

export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9093

If you want to use the node port to access Prometheus Server and Alertmanager, edit the service and set the NodePort.

The next place is the Prometheus documentation, especially Query Prothetheus

Use horizontal Pod autoscaler on Kubernetes EKS cluster

Enable CloudWatch logging in the EKS Kubernetes cluster

EKS Kubernetes persistent storage using EFS storage service

To
You can download this article in PDF format via the link below to support us.

Download the guide in PDF format

turn off
To

To
To

Sidebar