How to use Prometheus to monitor the uptime of a Linux server

You can download this article in PDF format via the link below to support us.
Download the guide in PDF formatturn off

“Some fish like to swim upstream. Some people like to overcome challenges.”-Amit Ray

Ensure that your infrastructure is maintained at the high SLA levels that companies and organizations always require, then you have no choice but to find ways to measure, observe, and submit reports that capture the truth about system performance. When asking those cold questions, there are no shortcuts. You can choose to present a blank, confused face, or a face full of warmth, confidence and sufficient specific reports.

In order to achieve the latter, this guide will guide us step by step to understand how to capture the uptime of the Linux Server system. For this, we must use a tool called Prometheus. We know that you have heard of Prometheus, but there is an excuse to reimplement it, please allow us to introduce it again. Prometheus is a monitoring tool designed to record real-time metrics in a time series database. This is an open source software project written in the Go language. Prometheus metrics are collected using HTTP pulls, thereby achieving higher performance and scalability.

In this tutorial, as the title suggests, we will discuss how to use Prometheus to monitor the uptime of a Linux server. We will install it and use its powerful features to get the data we are interested in. So we will start.

Step 1: Prepare the server

Depending on the environment the server is in, you should update and install all the basic software packages that you usually use, such as text editors (vim or nano), wget or curl, and other packages that you cannot live without.

####For Ubuntu####

sudo apt update && sudo apt upgrade
sudo apt install vim wget curl -y

####For CentOS####

sudo yum update
sudo yum install vim wget curl -y

Step 2: Install Prometheus

Since Prometheus is the center of this guide, we will continue to set up and make sure we are ready to serve us. Fortunately for us, we have a comprehensive installation guide. Whether you are using CentOS or Debian or Ubuntu, you can sort them. Check the following link to install Prometheus.

Install Prometheus Server DebianHow on Ubuntu Install Prometheus CentOS 8 on RHEL

Install node_exporter

First, let’s create a Node Exporter system user, which will help to run it safely. Create the following users:

sudo useradd -M -r -s /bin/false node_exporter

After that, download and install Node Exporter as shown below.Point your browser to Prometheus download page And get the latest version of Node Exporter.

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest| grep browser_download_url| grep linux-amd64| cut -d '"' -f 4| wget -qi -

To install Node Exporter, we just need to copy the node_exporter binary file from the archive folder to /usr/local/bin in the PATH environment variable.

tar -xvf node_exporter*.tar.gz
cd  node_exporter*/
sudo cp node_exporter /usr/local/bin

Create the node_exporter service.

cat << EOF | sudo tee /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target
EOF

Reload Systemd and start the service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

After that, if you are running a firewall, you must allow port 9100 used by node_exporter by default.

##On Ubuntu

sudo ufw allow 9100

##On CentOS

sudo firewall-cmd --add-port=9100/tcp --permanent && sudo firewall-cmd --reload

Step 3: Add the target to the Monitor via Prometheus

On the target node, install only node_exporter

Install node_exporter

Create node exporter system user

sudo useradd -M -r -s /bin/false node_exporter

After that, download and install Node Exporter as shown below.Point your browser to Prometheus download page And get the latest version of Node Exporter.

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest| grep browser_download_url| grep linux-amd64| cut -d '"' -f 4| wget -qi -

To install Node Exporter, we just need to copy the node_exporter binary file from the archive folder to /usr/local/bin.

tar -xvf node_exporter*.tar.gz
cd  node_exporter*/
sudo cp node_exporter /usr/local/bin

Verify the installation by checking the software version:

$ node_exporter --version
node_exporter, version 1.0.1 (branch: HEAD, revision: 3715be6ae899f2a9b9dbfd9c39f3e09a7bd4559f)
  build user:       [email protected]
  build date:       20200616-12:44:12
  go version:       go1.14.4

Create the node_exporter service.

cat << EOF | sudo tee /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target
EOF

Reload systemd and start the service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

After that, if you are running a firewall, you must allow port 9100 used by node_exporter by default.

##On Ubuntu

sudo ufw allow 9100

##On CentOS

sudo firewall-cmd --add-port=9100/tcp --permanent && sudo firewall-cmd --reload

Update Prometheus server configuration

After installing node_exporter, we will have to update the Prometheus server configuration file to get the target of the new node/server.

Add the section to the end of the configuration file as shown below

$ sudo vim /etc/prometheus/prometheus.yml

# This global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: node
    static_configs:
      - targets: ['172.28.204.30:9100']

Then restart Prometheus

sudo systemctl restart prometheus

When you return to the browser, you should see the following page with a new node in the list

Step 4: Add the black box exporter

The black box exporter allows black box detection of endpoints via HTTP, HTTPS, DNS, TCP and ICMP. It runs independently, and then Prometheus binds to it, makes requests to the endpoints exposed by the black box, and performs operations as configured.

First, let’s create a black box exporter system user, which will help to run it safely. Create the following users:

sudo useradd -M -r -s /bin/false blackbox

Install the black box exporter

We will get and install Blackbox Exporter from it Latest version page

cd ~
curl -s https://api.github.com/repos/prometheus/blackbox_exporter/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -

Unzip the downloaded file and copy the binary file to the following directory path Environment variable

tar xvzf blackbox_exporter-*.linux-amd64.tar.gz
cd blackbox_exporter-*.linux-amd64
sudo cp blackbox_exporter /usr/local/bin

Confirm installation:

$ blackbox_exporter --version
blackbox_exporter, version 0.18.0 (branch: HEAD, revision: 60c86e6ce5a1111f7958b06ae7a08222bb6ec839)
  build user:       [email protected]
  build date:       20201012-09:46:31
  go version:       go1.15.2

Ensure that the black box can read binary files

$ sudo chown blackbox:blackbox /usr/local/bin/blackbox_exporter

Before leaving the same directory, we will create a black box configuration file and ensure that the black box user has the necessary permissions to read the file, as shown below

cd ~/blackbox_exporter-*.linux-amd64
sudo mkdir -p /etc/blackbox
sudo cp blackbox.yml /etc/blackbox
sudo chown -R blackbox:blackbox /etc/blackbox/*

After copying the binary file to the directory path, We will continue to create a Systemd service file for it, as shown below:

$ sudo vim /etc/systemd/system/blackbox.service

[Unit]
Description=Blackbox Exporter Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter 
  --config.file=/etc/blackbox/blackbox.yml 
  --web.listen-address=":9115"

Restart=always

[Install]
WantedBy=multi-user.target

As you can see from the service file, the black box will listen on port 9115. You can guess that if you have a port running, we will need to allow that port on the firewall.

##On Ubuntu

sudo ufw allow 9115

##On CentOS

sudo firewall-cmd --add-port=9115/tcp --permanent && sudo firewall-cmd --reload

Adding a service file requires reloading the daemon before we can start the black box service. Execute the following command to get our black box exporter alert.

sudo systemctl daemon-reload
sudo systemctl enable blackbox
sudo systemctl start blackbox

You can then check the status to confirm that the exporter is indeed running:

$ sudo systemctl status blackbox

● blackbox.service - Blackbox Exporter Service
     Loaded: loaded (/etc/systemd/system/blackbox.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-12-03 08:13:03 UTC; 4min 20s ago
   Main PID: 2161 (blackbox_export)
      Tasks: 7 (limit: 2204)
     Memory: 2.2M
     CGroup: /system.slice/blackbox.service
             └─2161 /usr/local/bin/blackbox_exporter --config.file=/etc/blackbox/blackbox.yml --web.listen-address=:9115 

Once this point is reached, after pointing the browser to “http://ip-or-fqdn:9115”, we should be able to access the black box exporter web interface shared in the screenshot below.How to use Prometheus to monitor the uptime of a Linux server

Step 5: Use Blackbox Exporter to monitor endpoints

Now that we have installed and running Blackbox Exporter, it is time to use the gems it provides to make our lives easier. If you have any http or https endpoints to monitor, Blackbox Exporter comes with some modules to help you accomplish this. Think of modules as soldiers exploring HTTP, HTTPS, DNS, TCP, and ICMP on behalf of exporters.

We will monitor the simple endpoints present in the default black box exporter profile by binding it to our prometheus server. The default black box exporter configuration file comes with predefined modules, which we can use to bind to our prometheus server. You can view the modules in the “/etc/blackbox/blackbox.yml” file where blackbox is installed.

Bind the Blackbox exporter to Prometheus

On the Prometheus server, add the following configuration at the end of the file that reads the module on the Blackbox Exporter.

$ sudo vim /etc/prometheus/prometheus.yml

  # any [job_name]
  - job_name: 'ICMP_Probe_Blackbox'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        # The hostname or IP address of the host you are targeting
        - 172.28.204.30
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # Where Blackbox exporter was installed plust port
        replacement: 172.28.204.30:9115

  # Using the [ssh_banner] module
  - job_name: 'SSH_Probe_Blackbox'
    metrics_path: /probe
    params:
      module: [ssh_banner]
    static_configs:
      - targets:
        # The host you are targeting plus ssh port
        - 172.28.204.30:22
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # Where Blackbox exporter was installed plust port
        replacement: 172.28.204.30:9115

  # Using the [http_2xx] module
  - job_name: 'Blackbox_tcp'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        # The http host you are targeting
        - http://172.28.204.30
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        #Where Blackbox exporter was installed plust port
        replacement: 172.28.204.30:9115

Then restart Prometheus

sudo systemctl restart prometheus

When we go to the Prometheus web interface and refresh, we should see in the screenshot below that all the added probes are shared.How to use Prometheus to monitor the uptime of a Linux server

Step 6: View the probe

We can check the status of the probe on your Prometheus interface. To do this, click the “Graph” tab, then write it as “Expression” in the search area, type the probe, and many suggestions will pop up.How to use Prometheus to monitor the uptime of a Linux server

We will take probe_success as an example, and then click “Execute”How to use Prometheus to monitor the uptime of a Linux server

After selecting probe_success, click the “Graphics” tab below it, as shown below:How to use Prometheus to monitor the uptime of a Linux server

You should see the successful probe as shownHow to use Prometheus to monitor the uptime of a Linux server

Step 7: Visualize everything on Grafana

Grafana provides features with visual value, and we can’t help but forget to add it to this surveillance ecosystem. If you don’t have Grafana installed, we will provide an excellent guide to help you get started quickly. They are shared for you below:

How to install Grafana on CentOS 8 / RHEL 8 Install Grafana on FedoraSetup Install Grafana 7 on Ubuntu. Debian Linux

Once Grafana is installed, make sure you can access the IP and port of the Prometheus server, and then proceed to add it as a data source as shown below. Then log in:

Move your mouse over the settings gear icon and select “data sourceHow to use Prometheus to monitor the uptime of a Linux server

then click”Add data sourceHow to use Prometheus to monitor the uptime of a Linux server

Choose PrometheusHow to use Prometheus to monitor the uptime of a Linux server

Enter the required details, such as where Promethueus is running, etc. You can also give it a name.How to use Prometheus to monitor the uptime of a Linux server

When finished, scroll down and click “Save and test“. If all goes well, you will see a good message in green as shown belowHow to use Prometheus to monitor the uptime of a Linux server

After adding the data source, we need to add a dashboard that will summarize what we need to view/monitor.Move the mouse to + Icon and click “importHow to use Prometheus to monitor the uptime of a Linux server

Enter the information center with ID as via Sparanoid 7587. Click”import”How to use Prometheus to monitor the uptime of a Linux server

A new page is displayed with details of the creator of the dashboard. Select your Prometheus data source and click “importHow to use Prometheus to monitor the uptime of a Linux server

After clicking “Import”, its gorgeous dashboard should appear as the following shareHow to use Prometheus to monitor the uptime of a Linux server

We should complete the monitoring setup

Final words

Prometheus has a variety of functions, in addition to crawling indicators for visualization, you can also do more. It can be your monitoring server to get the status of sshd, http and https endpoints, your MySQL port, etc.The next place is the Prometheus documentation, especially Query Prothetheus

Use Prometheus and Grafana to monitor Apache Kafka

Use Prometheus and Grafana to monitor Linux server performance in 5 minutes

You can download this article in PDF format via the link below to support us.
Download the guide in PDF formatturn off

Sidebar