Installing Prometheus on Centos 8, NGINX Basic Auth

Prometheus is a free software application used for event monitoring and alerting. It writes metrics in real time to a time series database built using an HTTP request model, with flexible requests and real time alerts.

Installing Prometheus

Add the system user prometheus

$ sudo useradd -M -s /bin/false prometheus

Create the necessary directories for prometheus

$ sudo mkdir /etc/prometheus /var/lib/prometheus
$ sudo chown prometheus /var/lib/prometheus/

Download the latest version of prometheus to the / tmp directory

$ wget https://github.com/prometheus/prometheus/releases/download/v2.19.3/prometheus-2.19.3.linux-amd64.tar.gz -P /tmp
$ cd /tmp

Unpack

$ sudo dnf install tar
$ tar xvzf prometheus-2.19.3.linux-amd64.tar.gz

Install prometheus

$ cd prometheus-2.19.3.linux-amd64
$ sudo cp prometheus  /usr/local/bin
$ sudo cp promtool  /usr/local/bin
$ sudo cp prometheus.yml /etc/prometheus/

Editing the prometheus configuration file

$ sudo nano /etc/prometheus/prometheus.yml
# my global config
global:
  scrape_interval:     15s
  evaluation_interval: 15s

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093 
# Load rules once and periodically evaluate them according to the global 'evalu$
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
# В дальнейшем будем пользоваться схемой: новое правило - новый файл
# rule_files:
#   - /etc/prometheus/rules.d/*.rules.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090'] 

# В дальнейшем добавим авторизацию в node_exporter
  - job_name: 'node'
#    basic_auth:
#      username: prometheus
#      password: password
    static_configs:
    - targets: ['localhost:9100'] 

We open port 19090. We are using a non-standard Prometheus port, so that in the future we would add Basic auth via NGINX

$ sudo firewall-cmd --add-port=19090/tcp --permanent
$ sudo firewall-cmd --reload

Create a System Unit

$ sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Time Series Collection and Processing Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus 
    --config.file /etc/prometheus/prometheus.yml 
    --storage.tsdb.path /var/lib/prometheus/ 
    --web.console.templates=/etc/prometheus/consoles 
    --web.console.libraries=/etc/prometheus/console_libraries 
    --web.external-url http://localhost:19090 
    --web.route-prefix=/

[Install]
WantedBy=multi-user.target

Add the service to autoload and start it

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now prometheus

Installing NGINX and setting up Reverse proxy

Installing the dnf-utils utility

$ sudo dnf -y install dnf-utils

Add NGINX repository

$ sudo nano /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

The stable version will be used by default. If you need a mainline version, switch

$ sudo dnf config-manager --set-enabled nginx-mainline

Install NGINX

$ sudo dnf -y install nginx

Disable the default config

$ cd /etc/nginx/conf.d/
$ sudo mv default.conf default.conf.disable

Create config prometheus.conf

server {
    listen       19090;
    listen       [::]:19090;
    server_name  _;
    location / {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://localhost:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
 
        auth_basic "Prometheus";
        auth_basic_user_file "/etc/nginx/prometheus.htpasswd";
    }
}

Installing the httpd-tools utility

$ sudo dnf -y install httpd-tools

Generating a password

$ sudo htpasswd -c /etc/nginx/prometheus.htpasswd mypomethuser
    New password: password
    Re-type new password: password
    Adding password for user mypomethuser

SELinux configuration

Configuring SELinux

$ cd /tmp
$ sudo grep nginx /var/log/audit/audit.log | grep denied | audit2allow -m nginxlocalconf > nginxlocalconf.te
$ sudo grep nginx /var/log/audit/audit.log | grep denied | audit2allow -M nginxlocalconf
   ******************** IMPORTANT ***********************
   To make this policy package active, execute:
   semodule -i nginxlocalconf.pp
$ sudo semodule -i nginxlocalconf.pp

Add the Nginx service to autoload, start it, see the status

$ sudo systemctl enable --now nginx
$ sudo systemctl status nginx

Done, prometheus with authorization is available at http: //% your_ip%: 19090

Sidebar