Installing Alertmanager with authorization and connecting to Prometheus in Centos 8

Alertmanagerthis is an alert handling tool that eliminates duplicates, groups, and sends alerts to the appropriate recipient.

Installing Alertmanager

Add user

$ sudo useradd -M -s /bin/false alertmanager

Create directories

$ sudo mkdir /etc/alertmanager /var/lib/prometheus/alertmanager

Download alertmanager to / tmp directory

$ wget https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertmanager-0.21.0.linux-amd64.tar.gz -P /tmp
$ cd /tmp

Unpack and copy to system directories

$ tar -zxpvf alertmanager-0.21.0.linux-amd64.tar.gz
$ cd alertmanager-0.21.0.linux-amd64
$ sudo cp alertmanager amtool /usr/local/bin/
$ sudo cp alertmanager.yml /etc/alertmanager
$ sudo chown -R alertmanager:alertmanager /etc/alertmanager /var/lib/prometheus/alertmanager
$ sudo chown alertmanager:alertmanager /usr/local/bin/{alertmanager,amtool}

Create Systemd Unit

$ sudo nano /etc/systemd/system/alertmanager.service
[Unit]
Description=Alertmanager Service
After=network.target prometheus.service

[Service]
EnvironmentFile=-/etc/default/alertmanager
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
          --config.file=/etc/alertmanager/alertmanager.yml 
          --storage.path=/var/lib/prometheus/alertmanager 
          $ALERTMANAGER_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
Restart=always

[Install]
WantedBy=multi-user.target

Add to startup, start the service, check the status

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now alertmanager
$ sudo systemctl status alertmanager

Check if port 9093 is available

$ ss -tunlp | grep 9093

Setting up authorization

Installing the dnf-utils utility

$ sudo dnf -y install dnf-utils

Add NGINX repository

$ sudo nano /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

The stable version will be used by default. If you need a mainline version, switch

$ sudo dnf config-manager --set-enabled nginx-mainline

Install NGINX

$ sudo dnf -y install nginx

Disable the default config

$ cd /etc/nginx/conf.d/
$ sudo mv default.conf default.conf.disable

Create config alertmanager.conf

$ sudo nano alertmanager.conf
server {
    listen       19093;
    listen       [::]:19093;
    server_name  _;

    location / {
        proxy_set_header Accept-Encoding "";
        proxy_pass http://localhost:9093/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        auth_basic "AlertManager";
        auth_basic_user_file /etc/nginx/alertmanager.htpasswd;
   }
}

Restart NGINX, check the status

$ sudo systemctl restart nginx
$ sudo systemctl status nginx

Editing Systemd Unit Alertmanager

$ sudo nano /etc/systemd/system/alertmanager.service
[Unit]
Description=Alertmanager Service
After=network.target prometheus.service

[Service]
EnvironmentFile=-/etc/default/alertmanager
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
          --config.file=/etc/alertmanager/alertmanager.yml 
          --storage.path=/var/lib/prometheus/alertmanager 
          --web.external-url=http://localhost:19093 
          --web.route-prefix=/ 
          $ALERTMANAGER_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
Restart=always

[Install]
WantedBy=multi-user.target

We restart the service

$ sudo systemctl daemon-reload
$ sudo systemctl restart alertmanager
$ sudo systemctl status alertmanager

Generating a password for authorization

$ sudo htpasswd -c /etc/nginx/alertmanager.htpasswd myalertuser
    New password: passwor
    Re-type new password: password
    Adding password for user myalertuser

Opening port 19093

$ sudo firewall-cmd --add-port=19093/tcp --permanent
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all

SELinux configuration

$ cd /tmp
$ sudo grep nginx /var/log/audit/audit.log | grep denied | audit2allow -m nginxlocalconf > nginxlocalconf.te
$ sudo grep nginx /var/log/audit/audit.log | grep denied | audit2allow -M nginxlocalconf
    ******************** IMPORTANT ***********************
    To make this policy package active, execute:
    semodule -i nginxlocalconf.pp
$ sudo semodule -i nginxlocalconf.pp

Integration with Prometheus

Create a directory where the rules will be located

$ sudo mkdir /etc/prometheus/rules.d/

Let’s create files with notification rules for Promrtheus

$ sudo nano /etc/prometheus/rules.d/alert.rules.yml
groups:
- name: Instance.rules
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute.'
    summary: 'Instance {{ $labels.instance }} down'

- name: Endpoint.rules
  rules:
    - alert: EndpointDown
      expr: probe_success == 0
      for: 10s
      labels:
        severity: critical
      annotations:
        summary: 'Endpoint {{ $labels.instance }} down' 
$ sudo nano /etc/prometheus/rules.d/system.rules.yml
groups:
# Диск забит
- name: Disk-usage
  rules:
  - alert: 'Low data disk space'
    expr: ceil(((node_filesystem_size_bytes{mountpoint!="/boot"} - node_filesystem_free_bytes{mountpoint!="/boot"}) / node_filesystem_size_bytes{mountpoint!="/boot"} * 100)) > 95
    labels:
      severity: "critical"
    annotations:
      title: "Disk Usage"
      description: 'Partition : {{$labels.mountpoint}}'
      summary: "Disk usage is {{humanize $value}}%"
      host: "{{$labels.instance}}" 

# Память забита
- name: Memory-usage
  rules:
  - alert: 'High memory usage'
    expr: ceil((((node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes) / node_memory_MemTotal_bytes) * 100)) > 80
    labels:
      severity: "critical"
    annotations:
      title: "Memory Usage"
      description: 'Memory usage threshold set to 80%.'
      summary: "Memory usage is {{humanize $value}}%"
      host: "{{$labels.instance}}"

# Процессор загружен
- name: CPU-Hight-Load
  rules: 
  - alert: HighSystemLoad
    expr: systemload_average > 90
    for: 5s
    labels:
      severity: "critical"
    annotations:
      title: "Memory Usage"
      summary: "High system load: {{ $value | printf "%.2f" }}%"
      host: "{{$labels.instance}}" 
$ sudo nano /etc/prometheus/rules.d/services.rules.yml
groups:
- name: services.rules
  rules:
    - alert: services
      expr: node_systemd_unit_state{state="active"} == 0
      for: 1s
      annotations:
        summary: "Instance {{ $labels.instance }} is down"
 description: "{{ $labels.instance }} of job {{ $labels.job }} is down." 

Adding a list of rules to prometheus

$ sudo nano /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093 

rule_files:
# - "alert.rules.yml"
# - "system.rules.yml"
# - "web.rules.yml"
# - "services.rules.yml"
  - "/etc/prometheus/rules.d/*.rules.yml" 

scrape_configs:
[…]

Change rights

$ sudo chown -R prometheus. /etc/prometheus/

Checking the rules for errors

$ sudo /usr/local/bin/promtool check rules /etc/prometheus/alert.rules.yml
Checking /etc/prometheus/alert.rules.yml
  SUCCESS: 1 rules found

Restart prometheus

$ sudo systemctl restart prometheus
$ sudo systemctl status prometheus

Create a file with Alertmanager alert settings

$ sudo nano /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
#  smtp_smarthost: 'smtp.gmail.com:587'
#  smtp_from: '[email protected]'
#  smtp_auth_username: 'example@gmail.com'
#  smtp_auth_identity: 'example@gmail.com'
#  smtp_auth_password: 'passwd'

route:
  group_by: [Alertname]
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  # default - send 'info' to email only
  receiver: default

receivers:
- name: default
  email_configs:
  - to: [email protected]
    send_resolved: true
    from: [email protected]
    smarthost: smtp.gmail.com:587
    auth_username: "example@gmail.com"
    auth_identity: "example@gmail.com"
    auth_password: "passwd" 

Restart Alertmanager

$ sudo systemctl restart alertmanager
$ sudo systemctl status alertmanager

AlertManager with smtp works only on ports 25, 587

Tuning Node Exporter

To monitor running services, edit the Systemd Unit node exporter

$ sudo nano /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter 
     --collector.systemd 
     --collector.systemd.unit-whitelist="(sshd|chronyd|nginx).service" 
     --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/) 
     --web.config=/opt/node_exporter/web.yml
[Install]
WantedBy=multi-user.target

We restart the service

$ sudo systemctl daemon-reload
$ sudo systemctl restart node_exporter
Sidebar