Use Prometheus and Grafana to monitor Apache Kafka

To
You can download this article in PDF format via the link below to support us.

Download the guide in PDF format

turn off
To

To
To

In the final guide, we saw how to install Kafka in Ubuntu 20 and CentOS 8. We gave a brief introduction to Kafka and its general functions. As you continue to use Kafka, you will soon notice that you want to monitor the internals of the Kafka server. This is especially important for tracking producers, consumers, and other indicators (such as topics and partitions). In addition, it is beneficial to monitor the host server with Kafka installed so that you can understand its resources and monitor them before they get out of control. To meet the needs you will meet soon, this guide will focus on how to monitor Kafka using the familiar tools Prometheus and Grafana.

Set up prerequisites

In order to make this setup, we will need something that is already busy. First, we will need Kafka Cluster, you can refer to the following guide.

Install and configure Apache Kafka on Ubuntu

Install and configure Apache Kafka with CMAK on CentOS 8

Another requirement is Prometheus settings. If you haven’t installed Prometheus, don’t worry, because we already have a beautiful guide to help you install it quickly. Please follow the “How to install Prometheus guide” to install.

Install Prometheus Server Debian on Ubuntu

How to install Prometheus on RHEL 8 / CentOS 8

Second, we also need to run Grafana. If Grafana is not installed, we are happy to provide another guide. Please follow how to install Grafana to get it up and running.

How to install Grafana 6 on CentOS 7

How to install Grafana on CentOS 8 / RHEL 8

Install Grafana Debian on Ubuntu

Step 1: Download the Prometheus JMX exporter

Prometheus is a powerful and popular open source time series tool and database for storing and publicizing indicators and statistics. Public data can be used as a data source by tools such as Grafana to create beautiful and insightful graphs and charts to better view applications and servers. Apache Kafka is developed in Java, so we will need a Java exporter to grab (extract) metrics so that Prometheus can use, store, and publish.

Prometheus Exporters are used to extract data metrics and export them to your Prometheus instance. One of these exporters is the Java Management Extensions (JMX) exporter, which specializes in Java applications. It enables developers to expose metrics, statistics, and basic operations of Java applications in a standard way that Prometheus understands. Therefore, we will download and install the Prometheus exporter in order to extract Kafka metrics. Visit Mavens Prometheus JMX exporter repository Obtain the jar file. On the server, you can use wget or curl to download, as shown below:

cd ~
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.13.0/jmx_prometheus_javaagent-0.13.0.jar

After downloading the JMX exporter, we will continue to copy it to the lib directory of Kafka, where the jar file is stored. In the previous guide, we copied the Kafka files into the /usr/local/kafka-server/ directory. Therefore, we copy the jmx_prometheus_javaagent jar file to /usr/local/kafka-server/libs/. Make sure you know where the home directory of Kafka is, you will find the libs directory.

sudo cp jmx_prometheus_javaagent-0.13.0.jar /usr/local/kafka-server/libs/

Step 2: Configure our exporter

Next, we must configure the JMX exporter for it to understand what it will extract from Kafka. To briefly illustrate this point, the configuration is a collection of regular expressions that name and filter metrics for Prometheus. Thanks to Prometheus, they are This GitHub repository. In this setup, we will use the kafka-2_0_0.yml sample configuration.

$ cd /usr/local/kafka-server/config/
$ sudo nano sample_jmx_exporter.yml

Copy its content to the file in the config directory in the Kafka home directory.

lowercaseOutputName: true

rules:
# Special cases and very specific rules
- pattern : kafka.server<>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern : kafka.server<>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
- pattern : kafka.coordinator.(w+)<>Value
  name: kafka_coordinator_$1_$2_$3
  type: GAUGE

# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(w+)<>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(w+)<>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(w+)<>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER

- pattern: kafka.server<>([a-z-]+)
  name: kafka_server_quota_$3
  type: GAUGE
  labels:
    resource: "$1"
    clientId: "$2"

- pattern: kafka.server<>([a-z-]+)
  name: kafka_server_quota_$4
  type: GAUGE
  labels:
    resource: "$1"
    user: "$2"
    clientId: "$3"

# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(w+)<>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(w+)<>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
- pattern: kafka.(w+)<>Value
  name: kafka_$1_$2_$3
  type: GAUGE

# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
#
# Note that these are missing the '_sum' metric!
- pattern: kafka.(w+)<>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(w+)<>(d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
    quantile: "0.$8"
- pattern: kafka.(w+)<>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(w+)<>(d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    quantile: "0.$6"
- pattern: kafka.(w+)<>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
- pattern: kafka.(w+)<>(d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    quantile: "0.$4"

Save the file and continue to the next step.

Step 3: Configure Kafka Broker to use JMX exporter

So far, we have everything we need to start extracting Kafka metrics. The only thing left is to link the JMX exporter to our Kafka Broker. Without delay, let us finish it immediately. Open the Kafka Broker server startup script, and then add the JMX configuration at the end of the file, as shown below. All scripts are located in the bin directory in the main Kafka folder.

$ cd /usr/local/kafka-server/bin/
$ sudo vim kafka-server-start.sh

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

if [ $# -lt 1 ];
then
        echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
        exit 1
fi
base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}

COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    shift
    ;;
  *)
    ;;
esac

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "[email protected]"

### ADD THE LINE BELOW ###

export KAFKA_OPTS=' -javaagent:/usr/local/kafka-server/libs/jmx_prometheus_javaagent-0.13.0.jar=7071:/usr/local/kafka-server/config/sample_jmx_exporter.yml'

If you are using systemd, please add this line to the systemd file of kafka [Service] Part as the environment, as follows:

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
##Add the line below
Environment="KAFKA_OPTS=-javaagent:/usr/local/kafka-server/libs/jmx_prometheus_javaagent-0.13.0.jar=7075:/usr/local/kafka-server/config/sample_jmx_exporter.yml"
ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties
ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh
Restart=on-abnormal

After adding this line at the end of the kafka-server-start.sh script or in the systemd file, restart the Kafka agent.

sudo systemctl restart kafka.service

Check whether the service has been started by checking whether the configured port exists. If you are running a firewall and your Prometheus server is on another server, you should consider allowing access to this port.

$ sudo ss -tunelp | grep 7075

tcp    LISTEN     0      3      [::]:7075    [::]:*   users:(("java",pid=31609,fd=100)) uid:1000 ino:5391132 sk:ffff977c74f86b40 v6only:0 <->

Allow ports on firewall

----Ubuntu----
$ sudo ufw allow 7075

----CentOS----
$ sudo firewall-cmd --permanent --add-port=7075/tcp
$ sudo firewall-cmd --reload

Open a browser and point it to the IP or FQDN of the server and port. http: //[IP or FQDN]: 7075. You should see data metrics as shown below

good stuff! Our JMX exporter is working as expected. Now let’s start adding the data exposed to Prometheus

Step 4: Add Kafka data to Prometheus

Log in to the Prometheus server and configure this new source as a data target. If you follow this guide to install Prometheus on Debian | Ubuntu or RHEL 8 | CentOS 8, its configuration file is located in /etc/prometheus/prometheus.yml. Please find its configuration file, open and edit, as shown in the figure below

$ sudo vim /etc/prometheus/prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.

##### CHANGE THE JOB NAME TO KAFKA AS BELOW#######

  - job_name: 'kafka'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

#####CHANGE THE TARGET TO THE IP AND PORT OF JMX SERVICE JUST INSTALLED#######
    static_configs:
    - targets: ['10.38.83.154:7075']

You can confirm that this target has been successfully added to the Prometheus web interface. Open it with a browser and click Status>Target. If the addition is successful, you will see the following figure.

Its status should be “up”

So far, this is great. Next, we will use the data that Prometheus will store as Grafana’s data source so that we can view the metrics of the style.

Step 5: Add Kafka metrics to Grafana

Now we are at the last and best part. Here, we add Prometheus as a data source, and then use beautiful graphs and charts to visualize it. Log in to your Grafana web interface and perform the following operations. If you have not installed Grafana, please follow the guide below to quickly install it.

How to install Grafana on CentOS 8 / RHEL 8

Install Grafana Debian on Ubuntu

How to install Grafana on CentOS 7

Add data source

After entering the Grafana web interface, click the settings gear icon and select “data sourceThe “” option in the drop-down list.

This will open the “Data Sources” menu, where you can add more content. Click “Add data source“Label

As you might guess, we will choose Prometheus Because that is what we have configured before.

After selecting the Prometheus data source, we will have to tell Grafana where to find the Prometheus server. Enter a cute name next to the URL and the IP and port where Prometheus is running.

You can further add Crawl interval, query timeout with HTTP method. After that, click “Save and test“Button. If everything goes well, a green message will appear. If there is an error, please make sure your Prometheus server is running and accessible. If it is behind a firewall, open its ports.

Import dashboard

After adding the data source, we will continue to add the dashboard, which will visualize the content in the data source. While still on Grafana, click + Button and select import Because we will use the already created dashboard Robust perception. Its id is 721

On the import page, issue the ID 721 then click”load“Button.

The next page will ask you to enter a name, then you should select the data source we added in the drop-down menu at the bottom of the page. When you are done, just click “import“.

And you should display the metrics nicely as the following share.

in conclusion

Now your Kafka metrics can be displayed well on Grafana, so you can get a deeper understanding of your topic and other topics. Thank you for your continued support, and hope this guide will help you. We thank all the creators of the tools used in this guide to improve the lives of developers and administrators. For other guidelines similar to this one, please review the following.

Install and configure Telegraf on RHEL 8 / CentOS 8

Deploy Prometheus on the EKS Kubernetes cluster

How to install Prometheus and node_exporter on Debian 10 (Buster)

Use Prometheus and Grafana to monitor Etcd clusters

Use Prometheus and Grafana to monitor Redis server in 5 minutes

Use Prometheus and Grafana to monitor Linux server performance in 5 minutes

How to use Prometheus and Grafana to monitor Apache Web Server in 5 minutes

Use Prometheus and Grafana to monitor the BIND DNS server