Use Prometheus and Grafana to monitor Apache Kafka
To
You can download this article in PDF format via the link below to support us.
Download the guide in PDF format
turn off
To
To
To
In the final guide, we saw how to install Kafka in Ubuntu 20 and CentOS 8. We gave a brief introduction to Kafka and its general functions. As you continue to use Kafka, you will soon notice that you want to monitor the internals of the Kafka server. This is especially important for tracking producers, consumers, and other indicators (such as topics and partitions). In addition, it is beneficial to monitor the host server with Kafka installed so that you can understand its resources and monitor them before they get out of control. To meet the needs you will meet soon, this guide will focus on how to monitor Kafka using the familiar tools Prometheus and Grafana.
Set up prerequisites
In order to make this setup, we will need something that is already busy. First, we will need Kafka Cluster, you can refer to the following guide.
Install and configure Apache Kafka on Ubuntu
Install and configure Apache Kafka with CMAK on CentOS 8
Another requirement is Prometheus settings. If you haven’t installed Prometheus, don’t worry, because we already have a beautiful guide to help you install it quickly. Please follow the “How to install Prometheus guide” to install.
Install Prometheus Server Debian on Ubuntu
How to install Prometheus on RHEL 8 / CentOS 8
Second, we also need to run Grafana. If Grafana is not installed, we are happy to provide another guide. Please follow how to install Grafana to get it up and running.
How to install Grafana 6 on CentOS 7
How to install Grafana on CentOS 8 / RHEL 8
Install Grafana Debian on Ubuntu
Step 1: Download the Prometheus JMX exporter
Prometheus is a powerful and popular open source time series tool and database for storing and publicizing indicators and statistics. Public data can be used as a data source by tools such as Grafana to create beautiful and insightful graphs and charts to better view applications and servers. Apache Kafka is developed in Java, so we will need a Java exporter to grab (extract) metrics so that Prometheus can use, store, and publish.
Prometheus Exporters are used to extract data metrics and export them to your Prometheus instance. One of these exporters is the Java Management Extensions (JMX) exporter, which specializes in Java applications. It enables developers to expose metrics, statistics, and basic operations of Java applications in a standard way that Prometheus understands. Therefore, we will download and install the Prometheus exporter in order to extract Kafka metrics. Visit Mavens Prometheus JMX exporter repository Obtain the jar file. On the server, you can use wget or curl to download, as shown below:
cd ~
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.13.0/jmx_prometheus_javaagent-0.13.0.jar
After downloading the JMX exporter, we will continue to copy it to the lib directory of Kafka, where the jar file is stored. In the previous guide, we copied the Kafka files into the /usr/local/kafka-server/ directory. Therefore, we copy the jmx_prometheus_javaagent jar file to /usr/local/kafka-server/libs/. Make sure you know where the home directory of Kafka is, you will find the libs directory.
sudo cp jmx_prometheus_javaagent-0.13.0.jar /usr/local/kafka-server/libs/
Step 2: Configure our exporter
Next, we must configure the JMX exporter for it to understand what it will extract from Kafka. To briefly illustrate this point, the configuration is a collection of regular expressions that name and filter metrics for Prometheus. Thanks to Prometheus, they are This GitHub repository. In this setup, we will use the kafka-2_0_0.yml sample configuration.
$ cd /usr/local/kafka-server/config/
$ sudo nano sample_jmx_exporter.yml
Copy its content to the file in the config directory in the Kafka home directory.
lowercaseOutputName: true
rules:
# Special cases and very specific rules
- pattern : kafka.server<>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern : kafka.server<>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
broker: "$4:$5"
- pattern : kafka.coordinator.(w+)<>Value
name: kafka_coordinator_$1_$2_$3
type: GAUGE
# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(w+)<>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(w+)<>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(w+)<>Count
name: kafka_$1_$2_$3_total
type: COUNTER
- pattern: kafka.server<>([a-z-]+)
name: kafka_server_quota_$3
type: GAUGE
labels:
resource: "$1"
clientId: "$2"
- pattern: kafka.server<>([a-z-]+)
name: kafka_server_quota_$4
type: GAUGE
labels:
resource: "$1"
user: "$2"
clientId: "$3"
# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(w+)<>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(w+)<>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(w+)<>Value
name: kafka_$1_$2_$3
type: GAUGE
# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
#
# Note that these are missing the '_sum' metric!
- pattern: kafka.(w+)<>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(w+)<>(d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
quantile: "0.$8"
- pattern: kafka.(w+)<>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(w+)<>(d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
quantile: "0.$6"
- pattern: kafka.(w+)<>Count
name: kafka_$1_$2_$3_count
type: COUNTER
- pattern: kafka.(w+)<>(d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
quantile: "0.$4"
Save the file and continue to the next step.
Step 3: Configure Kafka Broker to use JMX exporter
So far, we have everything we need to start extracting Kafka metrics. The only thing left is to link the JMX exporter to our Kafka Broker. Without delay, let us finish it immediately. Open the Kafka Broker server startup script, and then add the JMX configuration at the end of the file, as shown below. All scripts are located in the bin directory in the main Kafka folder.
$ cd /usr/local/kafka-server/bin/
$ sudo vim kafka-server-start.sh
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ $# -lt 1 ];
then
echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
exit 1
fi
base_dir=$(dirname $0)
if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi
EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}
COMMAND=$1
case $COMMAND in
-daemon)
EXTRA_ARGS="-daemon "$EXTRA_ARGS
shift
;;
*)
;;
esac
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "[email protected]"
### ADD THE LINE BELOW ###
export KAFKA_OPTS=' -javaagent:/usr/local/kafka-server/libs/jmx_prometheus_javaagent-0.13.0.jar=7071:/usr/local/kafka-server/config/sample_jmx_exporter.yml'
If you are using systemd, please add this line to the systemd file of kafka [Service] Part as the environment, as follows:
[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
##Add the line below
Environment="KAFKA_OPTS=-javaagent:/usr/local/kafka-server/libs/jmx_prometheus_javaagent-0.13.0.jar=7075:/usr/local/kafka-server/config/sample_jmx_exporter.yml"
ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties
ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh
Restart=on-abnormal
After adding this line at the end of the kafka-server-start.sh script or in the systemd file, restart the Kafka agent.
sudo systemctl restart kafka.service
Check whether the service has been started by checking whether the configured port exists. If you are running a firewall and your Prometheus server is on another server, you should consider allowing access to this port.
$ sudo ss -tunelp | grep 7075
tcp LISTEN 0 3 [::]:7075 [::]:* users:(("java",pid=31609,fd=100)) uid:1000 ino:5391132 sk:ffff977c74f86b40 v6only:0 <->
Allow ports on firewall
----Ubuntu----
$ sudo ufw allow 7075
----CentOS----
$ sudo firewall-cmd --permanent --add-port=7075/tcp
$ sudo firewall-cmd --reload
Open a browser and point it to the IP or FQDN of the server and port. http: //[IP or FQDN]: 7075. You should see data metrics as shown below
good stuff! Our JMX exporter is working as expected. Now let’s start adding the data exposed to Prometheus
Step 4: Add Kafka data to Prometheus
Log in to the Prometheus server and configure this new source as a data target. If you follow this guide to install Prometheus on Debian | Ubuntu or RHEL 8 | CentOS 8, its configuration file is located in /etc/prometheus/prometheus.yml. Please find its configuration file, open and edit, as shown in the figure below
$ sudo vim /etc/prometheus/prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
##### CHANGE THE JOB NAME TO KAFKA AS BELOW#######
- job_name: 'kafka'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
#####CHANGE THE TARGET TO THE IP AND PORT OF JMX SERVICE JUST INSTALLED#######
static_configs:
- targets: ['10.38.83.154:7075']
You can confirm that this target has been successfully added to the Prometheus web interface. Open it with a browser and click Status>Target. If the addition is successful, you will see the following figure.
Its status should be “up”
So far, this is great. Next, we will use the data that Prometheus will store as Grafana’s data source so that we can view the metrics of the style.
Step 5: Add Kafka metrics to Grafana
Now we are at the last and best part. Here, we add Prometheus as a data source, and then use beautiful graphs and charts to visualize it. Log in to your Grafana web interface and perform the following operations. If you have not installed Grafana, please follow the guide below to quickly install it.
How to install Grafana on CentOS 8 / RHEL 8
Install Grafana Debian on Ubuntu
How to install Grafana on CentOS 7
Add data source
After entering the Grafana web interface, click the settings gear icon and select “data sourceThe “” option in the drop-down list.
This will open the “Data Sources” menu, where you can add more content. Click “Add data source“Label
As you might guess, we will choose Prometheus Because that is what we have configured before.
After selecting the Prometheus data source, we will have to tell Grafana where to find the Prometheus server. Enter a cute name next to the URL and the IP and port where Prometheus is running.
You can further add Crawl interval, query timeout with HTTP method. After that, click “Save and test“Button. If everything goes well, a green message will appear. If there is an error, please make sure your Prometheus server is running and accessible. If it is behind a firewall, open its ports.
Import dashboard
After adding the data source, we will continue to add the dashboard, which will visualize the content in the data source. While still on Grafana, click + Button and select import Because we will use the already created dashboard Robust perception. Its id is 721
On the import page, issue the ID 721 then click”load“Button.
The next page will ask you to enter a name, then you should select the data source we added in the drop-down menu at the bottom of the page. When you are done, just click “import“.
And you should display the metrics nicely as the following share.
in conclusion
Now your Kafka metrics can be displayed well on Grafana, so you can get a deeper understanding of your topic and other topics. Thank you for your continued support, and hope this guide will help you. We thank all the creators of the tools used in this guide to improve the lives of developers and administrators. For other guidelines similar to this one, please review the following.
Install and configure Telegraf on RHEL 8 / CentOS 8
Deploy Prometheus on the EKS Kubernetes cluster
How to install Prometheus and node_exporter on Debian 10 (Buster)
Use Prometheus and Grafana to monitor Etcd clusters
Use Prometheus and Grafana to monitor Redis server in 5 minutes
Use Prometheus and Grafana to monitor Linux server performance in 5 minutes
How to use Prometheus and Grafana to monitor Apache Web Server in 5 minutes
Use Prometheus and Grafana to monitor the BIND DNS server
To
You can download this article in PDF format via the link below to support us.
Download the guide in PDF format
turn off
To
To
To