Install and configure Apache Kafka with CMAK on CentOS 8

To
You can download this article in PDF format via the link below to support us.

Download the guide in PDF format

turn off
To

To
To

We live in this world, and most of the things we use and interact with every day are run by computers deployed somewhere in the world. Whether it is your social media, your online payment, your mobile currency transaction or even voice communication, all are done by the computer. Keeping this in mind, there are very important transactions happening on these computers, and interested parties need to visualize them before using them for other attacks. A good example is the visualization and processing of financial transactions to discover more opportunities or solve problems. So, even in the future, how can we capture such data in an efficient and reliable way? Well, technical solutions to this problem have been developed, one of which is Apache Kafka.

For Ubuntu: Install and configure Apache Kafka on Ubuntu

Apache Kafka is an open source distributed event streaming platform used by thousands of companies for high-performance data pipelines, stream analysis, data integration and mission-critical applications. Apache Kafka. Let us explain all these meanings step by step. An event records the fact that “something has happened” in the world or your business and is recorded digitally.

Event streaming is a practice of capturing data in real time from event sources (producers) (such as databases, sensors, mobile devices, cloud services, and software applications) in the form of event streams. Store these event streams persistently for later retrieval; process, process and respond to event streams in real time and retrospectively; and route event streams to different target technologies (consumers) as needed. Resources: Apache Kafka.

A producer is a program/application or entity that sends data to the Kafka cluster. The user sits on the other side and receives data from the Kafka cluster. A Kafka cluster can consist of one or more Kafka agents located on different servers.

“Life is really simple, but we insist on making life complicated.” β€”Confucius

Definition of other terms

  • Subject: The subject is the general name used to store and publish a specific data stream. For example, if you want to store all the data about the page that was clicked, you can specify a name for the topic, such as “page clicked.”
  • Partitions: Each topic is divided into multiple partitions (“buckets”). When creating a topic, you need to specify the number of partitions, but you can increase the number of partitions as needed in the future. Each message is stored in the partition with an incremental ID (called an offset value).
  • Kafka proxy: Each server with Kafka installed is called a proxy. It is a container that contains several topics with partitions.

Apache Kafka use cases

Here are some applications that can take advantage of Apache Kafka:

  • Message interruption: Compared with most messaging systems, Kafka has better throughput, built-in partitioning, replication and fault tolerance, which makes it an ideal solution for large-scale message processing applications
  • Website activity tracking
  • Log aggregation: Kafka extracts the detailed information of files and uses log or event data as a message flow for a clearer abstraction.
  • Stream processing: Capture data from event sources in real time; store these event streams persistently for later retrieval; and route event streams to different target technologies as needed
  • Event source: This is an application design style in which state changes are recorded in chronological order.
  • Commit log: Kafka can be used as an external commit log for distributed systems. This log helps to replicate data between nodes and acts as a resynchronization mechanism for failed nodes to recover their data.
  • Metrics: This involves aggregating statistical information from distributed applications to generate a centralized operational data feed.

Install Kafka on CentOS 8

Apache Kafka requires Java to run.

Step 1: Prepare the server

Therefore, we will start by updating the CentOS server and installing Java. Fortunately, the default CentOS 8 repository includes the latest two major Java 8 and Java 11 LTS versions as follows:

sudo dnf update
sudo dnf install java-11-openjdk-devel wget git unzip -y

Step 2: Get Kafka on CentOS 8

After installing Java correctly, let us now get the source code of Kafka. Go to Download And look for “Latest Version” and get the source under “Binary Download”. Click on the page recommended by Kafka and you will be redirected to the page that contains the link you can use to get it.

cd ~
wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz
sudo mkdir /usr/local/kafka-server && cd /usr/local/kafka-server
sudo tar -xvzf ~/kafka_2.13-2.6.0.tgz --strip 1

Since the -strip 1 flag is set, the archive content will be extracted to /usr/local/kafka-server/.

Step 3: Create Kafka and Zookeeper system unit files

The Systemd unit files of Kafka and Zookeeper will be of great help in performing common service operations such as starting, stopping and restarting Kafka. This makes it suitable for other services to start, stop and restart, which is beneficial and consistent.

Let’s start with the Zookeeper service: the way to manage Kafka’s cluster state and configuration through Zookeeper.

$ sudo vim /etc/systemd/system/zookeeper.service

[Unit]
Description=Apache Zookeeper Server
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka-server/bin/zookeeper-server-start.sh /usr/local/kafka-server/config/zookeeper.properties
ExecStop=/usr/local/kafka-server/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Then proceed to Kafka service

$ sudo vim /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties
ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

After adding the configuration file, reload the systemd daemon to apply the changes, and then start the service program.

sudo systemctl daemon-reload
sudo systemctl enable --now zookeeper
sudo systemctl enable --now kafka

Step 4: Install Cluster Manager for Apache Kafka (CMAK) | Kafka Manager

CMAK (formerly known as Kafka Manager) is an open source tool for managing Apache Kafka clusters developed by Yahoo.

cd ~
git clone https://github.com/yahoo/CMAK.git

Step 5: Configure CMAK

The minimum configuration is the Zookeeper host that will be used for the CMAK (pka kafka manager) state. It can be found in the application.conf file in the conf directory. Change cmak.zkhosts = “my.zookeeper.host.com:2181”, you can also specify multiple Zookeeper hosts separated by commas, for example: cmak.zkhosts = “my.zookeeper.host.com:2181,other.zookeeper .host.com: 2181”

$ vim ~/CMAK/conf/application.conf

cmak.zkhosts="localhost:2181

After adding the Zookeeper host, the following command will create a zip file that can be used to deploy the application. After downloading the file, you should see a lot of output on the terminal. Give it time to finish and compile.

cd ~/CMAK/
./sbt clean dist

After all operations are complete, you should see a message similar to the following:

[info] Your package is ready in /home/tech/CMAK/target/universal/cmak-3.0.0.5.zip

Switch to the directory where the zip file is located and unzip:

$ cd ~/CMAK/target/universal
$ unzip cmak-3.0.0.5.zip
$ cd cmak-3.0.0.5

Step 5: Start the service and access it

After unzipping the resulting zipfile and following the steps below to change the working directory to it Step 4, You can run the service like this:

$ cd ~/CMAK/target/universal/cmak-3.0.0.5
$ bin/cmak

By default, it will select port 9000, so open your favorite browser and point it to http://ip-or-domain-name-of-server:9000. If your firewall is running, please leave the port you want to access empty.

sudo firewall-cmd --permanent --add-port=9000/tcp
sudo firewall-cmd --reload

After everything is normal, you should see an interface as shown below:

You will realize that when we first entered the above interface, there were no clusters available. Therefore, we will continue to create a new cluster. Click “cluster“Drop-down list, and then click “Add cluster“.

Install and configure Apache Kafka with CMAK on CentOS 8

A form will be displayed for you to fill out, as shown below. Give your cluster a name, add Zookeeper hosts, if there are several, separate them with commas. You can fill in other details as needed.

Install and configure Apache Kafka with CMAK on CentOS 8

When everything is satisfied, scroll down and click “save“.

Install and configure Apache Kafka with CMAK on CentOS 8

And you should add the cluster to the CMAK interface/manager.

Step 6: Add a sample theme

Apache Kafka provides multiple shell scripts that can be used. Let’s first create an example topic called “ComputingForGeeksTopic” with a single copy and a single copy. Open a new terminal for CMAK to run and issue the following command:

cd /usr/local/kafka-server
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic ComputingForGeeksTopic

Created topic ComputingForGeeksTopic.

Confirm whether the theme is updated in the CMAK interface: Under the theme, click the number.

Install and configure Apache Kafka with CMAK on CentOS 8

You should be able to see the new theme we added as follows:

Install and configure Apache Kafka with CMAK on CentOS 8

Click it to view its other details, such as partitions, etc.

Install and configure Apache Kafka with CMAK on CentOS 8

Step 7: Create a theme in the CMAK interface

Another easier way to create themes is through the CMAK web interface. Just click “topic“Drop-down list and click”create“.As follows.

Install and configure Apache Kafka with CMAK on CentOS 8

You will need to enter all the details about the new topic (replication factor, partition, etc.). Fill out the form and click “create“Below it.

Install and configure Apache Kafka with CMAK on CentOS 8

in conclusion

Apache Kafka is now installed on CentOS 8 server. It should be noted that Kafka can be installed on multiple servers to create a cluster. Otherwise, thank you for your visit and keep paying attention to the end. Thank you for your continued support.

Learn more about Apache KafkaLearn more about Apache Kafka’s cluster manager

Find other amazing guides below:

SQL from scratch: how to start learning database?

Install Apache Spark on Ubuntu and Debian

To
You can download this article in PDF format via the link below to support us.

Download the guide in PDF format

turn off
To

To
To

Sidebar