Setting up Elasticsearch cluster on CentOS 8/7 | Ubuntu 20.04 / 18.04 with Ansible

The
You can download this article in PDF format to support us through the following link.

Download the guide as a PDF

turn off
The

The
The

Elasticsearch is a powerful open source RESTful distributed real-time search and analysis engine that provides full-text search capabilities. Elasticsearch is built on Apache Lucene, which is freely available under the Apache 2 license. In this article, we will use the Ansible automation tool to install Elasticsearch Cluster on CentOS 8/7 and Ubuntu 20.04 / 18.04.

This tutorial will help Linux users install and configure a highly available multi-node Elasticsearch cluster on CentOS 8 / CentOS 7 and Ubuntu 20.04 / 18.04 Linux systems. Some key uses of ElasticSearch are log analysis, search engine, full text search, business analysis, security intelligence, etc.

In this setup, we will install Elasticsearch 7.x Cluster with Ansible role. The character we use is the official ElasticSearch project, allowing you to choose freely.

Elasticsearch node type

There are two common types of Elasticsearch nodes:

  • Master node: Responsible for cluster-wide operations, such as index management and allocating data shard storage to data nodes.
  • Data node: They store the actual shards of index data and handle all CRUD, search and aggregation operations. They consume more CPU, memorywith input Output

Setting requirements

Before you start, you need at least three Installed and updated the CentOS 8/7 server. Only users with sudo privileges or root users can perform operations. My settings are based on the following node structure.

server nameglassesServer role
Elk Master 0116GB RAM, 8vpcusthe Lord
Elk Master 0216GB RAM, 8vpcusthe Lord
Elk Master 0316GB RAM, 8vpcusthe Lord
Elk Data 0132GB RAM, 16vpcusdata
Elk data0232GB RAM, 16vpcusdata
Elk Data 0332GB RAM, 16vpcusdata

note:

  • For small environments, nodes can be used for data and main operations.

Storage considerations

For data nodes, it is recommended that storage be properly configured considering scalability. In my laboratory, each data node has a 500GB The disk is installed under / data. This is configured by the following command.

caveat: Do not copy and run commands, they are just reference points.

sudo parted -s -a optimal -- /dev/sdb mklabel gpt
sudo parted -s -a optimal -- /dev/sdb mkpart primary 0% 100%
sudo parted -s -- /dev/sdb align-check optimal 1
sudo pvcreate /dev/sdb1
sudo vgcreate vg0 /dev/sdb1
sudo lvcreate -n lv01 -l+100%FREE vg0
sudo mkfs.xfs /dev/mapper/vg0-lv01
echo "/dev/mapper/vg0-lv01 /data xfs defaults 0 0" | sudo tee -a /etc/fstab
sudo mount -a

Step 1: Install Ansible on the workstation

We will use Ansible to set up Elasticsearch Cluster on CentOS 8/7. Make sure Ansible is installed on your computer for easy management.

On Fedora:

sudo dnf install ansible

On CentOS:

sudo yum -y install epel-release
sudo yum install ansible

RHEL 7 / RHEL 8:

--- RHEL 8 ---
sudo subscription-manager repos --enable ansible-2.9-for-rhel-8-x86_64-rpms
sudo yum install ansible

--- RHEL 7 ---
sudo subscription-manager repos --enable rhel-7-server-ansible-2.9-rpms
sudo yum install ansible

Ubuntu:

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

For other distributions, please see Official Ansible installation guide.

Confirm that Ansible is installed on your computer by checking the version.

$ ansible --version
ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/home/jkmutai/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.6 (default, Jan 30 2020, 09:44:41) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]

Step 2: Import Elasticsearch ansible characters

After installing Ansible, you can now use the Galaxy to import Elasticsearch ansible characters into your local system.

$ ansible-galaxy install elastic.elasticsearch,7.6.2

- downloading role 'elasticsearch', owned by elastic
- downloading role from https://github.com/elastic/ansible-elasticsearch/archive/7.6.2.tar.gz
- extracting elastic.elasticsearch to /var/home/jkmutai/.ansible/roles/elastic.elasticsearch
- elastic.elasticsearch (7.6.2) was installed successfully

Where 7.6.2 Is the release version of the Elasticsearch character to download. You can check Publish page Match the Elasticsearch version to be installed.

The role will be added to ~ / .Ansible / role table of Contents.

$ ls ~/.ansible/roles
total 4.0K
drwx------. 15 jkmutai jkmutai 4.0K May  1 16:28 elastic.elasticsearch

Use Elasticsearch cluster host to configure ssh.

$ vim ~/.ssh/config

This is what my other configuration looks like-updated to suit your environment.

# Elasticsearch master nodes
Host elk-master01
  Hostname 95.216.167.173
  User root
Host elk-master02
  Hostname 95.216.195.161
  User root
Host elk-master03
  Hostname 95.217.159.46
  User root

# Elasticsearch worker nodes
Host elk-data01
  Hostname 95.216.195.178
  User root
Host elk-data02
  Hostname 95.217.159.52
  User root
Host elk-data03
  Hostname 95.217.159.43
  User root

Make sure you have copied the ssh key to all computers.

--- Master nodes ---
for host in elk-master0{1..3}; do ssh-copy-id $host; done

--- Worker nodes ---
for host in elk-data0{1..3}; do ssh-copy-id $host; done

Confirm that you can use ssh without password verification.

$ ssh elk-master01
Warning: Permanently added '95.216.167.173' (ECDSA) to the list of known hosts.
[[email protected] ~]# 

If your private key ssh key has a passphrase, save it to avoid being prompted on each computer.

$ eval `ssh-agent -s` && ssh-add
Enter passphrase for /var/home/jkmutai/.ssh/id_rsa: 
Identity added: /var/home/jkmutai/.ssh/id_rsa (/var/home/jkmutai/.ssh/id_rsa)

Step 3: Create Elasticsearch Playbook and run it

Now that all the prerequisites are configured, let’s create a Playbook file for deployment.

$ vim elk.yml

My content is as follows.

- hosts: elk-master-nodes
  roles:
    - role: elastic.elasticsearch
  vars:
    es_enable_xpack: false
    es_data_dirs:
      - "/data/elasticsearch/data"
    es_log_dir: "/data/elasticsearch/logs"
    es_java_install: true
    es_heap_size: "1g"
    es_config:
      cluster.name: "elk-cluster"
      cluster.initial_master_nodes: "95.216.167.173:9300,95.216.195.161:9300,95.217.159.46:9300"
      discovery.seed_hosts: "95.216.167.173:9300,95.216.195.161:9300,95.217.159.46:9300"
      http.port: 9200
      node.data: false
      node.master: true
      bootstrap.memory_lock: false
      network.host: '0.0.0.0'
    es_plugins:
     - plugin: ingest-attachment

- hosts: elk-data-nodes
  roles:
    - role: elastic.elasticsearch
  vars:
    es_enable_xpack: false
    es_data_dirs:
      - "/data/elasticsearch/data"
    es_log_dir: "/data/elasticsearch/logs"
    es_java_install: true
    es_config:
      cluster.name: "elk-cluster"
      cluster.initial_master_nodes: "95.216.167.173:9300,95.216.195.161:9300,95.217.159.46:9300"
      discovery.seed_hosts: "95.216.167.173:9300,95.216.195.161:9300,95.217.159.46:9300"
      http.port: 9200
      node.data: true
      node.master: false
      bootstrap.memory_lock: false
      network.host: '0.0.0.0'
    es_plugins:
      - plugin: ingest-attachment

important:

  • The node.master of the master node is set to real And set node.data to false.
  • The node.data of the data node is set to real And node.master is set to false.
  • The es_enable_xpack variable is set to false Used to install the open source version of ElasticSearch.
  • cluster.initial_master_nodes and discovery.seed_hosts point to the master node
  • / data / elasticsearch / data is where Elasticsearch data shards are stored – due to performance and scalability, it is recommended to use it separately from the OS installation.
  • / data / elasticsearch / logs is where Elasticsearch logs are stored.
  • The directory will be created automatically by the ansible task. You just need to make sure / Data It is the installation point for the data storage required by Elasticsearch.

For more customization options, please check The project’s github documentation.

Create inventory file

Create a new manifest file.

$ vim hosts
[elk-master-nodes]
elk-master01
elk-master02
elk-master03

[elk-data-nodes]
elk-data01
elk-data02
elk-data03

After setting all, run the Playbook.

$ ansible-playbook -i hosts elk.yml

It should start executing. Please be patient as this may take a few minutes.

PLAY [elk-master-nodes] ********************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************
ok: [elk-master02]
ok: [elk-master01]
ok: [elk-master03]

TASK [elastic.elasticsearch : set_fact] ****************************************************************************************************************
ok: [elk-master02]
ok: [elk-master01]
ok: [elk-master03]

TASK [elastic.elasticsearch : os-specific vars] ********************************************************************************************************
ok: [elk-master01]
ok: [elk-master02]
ok: [elk-master03]
.......

Successful ansible execution will have output similar to the following.

PLAY RECAP *********************************************************************************************************************************************
elk-data01                 : ok=38   changed=10   unreachable=0    failed=0    skipped=119  rescued=0    ignored=0   
elk-data02                 : ok=38   changed=10   unreachable=0    failed=0    skipped=118  rescued=0    ignored=0   
elk-data03                 : ok=38   changed=10   unreachable=0    failed=0    skipped=118  rescued=0    ignored=0   
elk-master01               : ok=38   changed=10   unreachable=0    failed=0    skipped=119  rescued=0    ignored=0   
elk-master02               : ok=38   changed=10   unreachable=0    failed=0    skipped=118  rescued=0    ignored=0   
elk-master03               : ok=38   changed=10   unreachable=0    failed=0    skipped=118  rescued=0    ignored=0   

See the screenshot below.

Step 4: Confirm to install Elasticsearch Cluster on Ubuntu / CentOS

Log in to one of the master nodes.

$ elk-master01

Check the cluster health.

$ curl http://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "elk-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Check the master node.

$ curl -XGET 'http://localhost:9200/_cat/master'
G9X__pPXScqACWO6YzGx3Q 95.216.167.173 95.216.167.173 elk-master01

View the data node:

$ curl -XGET 'http://localhost:9200/_cat/nodes'
95.217.159.43   7 47 1 0.02 0.03 0.02 di - elk-data03
95.216.167.173 10 34 1 0.00 0.02 0.02 im * elk-master01
95.217.159.46  13 33 1 0.00 0.01 0.02 im - elk-master03
95.216.195.161 14 33 1 0.00 0.01 0.02 im - elk-master02
95.217.159.52   7 47 1 0.00 0.03 0.03 di - elk-data02
95.216.195.178  6 47 1 0.00 0.02 0.02 di - elk-data01

Confirm that you now have a Clean Elasticsearch Cluster on CentOS 8/7 and Ubuntu 20.04 / 18.04 Linux systems.

Similar guidelines:

Install Graylog 3 using Elasticsearch 6.x on CentOS 8 / RHEL 8 Linux

How to delete Elasticsearch Index data using curl

The
You can download this article in PDF format to support us through the following link.

Download the guide as a PDF

turn off
The

The
The

Sidebar