You can download this article in PDF format via the link below to support us.Download the guide in PDF formatClose
Elasticsearch is a log analysis engine that enables users to store, index, analyze, and visualize logs and metrics on the dashboard. Elastic Search uses Kibana to visualize data on the dashboard. In this guide, we will introduce how to install ElasticSearch, Kibana and how to use Beats to send logs to an Elastic search instance.
Beats is a shipper used to send logs from different endpoints to Elastic Search. They are installed on the client as a proxy, so they can send logs to the Elastic search instance.
Beats are divided into the following types:
- File shot -Analyze log files
- Package -Analyze network packets
- Winlogbeat -Used to analyze Windows events
- Metric system beat -Used to send your cloud environment indicators
- Audit signal -Used to transport information about system audit data
- Heartbeat -Used to monitor infrastructure availability
Install ElasticSearch on Ubuntu/Debian
In this guide, we will follow the steps below to install ElasticSearch on Ubuntu / Debian :
Update system
sudo apt update && sudo apt upgrade -y
Install Open-JDK 11 (recommended)
sudo apt install default-jdk -y
Import elastic search GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch --no-check-certificate | sudo apt-key add -
Add ElasticSearch repository
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
Install ElasticSearch
sudo apt update
sudo apt install elasticsearch
Configure Elasticsearch by changing Elasticsearch to allow remote connections Web host IP to 0.0.0.0 /etc/elasticsearch/elasticsearch.yml file
$ sudo nano /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
Start and enable Elasticsearch
sudo /etc/init.d/elasticsearch start
sudo systemctl enable --now elasticsearch
Check if Elasticsearch is up and running:
$ curl https://127.0.0.1:9200
Sample output:
$ curl https://127.0.0.1:9200
{
"name" : "ubuntu",
"cluster_name" : "computingforgeeks",
"cluster_uuid" : "EVzpAqUUSV6wQhO7yiPeKw",
"version" : {
"number" : "7.10.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
"build_date" : "2020-12-05T01:00:33.671820Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
Install Kibana on Ubuntu/Debian
Kibana provides a web interface where we can visually analyze the collected data.
Use the following steps to install Kibana on the same host:
sudo apt install kibana
Configure Kibana to allow external IP connections.edit /etc/kibana/kibana.yaml File and change Server host Your external IP or 0.0.0.0 option.
$ sudo nano /etc/kibana/kibana.yaml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
...
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"
Since we installed Kibana and Elasticsearch on the same host, we don’t need to change elasticsearch.hosts field.
Start and enable Kibana
sudo systemctl enable --now kibana
Now you can access the Kibana dashboard using port 5601 of the server IP (ie https://server-IP:5601).
You may need to allow ports through the firewall:
sudo ufw allow 5601/tcp
Install Metricbeat
Once Elasticsearch and Kibana are configured, you will need to set up Beats on the client server.
In this article, we will introduce how to install Filebeat and Metricbeats on the client server.
Install Metricbeat
You can download Metricbeat from the APT and YUM repositories:
Easy
- Set up GPG key for Elasticsearch
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
2. Install Metricbeat repository
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
3. Install Metricbeat
sudo apt-get update && sudo apt-get install metricbeat
Yum
- Download GPG key
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2. Create a repo file in the following location /etc/yum.repos.d/ Has the following content:
sudo tee /etc/yum.repos.d/elastic.repo<<EOF
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
3. Install Metricbeat
sudo yum -y install metricbeat
Use Metricbeat to send system metrics to Elasticsearch
- Enable system modules
sudo metricbeat modules enable system
2. Link Metricbeat to the remote Elastic search server.edit /etc/metricbeat/metricbeat.yml File and edit Host details Kibana with Elasticsearch output
$ sudo vim /etc/metricbeat/metricbeat.yml
Add the IP of the instance running Elasticsearch and kibana in the host option. In our example, Elasticsearch is running on the 172.16.56.5 host:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: https://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "172.16.56.5:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
Do the same for Elasticsearch
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["172.16.56.5:9200"]
3. Set up the initial environment-this will load the Kibana dashboard, if you have already set it, you can omit -e flag.
sudo metricbeat setup -e
You should see an attempt to connect to the Elasticsearch host and Kibana dashboard creation attempt.
$ sudo metricbeat setup -e
......
2020-12-19T09:56:50.585Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'metricbeat-7.10.1' as ILM is enabled.
2020-12-19T09:56:50.585Z INFO eslegclient/connection.go:99 elasticsearch url: https://172.16.56.5:9200
2020-12-19T09:56:50.586Z INFO [publisher] pipeline/module.go:113 Beat name: master
2020-12-19T09:56:50.612Z INFO add_kubernetes_metadata/kubernetes.go:71 add_kubernetes_metadata: kubernetes env detected, with version: v1.18.9+k3s1
2020-12-19T09:56:50.620Z INFO eslegclient/connection.go:99 elasticsearch url: https://172.16.56.5:9200
2020-12-19T09:56:50.622Z INFO [kubernetes] kubernetes/util.go:138 kubernetes: Using node master discovered by machine-id matching {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-19T09:56:50.625Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.1
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
2020-12-19T09:56:50.681Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2020-12-19T09:56:50.683Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2020-12-19T09:56:50.683Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2020-12-19T09:56:50.683Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metricbeat-7.10.1 {now/d}-000001}' as ILM is enabled.
2020-12-19T09:56:50.683Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metricbeat-7.10.1-*' as ILM is enabled.
2020-12-19T09:56:50.683Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.10.1 {now/d}-000001} as ILM is enabled.
2020-12-19T09:56:50.683Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-12-19T09:56:50.686Z INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2020-12-19T09:56:51.231Z INFO template/load.go:117 Try loading template metricbeat-7.10.1 to Elasticsearch
2020-12-19T09:56:52.677Z INFO template/load.go:109 template with name 'metricbeat-7.10.1' loaded.
2020-12-19T09:56:52.677Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2020-12-19T09:56:52.681Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-19T09:56:52.681Z INFO kibana/client.go:119 Kibana url: https://172.16.56.5:5601
2020-12-19T09:56:53.517Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-12-19T09:56:53.518Z INFO kibana/client.go:119 Kibana url: https://172.16.56.5:5601
2020-12-19T09:58:43.294Z INFO instance/beat.go:815 Kibana dashboards successfully loaded.
Loaded dashboards
3. Start and enable Metricbeat
sudo service metricbeat start
sudo systemctl enable metricbeat
You can now visualize by navigating to the data on the Kibana dashboard Dashboard .
Set up Filebeat
You can use APT and YUM repositories for Filebeat settings.
Easy
#Download GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
#Install apt-transport-https
sudo apt-get install apt-transport-https
#Add repository
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
#Install Filebeat
sudo apt-get update && sudo apt-get install filebeat
Yum
##Download GPG key
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
##create repo file
sudo tee /etc/yum.repos.d/elastic.repo<<EOF
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
sudo tee /etc/yum.repos.d/elasticsearch.repo<<EOF
##Install filebeat
sudo yum install filebeat
Connect Filebeat to Elastic Stack
edit /etc/filebeat/filebeat.ym l File and add remote host and port for Elasticsearch. You can also add the username and password of authorized users.
output.elasticsearch:
hosts: ["elasticsearch-IP:9200"]
username: "filebeat_internal"
password: "YOUR_PASSWORD"
Also set the Kibana details on the same file to connect to the host where Kibana is installed:
setup.kibana:
host: "mykibanahost:5601"
replace Elastic search with mykibanahost The IP address of the server.
Enable Filebeat module
List and identify the modules to be enabled:
filebeat modules list
Enable selected module
filebeat modules enable <module-name>
Set up the filebeat environment
filebeat setup -e
Start Filebeat service
systemctl start filebeat
You should see a confirmation that the dashboard was successfully created
2020-12-19T11:11:55.731Z INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2020-12-19T11:11:58.580Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-12-19T11:11:59.711Z INFO template/load.go:117 Try loading template filebeat-7.10.1 to Elasticsearch
2020-12-19T11:12:00.075Z INFO template/load.go:109 template with name 'filebeat-7.10.1' loaded.
2020-12-19T11:12:00.075Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2020-12-19T11:12:00.077Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-19T11:12:00.078Z INFO kibana/client.go:119 Kibana url: https://172.16.56.5:5601
2020-12-19T11:12:03.995Z INFO kibana/client.go:119 Kibana url: https://172.16.56.5:5601
2020-12-19T11:13:13.600Z INFO instance/beat.go:815 Kibana dashboards successfully loaded.
Loaded dashboards
Navigate to the kibana dashboard to visualize your data.
in conclusion
We have installed Elastic Stack and configured Beats to get metrics and logs. You can use the same process as above to configure other Beats.
Check out other articles to learn about interesting monitoring tools:
How to install Netdata on Kubernetes using Helm
Automate Icinga2 configuration with Icinga Director on CentOS | RHEL 8
You can download this article in PDF format via the link below to support us.Download the guide in PDF formatClose
