Use Beats to forward server logs and metrics to Elasticsearch

You can download this article in PDF format via the link below to support us.Download the guide in PDF formatClose

Elasticsearch is a log analysis engine that enables users to store, index, analyze, and visualize logs and metrics on the dashboard. Elastic Search uses Kibana to visualize data on the dashboard. In this guide, we will introduce how to install ElasticSearch, Kibana and how to use Beats to send logs to an Elastic search instance.

Beats is a shipper used to send logs from different endpoints to Elastic Search. They are installed on the client as a proxy, so they can send logs to the Elastic search instance.

Beats are divided into the following types:

File shot -Analyze log files
Package -Analyze network packets
Winlogbeat -Used to analyze Windows events
Metric system beat -Used to send your cloud environment indicators
Audit signal -Used to transport information about system audit data
Heartbeat -Used to monitor infrastructure availability

Install ElasticSearch on Ubuntu/Debian

In this guide, we will follow the steps below to install ElasticSearch on Ubuntu/Debian:

Update system

sudo apt update && sudo apt upgrade -y

Install Open-JDK 11 (recommended)

sudo apt install default-jdk -y

Import elastic search GPG key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch --no-check-certificate | sudo apt-key add -

Add ElasticSearch repository

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Install ElasticSearch

sudo apt update
sudo apt install elasticsearch

Configure Elasticsearch by changing Elasticsearch to allow remote connections Web host IP to 0.0.0.0 /etc/elasticsearch/elasticsearch.yml file

$ sudo nano /etc/elasticsearch/elasticsearch.yml

# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#

Start and enable Elasticsearch

sudo /etc/init.d/elasticsearch start
sudo systemctl enable --now elasticsearch

Check if Elasticsearch is up and running:

$ curl http://127.0.0.1:9200

Sample output:

$ curl http://127.0.0.1:9200
{
  "name" : "ubuntu",
  "cluster_name" : "computingforgeeks",
  "cluster_uuid" : "EVzpAqUUSV6wQhO7yiPeKw",
  "version" : {
    "number" : "7.10.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
    "build_date" : "2020-12-05T01:00:33.671820Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

Install Kibana on Ubuntu/Debian

Kibana provides a web interface where we can visually analyze the collected data.

Use the following steps to install Kibana on the same host:

sudo apt install kibana

Configure Kibana to allow external IP connections.edit /etc/kibana/kibana.yaml File and change Server host Your external IP or 0.0.0.0 option.

$ sudo nano /etc/kibana/kibana.yaml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

...
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

Since we installed Kibana and Elasticsearch on the same host, we don’t need to change elasticsearch.hosts field.

Start and enable Kibana

sudo systemctl enable --now kibana

Now you can access the Kibana dashboard using port 5601 of the server IP (ie http://server-IP:5601).

You may need to allow ports through the firewall:

sudo ufw allow 5601/tcp

Install Metricbeat

Once Elasticsearch and Kibana are configured, you will need to set up Beats on the client server.

In this article, we will introduce how to install Filebeat and Metricbeats on the client server.

Install Metricbeat

You can download Metricbeat from the APT and YUM repositories:

Easy

Set up GPG key for Elasticsearch

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

2. Install Metricbeat repository

sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

3. Install Metricbeat

sudo apt-get update && sudo apt-get install metricbeat

Yum

Download GPG key

sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

2. Create a repo file in the following location /etc/yum.repos.d/ Has the following content:

sudo tee /etc/yum.repos.d/elastic.repo<<EOF
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

3. Install Metricbeat

sudo yum -y install metricbeat

Use Metricbeat to send system metrics to Elasticsearch

Enable system modules

sudo metricbeat modules enable system

2. Link Metricbeat to the remote Elastic search server.edit /etc/metricbeat/metricbeat.yml File and edit Host details Kibana with Elasticsearch output

$ sudo vim /etc/metricbeat/metricbeat.yml

Add the IP of the instance running Elasticsearch and kibana in the host option. In our example, Elasticsearch is running on the 172.16.56.5 host:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "172.16.56.5:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

Do the same for Elasticsearch

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["172.16.56.5:9200"]

3. Set up the initial environment-this will load the Kibana dashboard, if you have already set it, you can omit -e flag.

sudo metricbeat setup -e

You should see an attempt to connect to the Elasticsearch host and Kibana dashboard creation attempt.

$ sudo metricbeat setup -e
......

2020-12-19T09:56:50.585Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'metricbeat-7.10.1' as ILM is enabled.
2020-12-19T09:56:50.585Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://172.16.56.5:9200
2020-12-19T09:56:50.586Z	INFO	[publisher]	pipeline/module.go:113	Beat name: master
2020-12-19T09:56:50.612Z	INFO	add_kubernetes_metadata/kubernetes.go:71	add_kubernetes_metadata: kubernetes env detected, with version: v1.18.9+k3s1
2020-12-19T09:56:50.620Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://172.16.56.5:9200
2020-12-19T09:56:50.622Z	INFO	[kubernetes]	kubernetes/util.go:138	kubernetes: Using node master discovered by machine-id matching	{"libbeat.processor": "add_kubernetes_metadata"}
2020-12-19T09:56:50.625Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.1
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2020-12-19T09:56:50.681Z	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2020-12-19T09:56:50.683Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:274	ILM policy successfully loaded.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:407	Set setup.template.name to '{metricbeat-7.10.1 {now/d}-000001}' as ILM is enabled.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:412	Set setup.template.pattern to 'metricbeat-7.10.1-*' as ILM is enabled.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:446	Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.10.1 {now/d}-000001} as ILM is enabled.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:450	Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-12-19T09:56:50.686Z	INFO	template/load.go:183	Existing template will be overwritten, as overwrite is enabled.
2020-12-19T09:56:51.231Z	INFO	template/load.go:117	Try loading template metricbeat-7.10.1 to Elasticsearch
2020-12-19T09:56:52.677Z	INFO	template/load.go:109	template with name 'metricbeat-7.10.1' loaded.
2020-12-19T09:56:52.677Z	INFO	[index-management]	idxmgmt/std.go:298	Loaded index template.
2020-12-19T09:56:52.681Z	INFO	[index-management]	idxmgmt/std.go:309	Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-19T09:56:52.681Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T09:56:53.517Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-12-19T09:56:53.518Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T09:58:43.294Z	INFO	instance/beat.go:815	Kibana dashboards successfully loaded.
Loaded dashboards

3. Start and enable Metricbeat

sudo service metricbeat start
sudo systemctl enable metricbeat

You can now visualize by navigating to the data on the Kibana dashboard Dashboard.

Set up Filebeat

You can use APT and YUM repositories for Filebeat settings.

Easy

#Download GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

#Install apt-transport-https
sudo apt-get install apt-transport-https

#Add repository
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

#Install Filebeat
sudo apt-get update && sudo apt-get install filebeat

Yum

##Download GPG key
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

##create repo file
sudo tee /etc/yum.repos.d/elastic.repo<<EOF
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
sudo tee /etc/yum.repos.d/elasticsearch.repo<<EOF


##Install filebeat
sudo yum install filebeat

Connect Filebeat to Elastic Stack

edit /etc/filebeat/filebeat.yml File and add remote host and port for Elasticsearch. You can also add the username and password of authorized users.

output.elasticsearch:
  hosts: ["elasticsearch-IP:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD"

Also set the Kibana details on the same file to connect to the host where Kibana is installed:

setup.kibana:
    host: "mykibanahost:5601"

replace Elastic search with mykibanahost The IP address of the server.

Enable Filebeat module

List and identify the modules to be enabled:

filebeat modules list

Enable selected module

filebeat modules enable <module-name>

Set up the filebeat environment

filebeat setup -e

Start Filebeat service

systemctl start filebeat

You should see a confirmation that the dashboard was successfully created

2020-12-19T11:11:55.731Z	INFO	template/load.go:183	Existing template will be overwritten, as overwrite is enabled.
2020-12-19T11:11:58.580Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-12-19T11:11:59.711Z	INFO	template/load.go:117	Try loading template filebeat-7.10.1 to Elasticsearch
2020-12-19T11:12:00.075Z	INFO	template/load.go:109	template with name 'filebeat-7.10.1' loaded.
2020-12-19T11:12:00.075Z	INFO	[index-management]	idxmgmt/std.go:298	Loaded index template.
2020-12-19T11:12:00.077Z	INFO	[index-management]	idxmgmt/std.go:309	Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-19T11:12:00.078Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T11:12:03.995Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T11:13:13.600Z	INFO	instance/beat.go:815	Kibana dashboards successfully loaded.
Loaded dashboards

Navigate to the kibana dashboard to visualize your data.