Use Fluentbit to forward Kubernetes logs to Elasticsearch (ELK)

You can download this article in PDF format via the link below to support us.
Download the guide in PDF formatshut down

Fluent Bit is an open source, lightweight log processing and forwarding service. The fluent bit allows logs, events or metrics to be collected from different sources and processed. This data can then be passed to different backends, such as Elastic Search, Splunk, Kafka, Data Dog, InfluxDB or New Relic.

The fluent bit is easy to set up and configure. It gives you complete control over the data to be collected and parses the data to provide the structure of the collected data. It allows to delete unnecessary data, filter the data and push to the output destination. Therefore, it provides an end-to-end solution for data collection.

Some of the wonderful features of a fluent drill bit are:

  1. high performance
  2. It is super light, fast and requires less resources and memory
  3. It supports multiple data formats.
  4. The configuration file of Fluent Bit is very easy to understand and modify.
  5. Fluent Bit has built-in TLS/SSL support. The communication with the output target is secure.
  6. Asynchronous I/O

Fluent Bit is compatible with docker and kubernetes, so it can be used to aggregate application logs. There are multiple ways to log in to Kubernetes.One way is by default Standard output Write the log of the host path “/var/log/containers” on the cluster nodes.This method requires a bit of fluency Daemon set Deploy. The daemon set deploys a smooth bit container on each node in the cluster.

The second way to record is to use Sustained amount. This allows logs to be written and stored persistently in internal or external storage (such as Cephfs).The fluent bit can be set to deploy Read the log from the persistent volume.

In this blog, we will study how to use fluent bits to send logs from Kubernetes persistent volumes to Elastic search. After sending the logs to elastic search, we can use kibana to visualize and create dashboards using application logs and metrics.

Prerequisites:

First, we need to have a running Kubernetes cluster. If you don’t have a guide yet, you can set it up with the following guide:

  • Use kubeadm to install a Kubernetes cluster on Ubuntu
  • Use kubeadm to install a Kubernetes cluster on CentOS 7
  • Use Rancher RKE to install a production Kubernetes cluster

Second, we will need an elastic search cluster setup. If you have not installed elastic search, you can use the elastic search installation guide. In this tutorial, we will use the stateful set deployed in the kubernetes environment to set up a sample elastic search environment. We will also need a kibana instance to help us visualize this log.

Deploy Elasticsearch

  1. Create a manifest file.This deployment assumes that we have a storage class Cephalosporin In our cluster. A persistent volume will be created next to the elastic search stateful set. Modify this configuration as needed.
$ vim elasticsearch-ss.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster 
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: cephfs
      resources:
        requests:
          storage: 5Gi

Apply this configuration

$ kubectl apply -f elasticsearch-ss.yaml

2. Create an elastic search service

$ vim elasticsearch-svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch 
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node
$ kubectl apply -f elasticsearch.svc

3. Deploy Kibana

$ vim kibana.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.2.0
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch:9200
        ports:
        - containerPort: 5601

---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  labels:
    app: kibana
spec:
  ports:
  - port: 5601
  selector:
    app: kibana

Apply this configuration:

$ kubectl apply -f kibana.yaml

4. Then, we need to configure and import routes for the kibana service, as shown below:

$ vim kibana-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/tls-acme: "true"
    ingress.kubernetes.io/force-ssl-redirect: "true"
  name: kibana

spec:
  rules:
  - host: kibana.computingforgeeks.com
    http:
      paths:
      - backend:
          serviceName: kibana
          servicePort: 5601
        path: /
  tls:
  - hosts:
    - kibana.computingforgeeks.com
    secretName: ingress-secret   // This can be created prior if using custom certs
$ kubectl apply -f kibana-ingress.yaml 

Kibana services can now be accessed via https://kibana.computingforgeeks.com/

Kibana page

After the setup is complete, we can continue to deploy fluent Bit.

Step 1: Deploy service accounts, roles and role binding

Create a deployment file with the following content:

$ vim  fluent-bit-role.yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: default

Apply the deployment configuration by running the following command.

kubectl apply -f fluent-bit-role.yaml

Step 2: Deploy Fluent Bit configMap

This configuration diagram allows us to configure our fluent bit service accordingly. Here, we define the log parsing and routing of Fluent Bit. Change this configuration to meet your needs.

$ vim fluentbit-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    k8s-app: fluent-bit
  name: fluent-bit-config

data:
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               *
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               *
        Path              /var/log/*.log
        Parser            json
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False
  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^[[^ ]* (?<time>[^]]*)] [(?<level>[^]]*)](?: [pid (?<pid>[^]]*)])?( [client (?<client>[^]]*)])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^<(?<pri>[0-9]+)>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_/.-]*)(?:[(?<pid>[0-9]+)])?(?:[^:]*:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
kubectl apply -f fluentbit-configmap.yaml

Step 3: Create persistent bulk claims

This is where we write application logs.

$ vim pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: cephfs #Change accordingly
  resources:
    requests:
      storage: 5Gi
$ kubectl apply -f pvc.yaml

Step 4: Use the configuration mapping in the file to deploy kubernetes deployment

$ vim fluentbit-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: fluent-bit-logging
  name: fluent-bit
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      annotations:
        prometheus.io/path: /api/v1/metrics/prometheus
        prometheus.io/port: "2020"
        prometheus.io/scrape: "true"
      labels:
        k8s-app: fluent-bit-logging
        kubernetes.io/cluster-service: "true"
        version: v1
    spec:
      containers:
      - env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: elasticsearch
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        image: fluent/fluent-bit:1.5
        imagePullPolicy: Always
        name: fluent-bit
        ports:
        - containerPort: 2020
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /fluent-bit/etc/
          name: fluent-bit-config
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: fluent-bit
      serviceAccountName: fluent-bit
      volumes:
      - name: varlog
        persistentVolumeClaim:
          claimName: logs-pvc
      - configMap:
          defaultMode: 420
          name: fluent-bit-config
        name: fluent-bit-config

Create the object by running the following command:

$ kubectl apply -f  fluentbit-deployment.yaml 

Step 5: Deploy the application

Let’s test whether our fluent bit service works as expected. We will use a test application to write logs to our persistent volume.

$ vim testpod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /var/log/app.log; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /var/log
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: logs-pvc

Apply the following commands:

$ kubectl apply -f testpod.yaml

Check whether the Pod is running.

$ kubectl get pods

You should see the following output:

NAME      READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          107s

After the pod is running, we can continue to check whether the logs are sent to Elastic search.

On Kibana, we will have to create an index as shown below. Click”Administration>Index Mode>Create Index Mode

index

kibana_timestamp

After creating the index. Click the discovery icon to see if our log is in place:

kibana_logs

Check out more guides on Kubernetes:

Install Ambassador API Gateway/Ingress Controller on Kubernetes

Use Kubespray to deploy a highly available Kubernetes cluster on CentOS 7

Use Weave Scope to monitor Docker containers and Kubernetes

Transport-secure access to Linux systems and Kubernetes

You can download this article in PDF format via the link below to support us.
Download the guide in PDF formatshut down

Sidebar