Kubernetes cluster deployment on CentOS 7 / CentOS 8 using Ansible and Calico CNI
You want to set up a three-node Kubernetes cluster for your development project on CentOS 7 / CentOS 8 – 1 master with two Or more worker nodes? This guide will guide you through the steps to set up a Kubernetes cluster on CentOS 8 / CentOS 7 Linux machines running Ansible and Calico CNI with a firewall configured. Kubernetes (K8s) is an open source system for automating the deployment, expansion, and management of containerized applications.
Similar Kubernetes deployment guides:
- How to use K3 to deploy a lightweight Kubernetes cluster in 5 minutes
- Deploy a Kubernetes cluster ready with Ansible and Kubespray
My lab is based on the following environments:
machine Types of | CPU name | IP address |
Control node | k8smaster01.computingforgeeks.com | 192.168.122.10 |
Working node 1 | k8snode01.computingforgeeks.com | 192.168.122.11 |
Working node 2 | k8snode01.computingforgeeks.com | 192.168.122.12 |
First, make sure your system is updated and all dependencies are installed, including container runtime, Kubernetes packages, and configuring a firewall for k8.
Step 1: Set standard requirements
I wrote an Ansible role to prepare standard Kubernetes nodes. The role contains the following tasks:
- Install required basic packages
- Set standard system requirements-disable swap, modify sysctl, disable SELinux
- Install and configure the container runtime of your choice-cri-o, Docker, Containerd
- Install Kubernetes packages-kubelet, kubeadm and kubectl
- Configure Firewalld on Kubernetes master and worker nodes
Visit my Github page to set it up:
https://github.com/jmutai/k8s-pre-bootstrap
This is the output I executed recently:
TASK [kubernetes-bootstrap : Open flannel ports on the firewall] ***************************************************************************************
skipping: [k8smaster01] => (item=8285)
skipping: [k8smaster01] => (item=8472)
skipping: [k8snode01] => (item=8285)
skipping: [k8snode01] => (item=8472)
skipping: [k8snode02] => (item=8285)
skipping: [k8snode02] => (item=8472)
TASK [kubernetes-bootstrap : Open calico UDP ports on the firewall] ************************************************************************************
ok: [k8snode01] => (item=4789)
ok: [k8smaster01] => (item=4789)
ok: [k8snode02] => (item=4789)
TASK [kubernetes-bootstrap : Open calico TCP ports on the firewall] ************************************************************************************
ok: [k8snode02] => (item=5473)
ok: [k8snode01] => (item=5473)
ok: [k8smaster01] => (item=5473)
ok: [k8snode01] => (item=179)
ok: [k8snode02] => (item=179)
ok: [k8smaster01] => (item=179)
ok: [k8snode02] => (item=5473)
ok: [k8snode01] => (item=5473)
ok: [k8smaster01] => (item=5473)
TASK [kubernetes-bootstrap : Reload firewalld] *********************************************************************************************************
changed: [k8smaster01]
changed: [k8snode01]
changed: [k8snode02]
PLAY RECAP *********************************************************************************************************************************************
k8smaster01 : ok=23 changed=3 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
k8snode01 : ok=23 changed=3 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
k8snode02 : ok=23 changed=3 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
Step 2: Initialize the single-node control plane
This deployment is for a single control plane node with integrated etcd. If you want to execute multiple control nodes (3 for HA), check out Create high availability cluster with kubeadm Official guide.
We will use kubeadm to bootstrap the smallest viable Kubernetes cluster in accordance with best practices. The benefit of kubeadm is that it supports other cluster lifecycle features, such as upgrade, DowngradeAnd management Boot token.
The single control node bootloader requires:
- Control node machine’s default IP address
- DNS name / load balancer IP (if more control nodes are planned to be added in the future)
- SSH access as root or sudo
Log in to the control node:
$ ssh [email protected]
Check the parameters that can be used to initialize the Kubernetes cluster:
$ kubeadm init --help
The standard parameters we will use are:
--pod-network-cidr
: Used to specify the IP address range of the Pod network.--apiserver-advertise-address
: The API server will announce the IP address it is listening on.--control-plane-endpoint
: Specify a stable IP address or DNS name for the control plane.--upload-certs
: Upload the control plane certificate to kubeadm-certs Secret.
- If Calico is used, the recommended Pod network is: 192.168.0.0/16
- For flannel, it is recommended to set the Pod network to 10.244.0.0/16
For me, I would run the following command:
sudo kubeadm init
--apiserver-advertise-address=192.168.122.10
--pod-network-cidr 192.168.0.0/16
--upload-certs
To be able to upgrade it Single control surface kubeadm cluster High availability You should specify –control-plane-endpoint to set a shared endpoint for all control plane nodes. Such an endpoint can be the DNS name or IP address of a load balancer.
kubeadm init
--apiserver-advertise-address=192.168.122.227
--pod-network-cidr 192.168.0.0/16
--control-plane-endpoint
--upload-certs
This is my installation output:
W0109 20:27:51.787966 18069 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0109 20:27:51.788126 18069 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8smaster01.computingforgeeks.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.122.10]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8smaster01.computingforgeeks.com localhost] and IPs [192.168.122.10 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8smaster01.computingforgeeks.com localhost] and IPs [192.168.122.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0109 20:32:51.776569 18069 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0109 20:32:51.777334 18069 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 15.507327 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
bce5c1ad320f4c64e42688e25526615d2ffd7efad3e749bc0c632b3a7834752d
[mark-control-plane] Marking the node k8smaster01.computingforgeeks.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8smaster01.computingforgeeks.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: nx1jjq.u42y27ip3bhmj8vj
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.122.10:6443 --token nx1jjq.u42y27ip3bhmj8vj
--discovery-token-ca-cert-hash sha256:c6de85f6c862c0d58cc3d10fd199064ff25c4021b6e88475822d6163a25b4a6c
Copy the kubectl configuration file.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Also checkout: Easily manage multiple Kubernetes clusters with kubectl and kubectx
Deploy Pod network to cluster
I will use Calico, but you are free to use any other Pod Network plugin of your choice.
kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
As shown in the output below, this will create many resources.
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Use the following command to confirm that all Pods are running.
watch kubectl get pods --all-namespaces
Once everything is working as expected, the output will look like this.
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5c45f5bd9f-c8mwx 1/1 Running 0 3m45s
kube-system calico-node-m5qmb 1/1 Running 0 3m45s
kube-system coredns-6955765f44-cz65r 1/1 Running 0 9m43s
kube-system coredns-6955765f44-mtch2 1/1 Running 0 9m43s
kube-system etcd-k8smaster01.computingforgeeks.com 1/1 Running 0 9m59s
kube-system kube-apiserver-k8smaster01.computingforgeeks.com 1/1 Running 0 9m59s
kube-system kube-controller-manager-k8smaster01.computingforgeeks.com 1/1 Running 0 9m59s
kube-system kube-proxy-bw494 1/1 Running 0 9m43s
kube-system kube-scheduler-k8smaster01.computingforgeeks.com 1/1 Running 0 9m59s
Please note that each pod has STATUS
of Running
.
Check Calico Document more details.
Step 3: Add the worker nodes to the cluster
Now that you have the control nodes ready, you can add new nodes where workloads (containers, pods, etc.) will run. You need to do this on every computer you use to run the Pod.
- SSH to the machine
$ ssh [email protected]
- Run the command output by kubeadm init. E.g:
sudo kubeadm join 192.168.122.10:6443 --token nx1jjq.u42y27ip3bhmj8vj
--discovery-token-ca-cert-hash sha256:c6de85f6c862c0d58cc3d10fd199064ff25c4021b6e88475822d6163a25b4a6c
If the token has expired, you can use the following command to generate a new token:
kubeadm token create
Get token
kubeadm token list
You can use the following command to get the value of Discovery-token-ca-cert-hash:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
This is a connection command output:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Run the same join command on all other Worker nodes, and then look at the available nodes to join the cluster using:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster01.computingforgeeks.com Ready master 26m v1.17.0
k8snode01.computingforgeeks.com Ready 4m35s v1.17.0
k8snode02.computingforgeeks.com Ready 2m4s v1.17.0
Step 4: Deploy Metrics Server to a Kubernetes cluster
Metrics Server is a cluster-wide aggregator for resource usage data. It collects metrics from the summary API, by Kubelet On each node. Use the following guidelines for deployment:
How to deploy Metrics Server to a Kubernetes cluster
With it, you now have a running Kubernetes cluster that you can use to develop Cloud native applications. We also provide additional guides on Kubernetes, such as:
How to manually pull out the container image used by Kubernetes kubeadm
Installing and using Helm 3 on a Kubernetes cluster
Installing and using Helm 2 on a Kubernetes cluster
Create Kubernetes service / user account and restrict it to one namespace using RBAC
How to configure Kubernetes dynamic volume configuration with Heketi and GlusterFS