motivation
Confidentiality management in an organization occupies a special place in daily business activities. From entering buildings to protecting personal and confidential documents in laptops or computers, secrets continue to appear, which shows that secrets are of great significance not only in our personal lives but also in the highways of enterprises. Secrets are anything you want to strictly control access to, such as API encryption keys, passwords, or certificates.
In addition, most applications in the current era are developing towards microservices, and Kubernetes has become the best platform for hosting applications designed in this new paradigm. Kubernetes also brings new opportunities and a series of challenges. It brings agility, self-healing, easy expansion, easy deployment, and a great way to run decoupled systems. Now is the secret question, Kubernetes provides a way to manage them locally. The only problem with it is that it can work if there are few workloads running or the team managing the cluster is relatively small. When the generated applications are in the range of hundreds, it becomes difficult to manage confidentiality in this way. In addition, the native Kubernetes Secrets engine lacks encryption capabilities, which brings security issues to the forefront.
HashiCorp Vault is a secret management solution designed to provide large-scale, easy secret management, and is well integrated with countless other tools including Kubernetes. It is an identity-based confidentiality and encryption management system. Let’s take a look at the features that Vault comes with:
Features of the vault
The main features of Vault are: Source Vault documentation
- Secure secret storage: Any key/value secret can be stored in the Vault. Vault encrypts these secrets before writing them to persistent storage, so gaining access to the original storage is not sufficient to access your secrets. Vault can be written to disk, Consul, etc.
- Dynamic secret: Vault can generate secrets on demand for certain systems (such as AWS or SQL databases).for example, When an application needs to access an S3 bucket, it will ask Vault to provide credentials, and Vault will generate an AWS key pair with valid permissions as needed. After creating these dynamic secrets, Vault will also automatically revoke them after the lease period ends.
- data encryption: Vault can encrypt and decrypt data without storing it. This allows the security team to define encryption parameters, and developers can store encrypted data in SQL and other locations without having to design their own encryption methods.
- Lease and renew: All secrets in the Vault have leases associated with them. At the end of the lease, Vault will automatically revoke the secret. Customers can renew the lease through the built-in renewal API.
- Revoke: Vault has built-in support for secret revocation. Vault can revoke not only a single secret, but also a secret tree, because example All secrets read by a specific user, or all secrets of a specific type. Revocation facilitates key rollover and locks the system in the event of an intrusion.
Project prerequisites
- Bit bucket account
- BitBucket pipeline has been set up
- Docker or any tool for creating images, such as Podman, Buildah, etc.
- Existing Google Cloud Credentials in BitBucket environment variables (json)
- Existing Google Cloud Bucket of Terraform backend (it will keep state)
We will create a terraform script, push it to BitBucket, and then the BitBucket pipeline will use the image we will build to take over and deploy the vault in Google Kubernetes Engine (GKE).
Install Vault cluster in Google Kubernetes Engine
We can now set up Vault on the existing Google Kubernetes Engine through Helm, BitBucket pipeline and Terraform. Here are the steps to help you get up and running. If you do not use BitBucket pipes in your settings, some parts are optional.
Step 1: Prepare Terraform and Google SDK images
In this step, we will create a Docker image containing Terraform and Google Cloud SKD, and then host it in DockerHub, so that BitBucket can pull and use it when deploying infrastructure. First let’s create a Dockerfile file and fill it with the following content. We will use Google’s clouddk image as a base, and then add Terraform.
$ vim Dockerfile
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
ENV TERRAFORM_VERSION=1.0.10
# Installing terraform
RUN apk --no-cache add curl unzip &&
cd /tmp &&
curl -o /tmp/terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip &&
unzip /tmp/terraform.zip &&
cp /tmp/terraform /usr/local/bin &&
chmod a+x /usr/local/bin/terraform &&
rm /tmp/terraform /tmp/terraform.zipAfter that, let’s build and label the image. Make sure that the Dockerfile file is the same location where you ran this command.
docker build -label imagename .Mark image
docker tag imagename penchant/cloudsdk-terraform:latestThen push it to the public DockerHub or any registry you like
docker push penchant/cloudsdk-terraform:latestWe completed the first part
Step 2: Prepare Terraform and Helm scripts
In order to avoid reinventing the wheel, this project borrowed heavily from a GitHub by mohsinrz. We thank and celebrate their excellent work. We will clone the project and customize it to fit our environment.
cd ~
git clone https://github.com/mohsinrz/vault-gke-raft.gitSince we already have a GKE cluster, we will not use modules specifically for creating clusters. We will further disable the use of certificates because BitBucket uses temporary containers and cannot store certificates in them, and we will have difficulty adding vault workers to the leader later.
We will add a GCP bucket to store Terraform state so that we can track changes in the content that will be deployed. Add the following content to the “main.yaml” file. Make sure that the bucket name already exists in GCP.
$ cd ~/vault-gke-raft
$ vim main.tf
## Disable the gke cluster module if you have on already
#module "gke-cluster" {
# source = "./modules/google-gke-cluster/"
# credentials_file = var.credentials_file
# region = var.region
# project_id = "project-id"
# cluster_name = "dev-cluster-1"
# cluster_location = "us-central1-a"
# network = "projects/${var.project_id}/global/networks/default"
# subnetwork = "projects/${var.project_id}/regions/${var.region}/subnetworks/default"
# initial_node_count = var.cluster_node_count
#}
module "tls" {
source = "./modules/gke-tls"
hostname = "*.vault-internal"
}
module "vault" {
source = "./modules/gke-vault"
num_vault_pods = var.num_vault_pods
#cluster_endpoint = module.gke-cluster.endpoint
#cluster_cert = module.gke-cluster.ca_certificate
vault_tls_ca = module.tls.ca_cert
vault_tls_cert = module.tls.cert
vault_tls_key = module.tls.key
}
terraform {
backend "gcs"{
bucket = "terraform-state-bucket"
credentials = "gcloud-api-key.json"
}
}Another modification we will make is to disable TLS, because in our setup, a temporary container in BitBucket will provide our infrastructure, and some certificates are intended to be stored where terraform is running. Therefore, we will lose the certificate after the deployment is complete. To disable, navigate to the module folder and enter the Vault directory module. Then edit the “Vault.tf” file and make it look like the following. The changes made are:
- We change the tlsDisable field from false to true
- We change the VAULT_ADDR environment variable from https to http
- We commented/removed the VAULT_CACERT environment variable
- We change the tls_disable field from 0 to 1
- We change VAULT_ADDR from 127.0.0.1 to 0.0.0.0
- And deleted the certification path under the listener block
The same content has also been updated in the following file.
$ cd ~/vault-gke-raft/modules/vault
$ vim vault.tf
resource "helm_release" "vault" {
name = "vault"
chart = "${path.root}/vault-helm"
namespace = kubernetes_namespace.vault.metadata.0.name
values = [<<EOF
global:
## changed tlsDisable field to true from false
#tlsDisable: false
tlsDisable: true
server:
extraEnvironmentVars:
#changed VAULT_ADDR environment variable from https to http
#VAULT_ADDR: https://127.0.0.1:8200
VAULT_ADDR: https://0.0.0.0:8200
VAULT_SKIP_VERIFY: true
#removed VAULT_CACERT environment variable
#VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca
extraVolumes:
- type: secret
name: vault-tls
ha:
enabled: true
replicas: ${var.num_vault_pods}
raft:
# Enables Raft integrated storage
enabled: true
config: |
ui = true
listener "tcp" {
#changed tls_disable field from 0 to 1
#tls_disable = 0
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
#removed the certificate paths here
#tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
#tls_key_file = "/vault/userconfig/vault-tls/vault.key"
#tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
}
storage "raft" {
path = "/vault/data"
}
ui:
enabled: true
serviceType: "LoadBalancer"
serviceNodePort: null
externalPort: 8200
EOF
]
}We will make another modification to enable the Kubernetes provider to communicate with the GKE API. Navigate to the module folder and vault module directory. Then edit the “provider.tf” file. We added the details of the existing GKE cluster and used the values in the Kubernetes provider. We commented on the one we got from the repo and added a new one as shown below. The helm provider was also edited by updating the host, token, and cluster ca certificate with existing content.
$ cd ~/vault-gke-raft/modules/vault
$ vim provider.tf
data "google_client_config" "provider" {}
data "google_container_cluster" "cluster-name" {
name = "cluster-name"
location = "us-central1-a"
project = "project-name"
}
# This file contains all the interactions with Kubernetes
provider "kubernetes" {
#host = google_container_cluster.vault.endpoint
host = "https://${data.google_container_cluster.dev_cluster_1.endpoint}"
token = data.google_client_config.provider.access_token
cluster_ca_certificate = base64decode(
data.google_container_cluster.dev_cluster_1.master_auth[0].cluster_ca_certificate,
)
}
#provider "kubernetes" {
# host = var.cluster_endpoint
# token = data.google_client_config.current.access_token
#
# cluster_ca_certificate = base64decode(
# var.cluster_cert,
# )
#}
provider "helm" {
kubernetes {
#host = var.cluster_endpoint
host = "https://${data.google_container_cluster.dev_cluster_1.endpoint}"
#token = data.google_client_config.current.access_token
token = data.google_client_config.provider.access_token
cluster_ca_certificate = base64decode(data.google_container_cluster.dev_cluster_1.master_auth[0].cluster_ca_certificate,
)
}
}After editing the file, let us clone vault-helm in the root directory, which will deploy the entire infrastructure for us at one time through the terraform helm provider
cd ~/vault-gke-raft
git clone https://github.com/hashicorp/vault-helmStep 3: Create BitBucket pipeline file
In this step, we will create and populate the BitBucket pipeline file that will guide our deployment.As you can see, we are using the image we pushed to DockerHub step 1.
$ cd ~
$ vim bitbucket-pipelines.yaml
image: penchant/cloudsdk-terraform:latest ## The image
pipelines:
branches:
vault:
- step:
name: Deploy to Vault Namespace
deployment: production
script:
- cd install-vault # I placed my files in this directory in the root of the files
- export TAG=$(git log -1 --pretty=%h)
- echo $GCLOUD_API_KEYFILE | base64 -d > ./gcloud-api-key.json
- gcloud auth activate-service-account --key-file gcloud-api-key.json
- export GOOGLE_APPLICATION_CREDENTIALS=gcloud-api-key.json
- gcloud config set project <your_project_id>
- gcloud container clusters get-credentials <your_cluster> --zone=<your_zone> --project <your_project_id>
- terraform init
- terraform plan -out create_vault
- terraform apply -auto-approve create_vault
services:
- dockerStep 4: Initialize the cluster by creating a leader node/pod
After installing the vault cluster through terraform and helm, it’s time to boot the cluster and unblock it. Initialize the cluster with the vault-0 node as the leader, then we can unseal it, then join the remaining nodes to the cluster and unseal them.
Initialize the cluster with node vault-0 as the leader, as shown below:
$ kubectl exec -ti vault-0 -n vault -- vault operator init
Unseal Key 1: 9LphBlg31dBKuVCoOYRW+zXrS5zpuGeaFDdCWV3x6C9Y
Unseal Key 2: zfWTzDo9nDUIDLuqRAc4cVih1XzuZW8iEolc914lrMyS
Unseal Key 3: 2O3QUiio8x5W+IJq+4ze45Q3INL1Ek/2cHDiNHb3vXIz
Unseal Key 4: DoPBFgPte+Xh6L/EljPc79ZT2mYwQL6IAeDTLiBefwPV
Unseal Key 5: OW1VTaXIMDt0Q57STeI4mTh1uBFPJ2JvmS2vgYAFuCPJ
Initial Root Token: s.rLs6ycvvg97pQNnvnvzNZgAJ
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.Now we have the key and root token. We will use the key to unseal the node and join each node to the cluster.
Step 5: Unblock the leader node/pod
We will use the unseal key in the output of the above command in step 4 to unseal the vault, as shown below. Run the command three times and provide the keys in the order generated above. After running the command, you will see a prompt asking you to enter one of the stamps generated above.Just copy and paste one of them and press Enter.
$ kubectl exec -ti vault-0 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 1>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce f4c34433-6ef1-59ca-c1c9-1a6cc0dfabff
Version 1.8.4
Storage Type raft
HA Enabled trueSecond run
$ kubectl exec -ti vault-0 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce f4c34433-6ef1-59ca-c1c9-1a6cc0dfabff
Version 1.8.4
Storage Type raft
HA Enabled trueRun for the third time
$ kubectl exec -ti vault-0 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.8.4
Storage Type raft
Cluster Name vault-cluster-3d108027
Cluster ID a75de185-7b51-6045-20ca-5a25ca9d9e70
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
Raft Committed Index 24
Raft Applied Index 24Currently, we have not added the remaining four vault statefulset nodes/pods to the cluster. If you check the status of the Pods, you will see that they are not ready yet. Let us confirm.
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 14m
vault-1 0/1 Running 0 14m
vault-2 0/1 Running 0 14m
vault-3 0/1 Running 0 14m
vault-4 0/1 Running 0 14m
vault-agent-injector-5c8f78854d-twllz 1/1 Running 0 14mAs you can see, vault-1 to vault-4 are not ready yet (0/1)
Step 6: Add the remaining nodes to the cluster and unblock them
This is the step where we will gradually add each of them to the cluster. The procedure is as follows:
- Add nodes to the cluster
- Then use the threshold number shown in the above status command (kubectl exec -ti vault-0 -n vault — vault status) to unblock it. In this 3 example.
- This means that we will run the unseal command three times for each node. When unblocking, please use the same key we used for node 1 above. let’s start.
Join other nodes to the cluster to Vault-1 node.
$ kubectl exec -ti vault-1 -n vault -- vault operator raft join --address "https://vault-1.vault-internal:8200" "https://vault-0.vault-internal:8200"
Key Value
--- -----
Joined trueAfter successfully joining, unblock the node 3 times.
## First Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 1>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 188f79d8-a87f-efdf-4186-73327ade371a
Version 1.8.4
Storage Type raft
HA Enabled true
## Second Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 188f79d8-a87f-efdf-4186-73327ade371a
Version 1.8.4
Storage Type raft
HA Enabled true
## Third Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.8.4
Storage Type raft
HA Enabled trueThen join the node Vault-2 To the cluster, and then unblock it as in the node Vault-1
$ kubectl exec -ti vault-2 -n vault -- vault operator raft join --address "https://vault-2.vault-internal:8200" "https://vault-0.vault-internal:8200"
Key Value
--- -----
Joined trueUnblock node Vault-2 Enter one of the 3 keys 3 times each time you run
##First Time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 1>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 60ab7a6a-e7dc-07c8-e73c-11c55bafc199
Version 1.8.4
Storage Type raft
HA Enabled true
##Second time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 60ab7a6a-e7dc-07c8-e73c-11c55bafc199
Version 1.8.4
Storage Type raft
HA Enabled true
##Third Time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.8.4
Storage Type raft
HA Enabled trueDo the same for the remaining pods/nodes in the cluster that are not yet ready. Just check your Pod as follows
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 14m
vault-1 1/1 Running 0 14m
vault-2 1/1 Running 0 14m
vault-3 0/1 Running 0 14m
vault-4 0/1 Running 0 14m
vault-agent-injector-5c8f78854d-twllz 1/1 Running 0 14mThose with 0/1 are not ready yet, so add them to the cluster and unblock them.
Add node Vault-3
$ kubectl exec -ti vault-3 -n vault -- vault operator raft join --address "https://vault-3.vault-internal:8200" "https://vault-0.vault-internal:8200"
Key Value
--- -----
Joined trueUnblock node Vault-3 Again and again.
##First Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 1>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 733264c0-bfc6-6869-a3dc-167e642ad624
Version 1.8.4
Storage Type raft
HA Enabled true
##Second Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 733264c0-bfc6-6869-a3dc-167e642ad624
Version 1.8.4
Storage Type raft
HA Enabled true
##Third Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.8.4
Storage Type raft
HA Enabled trueAdd node Vault-4
$ kubectl exec -ti vault-4 -n vault -- vault operator raft join --address "https://vault-4.vault-internal:8200" "https://vault-0.vault-internal:8200"
Key Value
--- -----
Joined trueUnblock node Vault-4 Again and again.
##First Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 1>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 543e3a67-28f9-9730-86ae-4560d48c2f2e
Version 1.8.4
Storage Type raft
HA Enabled true
##Second Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 543e3a67-28f9-9730-86ae-4560d48c2f2e
Version 1.8.4
Storage Type raft
HA Enabled true
##Third Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.8.4
Storage Type raft
HA Enabled trueNow let’s check our Pod
$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 32m
vault-1 1/1 Running 0 32m
vault-2 1/1 Running 0 32m
vault-3 1/1 Running 0 32m
vault-4 1/1 Running 0 32m
vault-agent-injector-5c8f78854d-twllz 1/1 Running 0 32mbeautiful. You can see that they are all successfully prepared. We finally completed the vault ha cluster setup on the GKE platform. Next, we will introduce how to authenticate Kubernetes/GKE, create a secret, and then launch the sample application and inject the secret into the pod through the sidecar model. We hope that this document provides insight and is helpful for your use case. If you know how to automatically open the seal, please also point us in the right direction.
Finally, the great support and information we continue to receive is a blessing, and we pray that you will continue to succeed in all your efforts while changing the world. Have an amazing year-end season and stick to it, stay safe, and may your hard work reap the fruits you want🙂.
Other guides you will like include:
- How to install gcloud GCP cli on Apple M1 macOS
- Use Helm Chart to deploy Nginx Ingress Controller on Kubernetes
- Separate Jenkinsfile and Kubernetes deployment YAML from Git source
- Use k0s to deploy a Kubernetes cluster on Linux
- Use Minikube to run Kubernetes on Debian