Install Vault cluster in GKE through Helm, Terraform and BitBucket pipeline

motivation

Confidentiality management in an organization occupies a special place in daily business activities. From entering buildings to protecting personal and confidential documents in laptops or computers, secrets continue to appear, which shows that secrets are of great significance not only in our personal lives but also in the highways of enterprises. Secrets are anything you want to strictly control access to, such as API encryption keys, passwords, or certificates.

In addition, most applications in the current era are developing towards microservices, and Kubernetes has become the best platform for hosting applications designed in this new paradigm. Kubernetes also brings new opportunities and a series of challenges. It brings agility, self-healing, easy expansion, easy deployment, and a great way to run decoupled systems. Now is the secret question, Kubernetes provides a way to manage them locally. The only problem with it is that it can work if there are few workloads running or the team managing the cluster is relatively small. When the generated applications are in the range of hundreds, it becomes difficult to manage confidentiality in this way. In addition, the native Kubernetes Secrets engine lacks encryption capabilities, which brings security issues to the forefront.

HashiCorp Vault is a secret management solution designed to provide large-scale, easy secret management, and is well integrated with countless other tools including Kubernetes. It is an identity-based confidentiality and encryption management system. Let’s take a look at the features that Vault comes with:

Features of the vault

The main features of Vault are: Source Vault documentation

  • Secure secret storage: Any key/value secret can be stored in the Vault. Vault encrypts these secrets before writing them to persistent storage, so gaining access to the original storage is not sufficient to access your secrets. Vault can be written to disk, Consul, etc.
  • Dynamic secret: Vault can generate secrets on demand for certain systems (such as AWS or SQL databases).for example, When an application needs to access an S3 bucket, it will ask Vault to provide credentials, and Vault will generate an AWS key pair with valid permissions as needed. After creating these dynamic secrets, Vault will also automatically revoke them after the lease period ends.
  • data encryption: Vault can encrypt and decrypt data without storing it. This allows the security team to define encryption parameters, and developers can store encrypted data in SQL and other locations without having to design their own encryption methods.
  • Lease and renew: All secrets in the Vault have leases associated with them. At the end of the lease, Vault will automatically revoke the secret. Customers can renew the lease through the built-in renewal API.
  • Revoke: Vault has built-in support for secret revocation. Vault can revoke not only a single secret, but also a secret tree, because example All secrets read by a specific user, or all secrets of a specific type. Revocation facilitates key rollover and locks the system in the event of an intrusion.

Project prerequisites

  • Bit bucket account
  • BitBucket pipeline has been set up
  • Docker or any tool for creating images, such as Podman, Buildah, etc.
  • Existing Google Cloud Credentials in BitBucket environment variables (json)
  • Existing Google Cloud Bucket of Terraform backend (it will keep state)

We will create a terraform script, push it to BitBucket, and then the BitBucket pipeline will use the image we will build to take over and deploy the vault in Google Kubernetes Engine (GKE).

Install Vault cluster in Google Kubernetes Engine

We can now set up Vault on the existing Google Kubernetes Engine through Helm, BitBucket pipeline and Terraform. Here are the steps to help you get up and running. If you do not use BitBucket pipes in your settings, some parts are optional.

Step 1: Prepare Terraform and Google SDK images

In this step, we will create a Docker image containing Terraform and Google Cloud SKD, and then host it in DockerHub, so that BitBucket can pull and use it when deploying infrastructure. First let’s create a Dockerfile file and fill it with the following content. We will use Google’s clouddk image as a base, and then add Terraform.

$ vim Dockerfile
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
ENV TERRAFORM_VERSION=1.0.10
# Installing terraform
RUN apk --no-cache add curl unzip && 
    cd /tmp && 
    curl -o /tmp/terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && 
    unzip /tmp/terraform.zip && 
    cp /tmp/terraform /usr/local/bin && 
    chmod a+x /usr/local/bin/terraform && 
    rm /tmp/terraform /tmp/terraform.zip

After that, let’s build and label the image. Make sure that the Dockerfile file is the same location where you ran this command.

docker build -label imagename .

Mark image

docker tag imagename penchant/cloudsdk-terraform:latest

Then push it to the public DockerHub or any registry you like

docker push penchant/cloudsdk-terraform:latest

We completed the first part

Step 2: Prepare Terraform and Helm scripts

In order to avoid reinventing the wheel, this project borrowed heavily from a GitHub by mohsinrz. We thank and celebrate their excellent work. We will clone the project and customize it to fit our environment.

cd ~
git clone https://github.com/mohsinrz/vault-gke-raft.git

Since we already have a GKE cluster, we will not use modules specifically for creating clusters. We will further disable the use of certificates because BitBucket uses temporary containers and cannot store certificates in them, and we will have difficulty adding vault workers to the leader later.

We will add a GCP bucket to store Terraform state so that we can track changes in the content that will be deployed. Add the following content to the “main.yaml” file. Make sure that the bucket name already exists in GCP.

$ cd ~/vault-gke-raft
$ vim main.tf

## Disable the gke cluster module if you have on already
#module "gke-cluster" {
#  source                     = "./modules/google-gke-cluster/"
#  credentials_file           = var.credentials_file
#  region                     = var.region
#  project_id                 = "project-id"
#  cluster_name               = "dev-cluster-1"
#  cluster_location           = "us-central1-a"
#  network                    = "projects/${var.project_id}/global/networks/default"
#  subnetwork                 = "projects/${var.project_id}/regions/${var.region}/subnetworks/default"
#  initial_node_count         = var.cluster_node_count
#}

module "tls" {
  source                     = "./modules/gke-tls"
  hostname                   = "*.vault-internal"
}

module "vault" {
  source                     = "./modules/gke-vault"
  num_vault_pods             = var.num_vault_pods
  #cluster_endpoint           = module.gke-cluster.endpoint
  #cluster_cert               = module.gke-cluster.ca_certificate
  vault_tls_ca               = module.tls.ca_cert
  vault_tls_cert             = module.tls.cert 
  vault_tls_key              = module.tls.key
}

terraform {
  backend "gcs"{
    bucket      = "terraform-state-bucket"
    credentials = "gcloud-api-key.json"
  }
}

Another modification we will make is to disable TLS, because in our setup, a temporary container in BitBucket will provide our infrastructure, and some certificates are intended to be stored where terraform is running. Therefore, we will lose the certificate after the deployment is complete. To disable, navigate to the module folder and enter the Vault directory module. Then edit the “Vault.tf” file and make it look like the following. The changes made are:

  • We change the tlsDisable field from false to true
  • We change the VAULT_ADDR environment variable from https to http
  • We commented/removed the VAULT_CACERT environment variable
  • We change the tls_disable field from 0 to 1
  • We change VAULT_ADDR from 127.0.0.1 to 0.0.0.0
  • And deleted the certification path under the listener block

The same content has also been updated in the following file.

$ cd ~/vault-gke-raft/modules/vault
$ vim vault.tf

resource "helm_release" "vault" {
  name          = "vault"
  chart         = "${path.root}/vault-helm"
  namespace     = kubernetes_namespace.vault.metadata.0.name

  values = [<<EOF
global:
## changed tlsDisable field to true from false
  #tlsDisable: false
  tlsDisable: true
server:
  extraEnvironmentVars:
#changed VAULT_ADDR environment variable from https to http
    #VAULT_ADDR: https://127.0.0.1:8200
    VAULT_ADDR: https://0.0.0.0:8200
    VAULT_SKIP_VERIFY: true
#removed VAULT_CACERT environment variable
    #VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca
  extraVolumes:
    - type: secret
      name: vault-tls
  ha:
    enabled: true
    replicas: ${var.num_vault_pods}    

    raft:      
      # Enables Raft integrated storage
      enabled: true
      config: |
        ui = true

        listener "tcp" {
#changed tls_disable field from 0 to 1
          #tls_disable = 0
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
#removed the certificate paths here
          #tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
          #tls_key_file  = "/vault/userconfig/vault-tls/vault.key"
          #tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"           
        }

        storage "raft" {
          path = "/vault/data"
        }
ui:
  enabled: true
  serviceType: "LoadBalancer"
  serviceNodePort: null
  externalPort: 8200
EOF
]
}

We will make another modification to enable the Kubernetes provider to communicate with the GKE API. Navigate to the module folder and vault module directory. Then edit the “provider.tf” file. We added the details of the existing GKE cluster and used the values ​​in the Kubernetes provider. We commented on the one we got from the repo and added a new one as shown below. The helm provider was also edited by updating the host, token, and cluster ca certificate with existing content.

$ cd ~/vault-gke-raft/modules/vault
$ vim provider.tf
data "google_client_config" "provider" {}

data "google_container_cluster" "cluster-name" {
  name     = "cluster-name"
  location = "us-central1-a"
  project  = "project-name"
}

# This file contains all the interactions with Kubernetes
provider "kubernetes" {
  #host = google_container_cluster.vault.endpoint
  host = "https://${data.google_container_cluster.dev_cluster_1.endpoint}"
  token = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(
    data.google_container_cluster.dev_cluster_1.master_auth[0].cluster_ca_certificate,
  )
}

#provider "kubernetes" {
#  host  = var.cluster_endpoint
#  token = data.google_client_config.current.access_token
#
#  cluster_ca_certificate = base64decode(
#    var.cluster_cert,
#  )
#}

provider "helm" {
  kubernetes {
    #host  = var.cluster_endpoint
    host  = "https://${data.google_container_cluster.dev_cluster_1.endpoint}"
    #token = data.google_client_config.current.access_token
    token = data.google_client_config.provider.access_token
    cluster_ca_certificate = base64decode(data.google_container_cluster.dev_cluster_1.master_auth[0].cluster_ca_certificate,
    )
  }
}

After editing the file, let us clone vault-helm in the root directory, which will deploy the entire infrastructure for us at one time through the terraform helm provider

cd ~/vault-gke-raft
git clone https://github.com/hashicorp/vault-helm

Step 3: Create BitBucket pipeline file

In this step, we will create and populate the BitBucket pipeline file that will guide our deployment.As you can see, we are using the image we pushed to DockerHub step 1.

$ cd ~
$ vim bitbucket-pipelines.yaml

image: penchant/cloudsdk-terraform:latest ## The image
pipelines:
  branches:
    vault:
      - step:
          name: Deploy to Vault Namespace
          deployment: production
          script:
            - cd install-vault # I placed my files in this directory in the root of the files
            - export TAG=$(git log -1 --pretty=%h)
            - echo $GCLOUD_API_KEYFILE | base64 -d  > ./gcloud-api-key.json
            - gcloud auth activate-service-account --key-file gcloud-api-key.json
            - export GOOGLE_APPLICATION_CREDENTIALS=gcloud-api-key.json
            - gcloud config set project <your_project_id>
            - gcloud container clusters get-credentials <your_cluster> --zone=<your_zone> --project <your_project_id>
            - terraform init
            - terraform plan -out create_vault 
            - terraform apply -auto-approve create_vault
          services:
            - docker

Step 4: Initialize the cluster by creating a leader node/pod

After installing the vault cluster through terraform and helm, it’s time to boot the cluster and unblock it. Initialize the cluster with the vault-0 node as the leader, then we can unseal it, then join the remaining nodes to the cluster and unseal them.

Initialize the cluster with node vault-0 as the leader, as shown below:

$ kubectl exec -ti vault-0 -n vault -- vault operator init

Unseal Key 1: 9LphBlg31dBKuVCoOYRW+zXrS5zpuGeaFDdCWV3x6C9Y
Unseal Key 2: zfWTzDo9nDUIDLuqRAc4cVih1XzuZW8iEolc914lrMyS
Unseal Key 3: 2O3QUiio8x5W+IJq+4ze45Q3INL1Ek/2cHDiNHb3vXIz
Unseal Key 4: DoPBFgPte+Xh6L/EljPc79ZT2mYwQL6IAeDTLiBefwPV
Unseal Key 5: OW1VTaXIMDt0Q57STeI4mTh1uBFPJ2JvmS2vgYAFuCPJ

Initial Root Token: s.rLs6ycvvg97pQNnvnvzNZgAJ

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Now we have the key and root token. We will use the key to unseal the node and join each node to the cluster.

Step 5: Unblock the leader node/pod

We will use the unseal key in the output of the above command in step 4 to unseal the vault, as shown below. Run the command three times and provide the keys in the order generated above. After running the command, you will see a prompt asking you to enter one of the stamps generated above.Just copy and paste one of them and press Enter.

$ kubectl exec -ti vault-0 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       f4c34433-6ef1-59ca-c1c9-1a6cc0dfabff
Version            1.8.4
Storage Type       raft
HA Enabled         true

Second run

$ kubectl exec -ti vault-0 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       f4c34433-6ef1-59ca-c1c9-1a6cc0dfabff
Version            1.8.4
Storage Type       raft
HA Enabled         true

Run for the third time

$ kubectl exec -ti vault-0 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.8.4
Storage Type            raft
Cluster Name            vault-cluster-3d108027
Cluster ID              a75de185-7b51-6045-20ca-5a25ca9d9e70
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24

Currently, we have not added the remaining four vault statefulset nodes/pods to the cluster. If you check the status of the Pods, you will see that they are not ready yet. Let us confirm.

$ kubectl get pods -n vault

NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          14m
vault-1                                 0/1     Running   0          14m
vault-2                                 0/1     Running   0          14m
vault-3                                 0/1     Running   0          14m
vault-4                                 0/1     Running   0          14m
vault-agent-injector-5c8f78854d-twllz   1/1     Running   0          14m

As you can see, vault-1 to vault-4 are not ready yet (0/1)

Step 6: Add the remaining nodes to the cluster and unblock them

This is the step where we will gradually add each of them to the cluster. The procedure is as follows:

  • Add nodes to the cluster
  • Then use the threshold number shown in the above status command (kubectl exec -ti vault-0 -n vault — vault status) to unblock it. In this 3 example.
  • This means that we will run the unseal command three times for each node. When unblocking, please use the same key we used for node 1 above. let’s start.

Join other nodes to the cluster to Vault-1 node.

$ kubectl exec -ti vault-1 -n vault --  vault operator raft join --address "https://vault-1.vault-internal:8200" "https://vault-0.vault-internal:8200"     

Key       Value
---       -----
Joined    true

After successfully joining, unblock the node 3 times.

## First Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       188f79d8-a87f-efdf-4186-73327ade371a
Version            1.8.4
Storage Type       raft
HA Enabled         true

## Second Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       188f79d8-a87f-efdf-4186-73327ade371a
Version            1.8.4
Storage Type       raft
HA Enabled         true

## Third Time
$ kubectl exec -ti vault-1 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Then join the node Vault-2 To the cluster, and then unblock it as in the node Vault-1

$ kubectl exec -ti vault-2 -n vault --  vault operator raft join --address "https://vault-2.vault-internal:8200" "https://vault-0.vault-internal:8200"

Key       Value
---       -----
Joined    true

Unblock node Vault-2 Enter one of the 3 keys 3 times each time you run

##First Time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       60ab7a6a-e7dc-07c8-e73c-11c55bafc199
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Second time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       60ab7a6a-e7dc-07c8-e73c-11c55bafc199
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Third Time
$ kubectl exec -ti vault-2 -n vault -- vault operator unseal
Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Do the same for the remaining pods/nodes in the cluster that are not yet ready. Just check your Pod as follows

$ kubectl get pods -n vault

NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          14m
vault-1                                 1/1     Running   0          14m
vault-2                                 1/1     Running   0          14m
vault-3                                 0/1     Running   0          14m
vault-4                                 0/1     Running   0          14m
vault-agent-injector-5c8f78854d-twllz   1/1     Running   0          14m

Those with 0/1 are not ready yet, so add them to the cluster and unblock them.

Add node Vault-3

$ kubectl exec -ti vault-3 -n vault --  vault operator raft join --address "https://vault-3.vault-internal:8200" "https://vault-0.vault-internal:8200"

Key       Value
---       -----
Joined    true

Unblock node Vault-3 Again and again.

##First Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       733264c0-bfc6-6869-a3dc-167e642ad624
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Second Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       733264c0-bfc6-6869-a3dc-167e642ad624
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Third Time
$ kubectl exec -ti vault-3 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Add node Vault-4

$ kubectl exec -ti vault-4 -n vault --  vault operator raft join --address "https://vault-4.vault-internal:8200" "https://vault-0.vault-internal:8200"

Key       Value
---       -----
Joined    true

Unblock node Vault-4 Again and again.

##First Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 1>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       543e3a67-28f9-9730-86ae-4560d48c2f2e
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Second Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 2>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       543e3a67-28f9-9730-86ae-4560d48c2f2e
Version            1.8.4
Storage Type       raft
HA Enabled         true

##Third Time
$ kubectl exec -ti vault-4 -n vault -- vault operator unseal

Unseal Key (will be hidden): <Enter Unseal Key 3>
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.4
Storage Type       raft
HA Enabled         true

Now let’s check our Pod

$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          32m
vault-1                                 1/1     Running   0          32m
vault-2                                 1/1     Running   0          32m
vault-3                                 1/1     Running   0          32m
vault-4                                 1/1     Running   0          32m
vault-agent-injector-5c8f78854d-twllz   1/1     Running   0          32m

beautiful. You can see that they are all successfully prepared. We finally completed the vault ha cluster setup on the GKE platform. Next, we will introduce how to authenticate Kubernetes/GKE, create a secret, and then launch the sample application and inject the secret into the pod through the sidecar model. We hope that this document provides insight and is helpful for your use case. If you know how to automatically open the seal, please also point us in the right direction.

Finally, the great support and information we continue to receive is a blessing, and we pray that you will continue to succeed in all your efforts while changing the world. Have an amazing year-end season and stick to it, stay safe, and may your hard work reap the fruits you want🙂.

Other guides you will like include:

  • How to install gcloud GCP cli on Apple M1 macOS
  • Use Helm Chart to deploy Nginx Ingress Controller on Kubernetes
  • Separate Jenkinsfile and Kubernetes deployment YAML from Git source
  • Use k0s to deploy a Kubernetes cluster on Linux
  • Use Minikube to run Kubernetes on Debian