Vault on Kubernetes deployment guide

This deployment guide covers the steps required to install and configure a single HashiCorp Vault cluster. Although not a strict requirement to follow the Vault Reference Architecture, please ensure you are familiar with the overall architecture design; for example:

• Install Vault on a dedicated Kubernetes cluster when possible
• If a dedicated cluster is unavailable, use correct node anti-affinity and taints/tolerances set for workload isolation
• Use Integrated Storage for the High Availability (HA) and storage backend

You will need to have enough worker node resources to host a 5 node Vault cluster for when using the Integrated Storage back-end. Given those assumptions, the below setup steps should be completed to install Vault.

1. Create Kubernetes namespace
2. Set up HashiCorp Helm repo
3. Configure Vault Helm chart
4. Install Vault
5. Initialize and unseal Vault
6. Next steps

Concepts overview

The Vault Helm chart is the recommended way to install and configure Vault on Kubernetes.

While the Helm chart automatically sets up complex resources and exposes the configuration to meet your requirements, it does not automatically operate Vault. You are still responsible for learning how to initialize, monitor, backup, upgrade, etc. the Vault cluster.

Kubernetes StatefulSets

In Kubernetes, there are multiple types of workload controller primitives and one of which is the StatefulSet. The StatefulSet, typically used to manage stateful applications, manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec, but unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

The Vault Helm Chart uses the StatefulSet deployment model. Although individual Pods in a StatefulSet are susceptible to failure, the persistent Pod identifiers make it easier to match existing volumes to the new Pods that replace any that have failed.

Kubernetes replicas

Before StatefulSets, there was the concept of ReplicaSets to manage Pods. It created a way to manage and ensure the correct number of Pods were running and the specification for what they should look like and how they should behave.

In the Vault Helm chart, it creates a default of 3 replicas in our StatefulSet for use with Consul Storage. (The Vault with Integrated Storage Reference Architecture requires 5 replicas) In a default Cloud Provider cluster such as Google's GKE or Amazon EKS, you will start with a 3 node Kubernetes cluster, so keep this in mind. Due to the requirement for Anti-Affinity rules to keep one pod on each node, you will likely want to either manually scale your cluster or setup node auto-scaling to match your needs.

Kubernetes namespaces

We recommend using a namespace other than default to deploy applications in production for reasons like isolation, access management, upgrade management, and overall logical separation of default Kubernetes elements and overall aid in long-term management.

Example:

Create a K8s namespace.

$ kubectl create namespace vault

View your new K8s objects.

$ kubectl --namespace='vault' get all

Kubernetes admission controllers

The Vault Helm chart can also optionally install the Vault Agent Sidecar Injector. The Vault Agent Sidecar Injector alters pod specifications to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Templates. By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault aware.

The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.

Setup Helm repo

Helm must be installed and configured on your machine. For help installing Helm, please refer to the Helm documentation or the Vault Installation to minikube via Helm with Integrated Storage tutorial.

To access the Vault Helm chart, add the Hashicorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Check that you have access to the chart.

$ helm search repo hashicorp/vault
NAME                                CHART VERSION   APP VERSION DESCRIPTION
hashicorp/vault                     0.25.0          1.14.0      Official HashiCorp Vault Chart

Using Helm charts

The helm install command requires the parameters [release name], [repo/chart], and has an option --namespace to declare what Kubernetes namespace to run the command against. It is common for Helm charts to be under significant development and thus we recommend to first run Helm with --dry-run before any install or upgrade to verify changes unless you are specifying an older version. The --dry-run flag will cause Helm to print the resulting YAML manifests that the Helm chart logically creates and applies. As no one company has the same needs as another, the helm install command also accepts parameters to override default configuration values inline or defined in a YAML file.

Examples:

Run helm install dry-run.

$ helm install vault hashicorp/vault --namespace vault --dry-run

List available releases.

$ helm search repo hashicorp/vault --versions
NAME                                CHART VERSION   APP VERSION DESCRIPTION
hashicorp/vault                     0.25.0          1.14.0      Official HashiCorp Vault Chart
hashicorp/vault                     0.24.1          1.13.1      Official HashiCorp Vault Chart
...

Install version 0.25.0.

$ helm install vault hashicorp/vault --namespace vault --version 0.25.0

Override default settings.

$ helm install vault hashicorp/vault \
    --namespace vault \
    --set "server.ha.enabled=true" \
    --set "server.ha.replicas=5" \
    --dry-run

Alternatively, specify the desired configuration in a file, override-values.yml.

$ cat > override-values.yml <<EOF
server:
  ha:
    enabled: true
    replicas: 5

EOF

Override the default configuration with the values read from the override-values.yml file.

$ helm install vault hashicorp/vault \
    --namespace vault \
    -f override-values.yml \
    --dry-run

Configure Vault Helm chart

For a production-ready install, we suggest that the Vault Helm chart is installed in high availability (HA) mode. This installs a StatefulSet of Vault server Pods with either Integrated Storage, or a Consul storage backend. The Vault Helm chart, however, can also configure Vault to run in dev, or standalone mode.

In this guide, we will demonstrate an HA mode installation with Integrated Storage.

The below override-values.yaml file is providing a subset of values for attributes that are commonly overridden when deploying Vault to production on Kubernetes. Create the below override-values.yaml file, and we will then discuss it and finally install the latest Vault Helm chart in HA Integrated Storage mode using these overrides.

$ cat > override-values.yml << EOF
# Vault Helm Chart Value Overrides
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
  # Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
  image:
    repository: "hashicorp/vault-k8s"
    tag: "latest"

  resources:
      requests:
        memory: 256Mi
        cpu: 250m
      limits:
        memory: 256Mi
        cpu: 250m

server:
  # Use the Enterprise Image
  image:
    repository: "hashicorp/vault-enterprise"
    tag: "1.15.0-ent"

  # These Resource Limits are in line with node requirements in the
  # Vault Reference Architecture for a Small Cluster
  resources:
    requests:
      memory: 8Gi
      cpu: 2000m
    limits:
      memory: 16Gi
      cpu: 2000m

  # For HA configuration and because we need to manually init the vault,
  # we need to define custom readiness/liveness Probe settings
  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 60

  # extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
  # used to include variables required for auto-unseal.
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/tls-ca/ca.crt

  # extraVolumes is a list of extra volumes to mount. These will be exposed
  # to Vault in the path `/vault/userconfig/<name>/`.
  extraVolumes:
    - type: secret
      name: tls-server
    - type: secret
      name: tls-ca
    - type: secret
      name: kms-creds

  # This configures the Vault Statefulset to create a PVC for audit logs.
  # See https://www.vaultproject.io/docs/audit/index.html to know more
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  # Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas: 5
    raft:
      enabled: true
      setNodeId: true

      config: |
        ui = true
        cluster_name = "vault-integrated-storage"
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/tls-server/fullchain.pem"
          tls_key_file = "/vault/userconfig/tls-server/server.key"
          tls_client_ca_file = "/vault/userconfig/tls-server/client-auth-ca.pem"
        }

        storage "raft" {
          path = "/vault/data"
            retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
            leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
            leader_client_key_file = "/vault/userconfig/tls-server/server.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
            leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
            leader_client_key_file = "/vault/userconfig/tls-server/server.key"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
            leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
            leader_client_key_file = "/vault/userconfig/tls-server/server.key"
          }
          retry_join {
              leader_api_addr = "https://vault-3.vault-internal:8200"
              leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
              leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
              leader_client_key_file = "/vault/userconfig/tls-server/server.key"
          }
          retry_join {
              leader_api_addr = "https://vault-4.vault-internal:8200"
              leader_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
              leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
              leader_client_key_file = "/vault/userconfig/tls-server/server.key"
          }

        }

# Vault UI
ui:
  enabled: true
  serviceType: "LoadBalancer"
  serviceNodePort: null
  externalPort: 8200

  # For Added Security, edit the below
  #loadBalancerSourceRanges:
  #   - < Your IP RANGE Ex. 10.0.0.0/16 >
  #   - < YOUR SINGLE IP Ex. 1.78.23.3/32 >
EOF

Overrides

In the above override-values.yml file we have created several YAML stanzas to tell Vault how to operate. We have made changes to the global, injector, server, and ui stanzas, lets discuss the contents of ones that are commonly changed.

TLS certificates

If a private Certificate Authority (CA) is used, you can pass the path to the CA Cert using the environment variable VAULT_CACERT through the use of the server.extraEnvironmentVars attribute, such as:

server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/tls-ca/ca.crt

To create this file and path inside the Vault Pods, you can create a Kubernetes Secret from the contents of the TLS Certificate file and this can be mounted using the server.extraVolumes attribute.

The Kubernetes Secret needs to be created before the installation of the Vault Helm chart and can be created.

For PEM encoded TLS Certs where you have the key:

$ kubectl --namespace='vault' create secret tls vault-ca-crt --cert ./tls-ca.cert --key ./tls-ca.key

For generic things like connection strings or consul tokens:

$ kubectl create secret generic consul-token --from-file=./consul_token

Version pinning

In production, we recommend that the Vault Agent Sidecar Injector Docker image is pinned to a specific version as newer releases of the Vault Helm chart may upgrade the admission webhook controller.

See Vault Agent Injector for additional details.

For Vault Enterprise version, the Docker image repository and version tag can be configured as such:

injector:
  enabled: true
  # Use the Vault K8s Image
  image:
    repository: 'hashicorp/vault-k8s'
    tag: 'latest'

Pod resource limits

In the Kubernetes environment, it is best practice to place resource limits on workloads to keep them from becoming noisy neighbors and possibly overrunning the node. In the above override-values.yaml file, request limits are defined for the Vault Agent Injector and Vault server Pod(s) based on the requirements listed in the Vault Reference Architecture. These should be taken into consideration carefully for the environment and used to ensure cluster stability in the event of a runaway Pod resource.

server:
  resources:
    requests:
      memory: 8Gi
      cpu: 2000m
    limits:
      memory: 16Gi
      cpu: 2000m

In this specification, we are requesting 8 Gigabytes (Gi) of memory and 2000 mili-cpu's (m) (equivalent to 2 vCPU's) and placing a hard limit of 16Gi and 2000m. The runtime prevents the container from using more than the configured resource limit. For example: when a process in the container tries to consume more than the allowed amount of memory, the system kernel terminates the process that attempted the allocation, with an out of memory (OOM) error.

See note on request limits

Stateful storage

Both the dataStorage and the auditStorage stanzas of the Vault Helm chart values file make use of PhysicalVolumeClaims (PVC). This configures the Vault Statefulset to create a PVC for data storage when using the file or Raft storage backend Creating these PVCs is what allows for persisting the Vault data through redeployment of the Vault server Pod(s).

server:
  dataStorage:
    enabled: true

  auditStorage:
    enabled: true

Refer to Vault Integrated Storage for more information.

Vault listener config

In the YAML stanza server.ha.config is creating the HCL configuration file that Vault will use. The listener portion of the HCL configures the addresses and ports on which Vault will respond to requests.

listener "tcp" {
  address         = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_client_ca_file = "/path/to/ca_for_client_auth.pem"
  tls_cert_file = "/path/to/fullchain.pem"
  tls_key_file  = "/path/to/privkey.pem"
}

The following parameters are set for the tcp listener portion of the HCL config:

• address(string: "127.0.0.1:8200") - Changing from the loopback address to allow external access to the Vault UI
• cluster_address(string: "127.0.0.1:8201") – Specifies the address to bind to for cluster server-to-server requests.
• tls_client_ca_file(string: <required-if-enabled>) - Must be set when using TLS client authentication
• tls_cert_file(string: <required-if-enabled>) - Must be set when using TLS; full chain of both leaf, any intermediates, and root certificate
• tls_key_file(string: <required-if-enabled>) - Must be set when using TLS

More information about tcp listener configuration.

For example, to create a secret object in the vault namespace, do the following:

$ kubectl --namespace='vault' create secret tls tls-secret --cert=path/to/tls.cert --key=path/to/tls.key

Vault seal config

The seal portion of the Vault configuration specifies the seal type to use for additional data protection such as using hardware security module (HSM) or Cloud Key Management Server (KMS) solutions to encrypt and decrypt the Vault root key (previously known as master key) to automatically unseal Vault. This config portion is optional, and if this is not configured, Vault will use the Shamir algorithm to cryptographically split the root key. However, you can migrate to Auto Unseal after the fact.

Review the seal configuration documentation for more detail.

An example Cloud KMS example is:

  seal "gcpckms" {
    project     = "<NAME OF PROJECT>"
    region      = "global"
    key_ring    = "<NAME OF KEYRING>"
    crypto_key  = "<NAME OF KEY>"
  }

Auto unseal

The Vault Helm chart may be configured with one of several options for enabling the Auto Unseal feature.

Google KMS auto unseal

The Helm chart may be run with Google KMS for Auto Unseal. This enables Vault server pods to auto unseal if they are rescheduled.

Vault Helm requires the Google Cloud KMS credentials stored in credentials.json and mounted as a secret in each Vault server pod.

Create the secret - GCP KMS

Before adding the seal to your configuration, create the secret in Kubernetes:

$ kubectl create secret generic kms-creds --from-file=credentials.json

Vault Helm mounts this to /vault/userconfig/kms-creds/credentials.json.

Config example - GCP KMS

This is an example Vault Helm configuration that uses Google KMS:

global:
  enabled: true

server:
  extraEnvironmentVars:
    GOOGLE_REGION: global
    GOOGLE_PROJECT: <PROJECT NAME>
    GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/kms-creds/credentials.json

  extraVolumes:
    - type: 'secret'
      name: 'kms-creds'

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true
      
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "gcpckms" {
        project     = "<NAME OF PROJECT>"
        region      = "global"
        key_ring    = "<NAME OF KEYRING>"
        crypto_key  = "<NAME OF KEY>"
      }

      storage "raft" {
        path = "/vault/data"
      }

Amazon EKS auto unseal

The Helm chart may be run with AWS EKS for Auto Unseal. This enables Vault server pods to auto unseal if they are rescheduled.

Vault Helm requires the AWS credentials stored as environment variables that are defined in each Vault server pod.

Create the secret - AWS EKS

Before adding the seal to your configuration, create a secret with your EKS access key/secret:

$ kubectl create secret generic eks-creds \
    --from-literal=AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID?}" \
    --from-literal=AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY?}"

Config example - AWS EKS

This is an example Vault Helm configuration that uses AWS EKS:

global:
  enabled: true

server:
  extraSecretEnvironmentVars:
    - envName: AWS_ACCESS_KEY_ID
      secretName: eks-creds
      secretKey: AWS_ACCESS_KEY_ID
    - envName: AWS_SECRET_ACCESS_KEY
      secretName: eks-creds
      secretKey: AWS_SECRET_ACCESS_KEY

  ha:
    enabled: true
    config: |
      ui = true
      cluster_name = "vault-integrated-storage"

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "awskms" {
        region     = "KMS_REGION_HERE"
        kms_key_id = "KMS_KEY_ID_HERE"
      }

      storage "raft" {
        path = "/vault/data"
      }

Vault storage config

The storage portion of the HCL configures the storage backend, which represents the path location for the durable storage of Vault's data. Below, retry_join statements are used and reflect the number of replicas defined in the Vault Helm chart. The Vault Helm chart assigns index values starting at <deployment name> - <index name> for the Pod name values.

As we are focused on Vault with Integrated Storage, we configured the Vault HA Raft portion of the override-values.yaml file above as seen here:

storage "raft" {
  path = "/vault/data"

  retry_join {
    leader_api_addr = "https://vault-0.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-ca/ca.crt"
    leader_client_cert_file = "/vault/userconfig/vault-cert/tls.crt"
    leader_client_key_file = "/vault/userconfig/vault-key/tls.key"
  }
  retry_join {
    leader_api_addr = "https://vault-1.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-ca/ca.crt"
    leader_client_cert_file = "/vault/userconfig/vault-cert/tls.crt"
    leader_client_key_file = "/vault/userconfig/vault-key/tls.key"
  }
  retry_join {
    leader_api_addr = "https://vault-2.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-ca/ca.crt"
    leader_client_cert_file = "/vault/userconfig/vault-cert/tls.crt"
    leader_client_key_file = "/vault/userconfig/vault-key/tls.key"
  }

}

Cloud auto_join

Cloud auto_join enables Vault nodes to discover Vault cluster leaders with non-static IP addresses with go-discover.

storage "consul" {
  address = "consul.svc.local"
  path = "vault/"
  # Optional Token
  # token = "<CONSUL_TOKEN>""
}

Additional server config

There are additional things that can be added to the server.ha.config and server YAML stanzas, but are highly dependent on your environment such as:

• Cluster Name
• Vault Telemetry Config
• Vault Service Registration
• Load Balancers and Replication
• Vault UI
• Liveness/Readiness Probes

Cluster name

If you are using Prometheus for monitoring and alerting, we recommend to set the cluster_name parameter in the Vault configuration.

With the Vault Helm chart, a Vault configuration is set with the config parameter.

server:
  affinity: ""
  standalone:
    enabled: true
    config: |
      ui = true
      cluster_name = "standalone"

      storage "file" {
        path    = "/vault/data"
      }

      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_disable = "true"
      }

Vault telemetry config

The telemetry portion of the HCL config specifies various parameters needed for Vault to publish metrics to upstream systems.

An example for configuring Vault telemetry to forward metrics to a statsd server would look like the below:

telemetry {
  statsd_address = "statsd.company.local:8125"
}

If you decide to configure Vault to publish telemetry data, refer to the telemetry configuration section documentation.

Vault service registration

Kubernetes Service Registration tags Vault pods with their current status for use with selectors. Service registration is only available when Vault is running in High Availability mode.

Add the below configuration to the server.ha.config YAML stanza to automatically configure service registration via environment variables.

service_registration "kubernetes" {}

Alternatively, you can add the below configuration to the server.ha.config YAML stanza to manually configure the Namespace and Pod names:

service_registration "kubernetes" {
  namespace      = "my-namespace"
  pod_name       = "my-pod-name"
}

For more information, refer to the Kubernetes Service Registration documentation.

Load balancers and replication

When using a load balancer in front of Vault, replication traffic should always be directed to the active Vault. When using the Helm chart and enabling the server.service attribute, you can choose from an internal "headless" ClusterIP, NodePort, or LoadBalancer.

As some customer environments may only use an ingress controller such as nginx to reach the outside world. The Vault Helm chart via the server.ingress attribute, will deploy Kubernetes services called vault and vault-active, which can be used as a service selector for user-deployed front-end load balancers such as nginx.

Additionally, the Helm chart has annotation values for nginx:

kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: 'true'

However, Ingresses are not recommended for use with Vault as they operate above the TCP Layer (L7) and require extra work to configure for use with TLS. As per the Production Hardening Guide, the recommended pattern Layer 4 (TCP) LoadBalancer for Vault cluster replication and bidirectional Vault cluster traffic typically traveling over port 8201 when the Vault cluster is being used by clients or replicating to Vault clusters external to the Kubernetes cluster.

For more information about load balancers and replication, refer to the Port Traffic Consideration with Load Balancer section of the Monitoring Vault Replication tutorial.

Vault UI

Vault features a web-based user interface, allowing you to create, read, update, and delete secrets, authenticate, unseal, and more using a graphical user interface, rather than the CLI or API.

Vault should not be deployed in a public internet facing environment. As such, enabling the Vault UI is typically of benefit to provide a more familiar experience to administrators who are not as comfortable working on the command line, or who do not have alternative access. The Vault Helm chart makes it accessible via an internal Service, or a LoadBalancer object.

Viewing the Vault UI

If for security reasons you chose not to expose the UI external to the Kubernetes cluster, the Vault UI can also be exposed via port-forwarding.

In this case, you can expose the Vault UI with port-forwarding:

$ kubectl port-forward vault-0 8200:8200
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200
##...

Liveness/Readiness probes

Probes are essential for detecting failures, rescheduling and using pods in Kubernetes. The Helm chart offers configurable readiness and liveliness probes which can be customized for a variety of use cases.

Vault's /sys/health endpoint can be customized to change the behavior of the health check. For example, we can change the Vault readiness probe to show the Vault pods are ready even if they're still uninitialized and sealed using the following probe:

server:
  readinessProbe:
    enabled: true
    path: '/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204'

Using this customized probe, a postStart script could automatically run once the pod is ready for additional setup.

Install vault

Once you have finished creating the override-values.yaml file, go ahead and install the latest version of the Vault Helm chart in the vault namespace with parameters override-values.yml applied.

$ helm install vault hashicorp/vault --namespace vault -f override-values.yaml

Initialize and unseal Vault

After the Vault Helm chart is installed, one of the Vault servers need to be initialized. The initialization generates the credentials necessary to unseal all the Vault servers.

CLI initialize and unseal

View all the Vault pods in the current namespace:

$ kubectl get pods --selector='app.kubernetes.io/name=vault' --namespace='vault'
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          1m49s
vault-1                                 0/1     Running   0          1m49s
vault-2                                 0/1     Running   0          1m49s

Initialize one Vault server with the default number of key shares and default key threshold:

$ kubectl exec --stdin=true --tty=true vault-0 -- vault operator init  --namespace='vault'
Unseal Key 1: MBFSDepD9E6whREc6Dj+k3pMaKJ6cCnCUWcySJQymObb
Unseal Key 2: zQj4v22k9ixegS+94HJwmIaWLBL3nZHe1i+b/wHz25fr
Unseal Key 3: 7dbPPeeGGW3SmeBFFo04peCKkXFuuyKc8b2DuntA4VU5
Unseal Key 4: tLt+ME7Z7hYUATfWnuQdfCEgnKA2L173dptAwfmenCdf
Unseal Key 5: vYt9bxLr0+OzJ8m7c7cNMFj7nvdLljj0xWRbpLezFAI9

Initial Root Token: s.zJNwZlRrqISjyBHFMiEca6GF
##...

The output displays the key shares and initial root key generated.

Unseal the Vault server with the key shares until the key threshold is met:

## Unseal the first vault server until it reaches the key threshold
$ kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal # ... Unseal Key 1
$ kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal # ... Unseal Key 2
$ kubectl exec --stdin=true --tty=true vault-0 -- vault operator unseal # ... Unseal Key 3

Repeat the unseal process for all Vault server pods. When all Vault server pods are unsealed they report READY 1/1.

$ kubectl get pods --selector='app.kubernetes.io/name=vault'
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          1m49s
vault-1                                 1/1     Running   0          1m49s
vault-2                                 1/1     Running   0          1m49s

Operational considerations

Upgrading Vault on Kubernetes

To upgrade Vault on Kubernetes, we recommend that you follow the same pattern as generally upgrading Vault, except you can use the Helm chart to update the Vault server StatefulSet. It is important to understand how to generally upgrade Vault before reading this section.

The Vault StatefulSet uses OnDelete update strategy. It is critical to use OnDelete instead of RollingUpdate because standbys must be updated before the active primary. A failover to an older version of Vault must always be avoided.

Upgrading Vault servers

To initiate the upgrade, set the server.image values to the desired Vault version, either in a values yaml file or on the command line. For illustrative purposes, the example below uses an image tag of hashicorp/vault:123.456.

server:
  image:
    repository: 'hashicorp/vault'
    tag: '123.456'

Next, list the Helm versions and choose the desired version to install.

$ helm search repo hashicorp/vault
NAME            CHART VERSION APP VERSION DESCRIPTION
hashicorp/vault 0.25.0         1.14.0      Official HashiCorp Vault Chart

Next, test the upgrade with --dry-run first to verify the changes sent to the Kubernetes cluster.

$ helm upgrade vault hashicorp/vault --version=0.25.0 \
    --set='server.image.repository=hashicorp/vault' \
    --set='server.image.tag=123.456' \
    --dry-run

This should cause no changes (although the resources are updated). If everything is stable, helm upgrade can be run.

The helm upgrade command should have updated the StatefulSet template for the Vault servers, however, no pods have been deleted. The pods must be manually deleted to upgrade. Deleting the pods does not delete any persisted data.

If Vault is not deployed using ha mode, the single Vault server may be deleted by running:

$ kubectl delete pod <name of Vault pod>

If Vault is deployed using ha mode, the standby pods must be upgraded first. Use the selector vault-active=false to delete all non-active primary vault pods:

$ kubectl delete pod  --selector="vault-active=false"

If auto-unseal is not being used, the newly scheduled Vault standby pods needs to be unsealed:

$ kubectl exec --stdin=true --tty=true <name of pod> -- vault operator unseal

Finally, once the standby nodes have been updated and unsealed, delete the active primary:

$ kubectl delete pod <name of Vault primary>

Similar to the standby nodes, the former primary also needs to be unsealed:

$ kubectl exec --stdin=true --tty=true <name of pod> -- vault operator unseal

After a few moments the Vault cluster should elect a new active primary. The Vault cluster is now upgraded!

Protecting sensitive Vault configurations

Vault Helm renders a Vault configuration file during installation and stores the file in a Kubernetes configmap. Some configurations require sensitive data to be included in the configuration file and would not be encrypted at rest once created in Kubernetes.

The following example shows how to add extra configuration files to Vault Helm to protect sensitive configurations from being in plaintext at rest using Kubernetes secrets.

First, create a partial Vault configuration with the sensitive settings Vault loads during startup:

$ cat > config.hcl <<EOF
storage "mysql" {
username = "user1234"
password = "secret123!"
database = "vault"
}
EOF

Next, create a Kubernetes secret containing this partial configuration:

$ kubectl create secret generic vault-storage-config \
    --from-file=config.hcl

Finally, mount this secret as an extra volume and add an additional -config flag to the Vault startup command:

$ helm install vault hashicorp/vault \
  --set='server.extraVolumes[0].type=secret' \
  --set='server.extraVolumes[0].name=vault-storage-config' \
  --set='server.extraArgs=-config=/vault/userconfig/vault-storage-config/config.hcl'

Architecture

We recommend running Vault on Kubernetes with the same general architecture as running it anywhere else. There are some benefits Kubernetes can provide that eases operating a Vault cluster and we document those below. The standard production deployment guide is still an important to read even if running Vault within Kubernetes.

Production deployment checklist

• End-to-End TLS: Vault should always be used with TLS in production. If intermediate load balancers or reverse proxies are used to front Vault, they should not terminate TLS. This way traffic is always encrypted in transit to Vault and minimizes risks introduced by intermediate layers. See the official documentation for example on configuring Vault Helm to use TLS.

• Single Tenancy: Vault should be the only main process running on a machine. This reduces the risk that another process running on the same machine is compromised and can interact with Vault. This can be accomplished by using Vault Helm's affinity configurable. See the official documentation for example on configuring Vault Helm to use affinity rules.

• Enable Auditing: Vault supports several auditing backends. Enabling auditing provides a history of all operations performed by Vault and provides a forensics trail in the case of misuse or compromise. Audit logs securely hash any sensitive data, but access should still be restricted to prevent any unintended disclosures. Vault Helm includes a configurable auditStorage option that provisions a persistent volume to store audit logs. See the official documentation for an example on configuring Vault Helm to use auditing.

• Immutable Upgrades: Vault relies on an external storage backend for persistence, and this decoupling allows the servers running Vault to be managed immutably. When upgrading to new versions, new servers with the upgraded version of Vault are brought online. They are attached to the same shared storage backend and unsealed. Then the old servers are destroyed. This reduces the need for remote access and upgrade orchestration which may introduce security gaps. See the upgrade section for instructions on upgrading Vault on Kubernetes.

• Upgrade Frequently: Vault is actively developed, and updating frequently is important to incorporate security fixes and any changes in default settings such as key lengths or cipher suites. Subscribe to the Vault mailing list and GitHub CHANGELOG for updates.

• Restrict Storage Access: Vault encrypts all data at rest, regardless of which storage backend is used. Although the data is encrypted, an attacker with arbitrary control can cause data corruption or loss by modifying or deleting keys. Access to the storage backend should be restricted to only Vault to avoid unauthorized access or operations.