Vault on Amazon EKS

Quick Start Reference Deployment

QS

October 2020
Daniel Callao, HashiCorp
Andrew Gargan, AWS Quick Start team

Visit the GitHub repository for source files and to post feedback, report bugs, or submit feature ideas for this Quick Start.

This Quick Start was created by HashiCorp in collaboration with Amazon Web Services (AWS). Quick Starts are automated reference deployments that use AWS CloudFormation templates to deploy key technologies on AWS, following AWS best practices.

Overview

This Quick Start reference deployment guide provides step-by-step instructions for deploying HashiCorp Vault on Amazon Elastic Kubernetes Service (Amazon EKS) via HashiCorp Vault’s Helm chart.

HashiCorp Vault is a product that centrally secures, stores, and controls access to tokens, passwords, certificates, and encryption keys through its user interface (UI), command line interface (CLI), or HTTP application programming interface (API). HashiCorp Vault’s core use cases include the following:

  • Secrets management: Helps to securely manage and deploy secrets across different environments, applications, and services.

  • Encryption and data protection: Manage encryption and keys for developers and operators across different environments, applications, and services.

  • Privileged-access management: Helps to secure workloads for application-to-application and user-to-application credential management across different environments and services.

This Quick Start is for DevOps professionals and application developers who want to manage their secrets, data, and key-value stores. It is deployed via HashiCorp Vault’s Helm chart, which contains all of the resource definitions to install and configure Vault inside of a Kubernetes cluster.

Amazon may share user-deployment information with the AWS Partner that collaborated with AWS on the Quick Start.

Software licenses

The software included in this deployment is licensed under the Mozilla Public License. For more information, see Mozilla Public License 2.0. No other licenses are required.

Architecture

Deploying this Quick Start with default parameters into an existing Amazon EKS cluster builds the following environment. For a diagram of the new VPC and Amazon EKS cluster, see the Amazon EKS on the AWS Cloud.

Architecture
Figure 1. Quick Start architecture for Vault on Amazon EKS

As shown in figure 1, the Quick Start sets up the following:

In AWS:

  • An Elastic Load Balancer for the Vault UI.

  • An AWS Certificate Manager (ACM) certificate for the Vault UI.

  • A boot-vault IAM role to bootstrap the Vault servers.

  • A vault-server IAM role for Vault to access AWS Key Management Service (AWS KMS) for auto unseal.

  • AWS Secrets Manager to store the Vault on Amazon EKS root secret.

  • An AWS KMS key for auto unseal.

In Kubernetes:

  • A dedicated node group for Vault on Amazon EKS.

  • A dedicated namespace for Vault on Amazon EKS.

  • An internal Vault TLS certificate and certificate authority for securing communications.

  • For the Vault service:

    • Vault server pods.

    • A Vault UI.

Planning the deployment

Specialized knowledge

This deployment guide requires a moderate level of familiarity with AWS services. If you’re new to AWS, see the Getting Started Resource Center and AWS Training and Certification. These sites provide materials for learning how to design, deploy, and operate your infrastructure and applications on the AWS Cloud.

This Quick Start assumes familiarity with Amazon EKS, AWS CloudFormation, and Kubernetes.

AWS account

If you don’t already have an AWS account, create one at https://aws.amazon.com by following the on-screen instructions. Part of the sign-up process involves receiving a phone call and entering a PIN using your phone’s keypad.

Your AWS account is automatically signed up for all AWS services. You are charged only for the services you use.

Amazon EKS cluster

If you deploy your cluster into an existing Amazon EKS cluster that was not created by the Amazon EKS on the AWS Cloud Quick Start, you must configure your cluster to allow this Quick Start to manage it. For more information, see the Deployment steps section.

IAM permissions

Before launching the Quick Start, you must log in to the AWS Management Console with IAM permissions for the resources and actions the templates deploy.

The AdministratorAccess managed policy within IAM provides sufficient permissions, although your organization may choose to use a custom policy with more restrictions.

Deployment options

This Quick Start provides three deployment options:

  • Deploy Vault into a new VPC (end-to-end deployment). This option builds a new AWS environment consisting of the VPC, subnets, NAT gateways, security groups, bastion hosts, EKS cluster, a node group, and other infrastructure components. It then deploys Vault into this new EKS cluster.

  • Deploy Vault into a new EKS cluster of an existing VPC. This option builds a new Amazon EKS cluster, node group, and other infrastructure components into an existing VPC. It then deploys Vault into this new EKS cluster.

  • Deploy Vault into an existing EKS cluster. This option provisions Vault in your existing AWS infrastructure. Note that when deploying into an EKS cluster that was not created by the Amazon EKS on the AWS Cloud Quick Start, you must prepare the cluster as described in the Deployment steps section.

Deployment steps

Sign in to your AWS account

  1. Sign in to your AWS account at https://aws.amazon.com with an IAM user role that has the necessary permissions. For details, see Planning the deployment, earlier in this guide.

  2. Ensure that your AWS account is configured correctly, as discussed in the Technical requirements section.

Prepare existing EKS cluster

This step is only required if you launch this Quick Start into an existing Amazon EKS cluster that was not created using the Amazon EKS on the AWS Cloud deployment. If you want to create a new EKS cluster with your deployment, skip to step 3.
  1. Sign in to your AWS account, and launch the cluster preparation template.

  2. The template launches in the US East (Ohio) Region by default. To change the Region, choose another Region from list in the upper-right corner of the navigation bar.

  3. On the Create stack page, keep the default setting for the template URL, and then choose Next.

  4. On the Specify stack details page, change the stack name if needed. Enter the name of the Amazon EKS cluster you want to deploy to as well as the subnet IDs and security group ID associated with the cluster. These can be obtained from the EKS cluster console.

  5. On the Options page, specify the key-value pairs for resources in your stack, and set advanced options. When you’re done, choose Next.

  6. On the Review page, review and confirm your template settings. Under Capabilities, select the two check boxes to acknowledge that the template creates IAM resources and might require the ability to automatically expand macros.

  7. Choose Create stack to deploy the stack.

  8. Monitor the stack’s status until it is CREATE_COMPLETE.

  9. From the Outputs section of the stack, note the KubernetesRoleArn and HelmRoleArn roles.

  10. Add the roles to the aws-auth config map in your cluster, specifying system:masters for the groups. This allows the Quick Start to manage your cluster via AWS CloudFormation. For more information, see Managing users or IAM roles for your cluster.

Unless you are customizing the Quick Start templates for your own deployment projects, we recommend that you keep the default settings for the parameters labeled Quick Start S3 bucket name, Quick Start S3 bucket Region, and Quick Start S3 key prefix. Changing these parameter settings automatically updates code references to point to a new Quick Start location. For more information, see the AWS Quick Start Contributor’s Guide.

Launch the Quick Start

You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional cost for using this Quick Start. For full details, see the pricing pages for each AWS service used by this Quick Start. Prices are subject to change.
  1. Sign in to your AWS account, and choose one of the following options to launch the AWS CloudFormation template. For help with choosing an option, see the Deployment options section, earlier in this guide.

Deploy into a new VPC and new Amazon EKS cluster

Deploy into a new Amazon EKS cluster in an existing VPC

Deploy into an existing Amazon EKS cluster

View template

View template

View template

New cluster deployments take about 90 minutes to complete. Existing-cluster deployments take about 10 minutes to complete.

If you deploy Vault into an existing VPC, ensure that any private subnets have NAT gateways in their route tables to allow the Quick Start to download packages and software. Also, ensure that the domain name in the DHCP options is configured. For more information, see DHCP options sets.
  1. Check the AWS Region that’s displayed in the upper-right corner of the navigation bar, and change it if necessary. This is where the network infrastructure for Vault is built. The template launches in the us-east-1 Region by default.

  2. On the Create stack page, keep the default setting for the template URL, and then choose Next.

  3. On the Specify stack details page, change the stack name if needed. Review the parameters for the template. A reference is provided in the Parameter reference section. Provide values for the parameters that require input. For all other parameters, review the default settings, and customize them as necessary.

  4. On the Options page, specify the key-value pairs for resources in your stack, and set advanced options. When you’re done, choose Next.

  5. On the Review page, review and confirm the template settings. Under Capabilities, select the two check boxes to acknowledge that the template creates IAM resources and might require the ability to automatically expand macros.

  6. Choose Create stack to deploy the stack.

  7. Monitor the status of the stack. When the status is CREATE_COMPLETE, the Vault deployment is ready.

  8. Use the values displayed in the Outputs tab for the stack, as shown in Vault outputs after successful deployment, to view the created resources.

cfn_outputs
Figure 2. Vault outputs after successful deployment

Post deployment steps

Kubernetes Consul deployment namespace and dedicated node selection

This deployment creates a vault-server namespace by default. Verify the namespace in Kubernetes:

$ kubectl get ns
NAME              STATUS   AGE
default           Active   4d7h
kube-node-lease   Active   4d7h
kube-public       Active   4d7h
kube-system       Active   4d7h
vault-server      Active   30m

This deployment builds Kubernetes server pods on dedicated nodes in the vault-server namespace. Verify the nodes:

$ kubectl get pods -o wide -n vault-server
NAME                                                        READY   STATUS      RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
boot-vault-sg-01f5e0c0d6458ed88-5hrf8                       0/1     Completed   0          25m   10.0.32.188   ip-10-0-60-134.eu-north-1.compute.internal   <none>           <none>
boot-vault-sg-01f5e0c0d6458ed88-dfwkp                       0/1     Error       0          27m   10.0.59.145   ip-10-0-60-134.eu-north-1.compute.internal   <none>           <none>
certificate-vault-sg-01f5e0c0d6458ed88-24h6n                0/1     Completed   0          29m   10.0.30.86    ip-10-0-16-209.eu-north-1.compute.internal   <none>           <none>
vault-sg-01f5e0c0d6458ed88-0                                1/1     Running     0          26m   10.0.12.215   ip-10-0-6-233.eu-north-1.compute.internal    <none>           <none>
vault-sg-01f5e0c0d6458ed88-1                                1/1     Running     0          26m   10.0.64.124   ip-10-0-86-92.eu-north-1.compute.internal    <none>           <none>
vault-sg-01f5e0c0d6458ed88-2                                1/1     Running     0          26m   10.0.55.38    ip-10-0-60-134.eu-north-1.compute.internal   <none>           <none>
vault-sg-01f5e0c0d6458ed88-agent-injector-b76f744b6-6pjp9   1/1     Running     0          26m   10.0.86.51    ip-10-0-86-92.eu-north-1.compute.internal    <none>           <none>

Kubernetes services

Check that this deployment creates at least the following seven services:

$ kubectl get svc -n vault-server
NAME                                            TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)             AGE
vault-sg-01f5e0c0d6458ed88                      ClusterIP      172.20.238.238   <none>                                                                     8200/TCP,8201/TCP   27m
vault-sg-01f5e0c0d6458ed88-active               ClusterIP      172.20.9.90      <none>                                                                     8200/TCP,8201/TCP   27m
vault-sg-01f5e0c0d6458ed88-agent-injector-svc   ClusterIP      172.20.235.220   <none>                                                                     443/TCP             27m
vault-sg-01f5e0c0d6458ed88-internal             ClusterIP      None             <none>                                                                     8200/TCP,8201/TCP   27m
vault-sg-01f5e0c0d6458ed88-standby              ClusterIP      172.20.169.201   <none>                                                                     8200/TCP,8201/TCP   27m
vault-sg-01f5e0c0d6458ed88-ui                   LoadBalancer   172.20.59.230    a4b85f61XXXXXXXXX01d-1681023831.eu-north-1.elb.amazonaws.com   443:32436/TCP       27m

Vault configuration

Verify that Vault is configured for high availability (HA):

$ kubectl exec -ti -n vault-server vault-sg-01f5e0c0d6458ed88-0 -- /bin/sh
/ $ export VAULT_SKIP_VERIFY=true
/ $ vault login s.JWF4aXXXXXXXXXXXXX9cgZ
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.JWF4aKPXXXXXXXXXXXojl9cgZ
token_accessor       xceUAbCKAAS86OKupBK2Bhlr
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
/ $ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.5.3
Cluster Name             vault-cluster-9abfeb1c
Cluster ID               f04374ee-3ebe-4e0f-fa50-892d48421e70
HA Enabled               true
HA Cluster               https://vault-sg-01f5e0c0d6458ed88-0.vault-sg-01f5e0c0d6458ed88-internal:8201
HA Mode                  active
Raft Committed Index     119
Raft Applied Index       119

In the previous output, note values for HA Enabled, HA Cluster, and HA Mode configuration entries.

Vault UI Secure Socket Layer (SSL) certificate

Verify the DNS endpoint of the deployment, and check for the SSL certificate:

$ openssl s_client -connect  lonconsul.gargana.myinstance.com:443
CONNECTED(00000007)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = lonconsul.gargana.myinstance.com
verify return:1
---
Certificate chain
0 s:CN = lonconsul.gargana.myinstance.com
i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
i:C = US, O = Amazon, CN = Amazon Root CA 1
2 s:C = US, O = Amazon, CN = Amazon Root CA 1
i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFrDCCBJSgAwIBAgIQA+/KZ0HG5aT6xAZLv0NjlDANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg
Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0yMDEwMTUwMDAwMDBaFw0yMTExMTMy
MzU5NTlaMCsxKTAnBgNVBAMTIGxvbmNvbnN1bC5nYXJnYW5hLm15aW5zdGFuY2Uu
....
....
....
....
NOEMjRe008xraTpAzfSjr2fupjltJB6lXehPe5sJaWPJ0mX3OBt4VyfrO6MYdmpy
iGLhMXM357+CN75aMv1BD4pVA+a75dhvcUOfZCni4guQ+7wbbwONrKdwtg9FudWf
XzvTdg1Q8VPfuQWUJb8tmITseg+8KDTyUn1u2SiNWHj17hBTSBTjkVt97id0BtZ/
UYrBWVldmJw0pJ6XYgQc6pBg6A86390sGkRzOfhYkT8AIbKNKSwtCRV0aBY2Nb4+
i81nP0KKeSvWcRf4/Gj+WA==
-----END CERTIFICATE-----
subject=CN = lonconsul.gargana.myinstance.com

issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

---

Vault Raft-peer election

Check the Raft-peer election status:

$ kubectl exec -ti -n vault-server vault-sg-01f5e0c0d6458ed88-0 -- /bin/sh
/ $ vault operator raft list-peers
Node                            Address                                                                  State       Voter
----                            -------                                                                  -----       -----
vault-sg-01f5e0c0d6458ed88-0    vault-sg-01f5e0c0d6458ed88-0.vault-sg-01f5e0c0d6458ed88-internal:8201    leader      true
vault-sg-01f5e0c0d6458ed88-1    vault-sg-01f5e0c0d6458ed88-1.vault-sg-01f5e0c0d6458ed88-internal:8201    follower    true
vault-sg-01f5e0c0d6458ed88-2    vault-sg-01f5e0c0d6458ed88-2.vault-sg-01f5e0c0d6458ed88-internal:8201    follower    true

Best practices for using Vault on AWS

The following best practices are enabled by default for this Quick Start:

  • Enabled AWS KMS auto-unseal: This uses AWS KMS to store and encrypt Vault’s unseal keys. For more information, see Auto-unseal using AWS KMS.

  • Enable cluster HA: This helps to ensure that Vault is set up for fault tolerance. For more information, see Vault HA Cluster with Integrated Storage.

  • Enable Raft storage for HA: This sets up the Raft consensus protocol as Vault’s storage backend. For more information, see Use Integrated Storage for HA Coordination.

  • Enable Vault audit to Amazon CloudWatch: This lets you troubleshoot audit logs. For more information, see Enabling audit devices.

  • Enable SSL at the Vault UI endpoint: This helps to secure the Vault UI endpoint with an SSL certificate. For more information, see Vault UI.

Other useful information

FAQ

Q. I encountered a CREATE_FAILED error when I launched the Quick Start.

A. If AWS CloudFormation fails to create the stack, we recommend that you relaunch the template with Rollback on failure set to Disabled. (This setting is under Advanced in the AWS CloudFormation console, Options page.) With this setting, the stack’s state is retained and the instance is left running, so you can troubleshoot the issue. (For Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService and C:\cfn\log.)

When you set Rollback on failure to Disabled, you continue to incur AWS charges for this stack. Please make sure to delete the stack when you finish troubleshooting.

For more information, see Troubleshooting AWS CloudFormation.

Q. I encountered a size limitation error when I deployed the AWS CloudFormation templates.

A. We recommend that you launch the Quick Start templates from the links in this guide or from another S3 bucket. If you deploy the templates from a local copy on your computer or from a location other than an S3 bucket, you might encounter template size limitations. For more information, see AWS CloudFormation quotas.

Troubleshooting

For troubleshooting Amazon EKS, see Amazon EKS on the AWS Cloud.

For troubleshooting HashiCorp Vault on Amazon EKS, see Monitoring & Troubleshooting.

Parameter reference

Deploy into a new VPC and new Amazon EKS cluster

The full list of parameters for this entrypoint are documented in Amazon EKS on the AWS Cloud.

Deploy into a new Amazon EKS cluster in an existing VPC

The full list of parameters for this entrypoint are documented in Amazon EKS on the AWS Cloud.

Launch into an existing Amazon EKS cluster

Table 1. Amazon EKS cluster
Parameter label (name) Default value Description

Amazon EKS cluster name (ClusterName)

Requires input

Amazon EKS cluster.

Table 2. Network configuration
Parameter label (name) Default value Description

Permitted IP range for Vault UI (AccessCIDR)

127.0.0.1/32

CIDR IP range that is permitted to access Vault. Note that a value of 0.0.0.0/0 allows access from any IP address.

First subnet ID for Auto Scaling group (PrivateSubnet1ID)

Requires input

ID of the private subnet in Availability Zone 1 of your existing VPC (e.g., subnet-fe9a8b32).

Second subnet ID for Auto Scaling group (PrivateSubnet2ID)

Requires input

ID of the private subnet in Availability Zone 2 of your existing VPC (e.g., subnet-fe9a8b32).

Third subnet ID for Auto Scaling group (PrivateSubnet3ID)

Requires input

ID of the private subnet in Availability Zone 3 of your existing VPC (e.g., subnet-fe9a8b32).

HTTP proxy (HttpProxy)

Blank string

HTTP proxy server address e.g. "http://10.32.20.10:3128/."

EC2 key pair (KeyPairName)

Requires input

Name of an existing key pair, which helps to secure your connection to instances after they launch.

Table 3. HashiCorp Vault Configuration
Parameter label (name) Default value Description

HashiCorp Vault version (VaultVersion)

1.6.0

Version of Vault to use.

HashiCorp Vault deployment size (VaultDeploymentSize)

small

Deployment size of dedicated HashiCorp Vault nodes. (see https://learn.hashicorp.com/tutorials/vault/kubernetes-reference-architecture).

Kubernetes namespace for Vault (KubernetesNameSpace)

vault-server

Kubernetes namespace for HashiCorp Vault server.

Internal or external load balancer? (LoadBalancerType)

Internal

Choose whether the load balancer for HashiCorp Vault is internal or external to the VPC.

Load balancer DNS name (DomainName)

Blank string

Fully qualified DNS name for the "vault-ui" service load balancer. If you don’t provide a value for "ACM SSL certificate ARN", use "HostedZoneID."

Route 53–hosted zone ID (HostedZoneID)

Blank string

Route 53–hosted zone ID of the domain name. If you don’t provide a value for "ACMSSLCertificateArn,"" the Quick Start creates an ACM certificate for you using "HostedZoneID" in conjunction with the domain name.

ACM SSL certificate ARN (ACMSSLCertificateArn)

Blank string

ARN of the load balancer’s ACM SSL certificate. If you don’t provide values for "Domain name" and "HostedZoneID," provide a value for "ACM SSL certificate ARN."

HashiCorp Vault server nodes (Nodes)

3

Number of dedicated Vault server nodes and pods.

Table 4. Quick Start Configuration
Parameter label (name) Default value Description

Quick Start S3 bucket name (QSS3BucketName)

aws-quickstart

S3 bucket name for the Quick Start assets. Quick Start bucket name can include numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen (-).

Quick Start S3 bucket Region (QSS3BucketRegion)

us-east-1

AWS Region where the Quick Start S3 bucket (QSS3BucketName) is hosted. When using your own bucket, you must specify this value.

Quick Start S3 object key prefix (QSS3KeyPrefix)

quickstart-eks-hashicorp-vault/

S3 key prefix for the Quick Start assets. Quick Start key prefix can include numbers, lowercase letters, uppercase letters, hyphens (-), and forward slash (/).

Send us feedback

To post feedback, submit feature ideas, or report bugs, use the Issues section of the GitHub repository for this Quick Start. If you want to submit code, review the Quick Start Contributor’s Guide.

Quick Start reference deployments

GitHub repository

See the GitHub repository to download the templates and scripts for this Quick Start, post comments, and share customizations with others.


Notices

This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for specific language governing permissions and limitations.