How to Configure AWS KMS for Vault Auto-Unseal

When I started using HashiCorp Vault in anger in my lab last year, I did what a lot of people probably did:

1. Start with a single Vault host using file-based storage
2. Create a Consul cluster for storage, still using a single Vault host
3. Add another Vault host and stick a load-balancer in front of them

What I never changed throughout all of those steps was the mechanism used to seal and unseal Vault. By default, when you first initialise Vault a series of 5 unseal keys are created. These should be given to 5 different people and any 3 of those keys can be used to unseal Vault (think of it like undoing the deadbolts on your door).

Until it's unsealed, no-one can read or write any secrets and load-balancer health checks will fail. Yes it's secure, but if you path a Vault server or restart the service then it comes back up sealed. Manual intervention is required to unseal it.

In my lab it's not a huge problem, and probably why I did nothing about it for so long. I have all 5 unseal keys in my password safe and I'm the one who uses Vault. But, if I create a process to automate host patching that relies on being able to get credentials out of Vault then I've got myself a bit of a conundrum.

Luckily there are some options to configure auto-unsealing using an AWS Key Management Server that works very nicely with Vault. I decided to get it setup. Here's how...

Instructions

This guide assumes that you have an AWS account and have the AWS CLI tool installed. You'll also need access / secret keys for an AWS account that has sufficient privileges to perform the actions below.

Planning

The first step is to do a little planning. I selected the values for the following configuration items ahead of time:

Create a new IAM group

1. Open a new terminal
2. Execute the following command to create a new IAM group:
   aws iam create-group --group-name {{IAM Group Name}} --path {{IAM Path}}
3. Verify that a success response is received that includes the ARN of the group created

Create a new IAM policy

1. In the same terminal from before, create a JSON file with the following content (adjust the region in the Resource line to match the table above):
   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"], "Resource": "arn:aws:kms:eu-west-2::key/*" } ] }
2. Optionally, the resource used can be refined or opened up to limit access to specific resources.
3. Execute the following command to create a new IAM policy:
   aws iam create-policy --policy-name {{IAM Policy Name}} --path {{IAM Path}} --policy-document file://policy.json
4. Verify that a success response is received that includes the ARN of the policy created

Attach the IAM policy to the group

1. Copy the policy ARN from the output of the previous command
2. Paste it in to the following command to attach the policy to the group (replace the example ARN shown):
   aws iam attach-group-policy --group-name {{IAM Group Name}} --policy-arn arn:aws:iam::nnnn:policy/v12n/production/vault/VaultKMSUnsealPolicy
3. The above command doesnt provide any output but success can be verified using the following:
   aws iam list-attached-group-policies --group-name {{IAM Group Name}}

Create a new IAM user and access keys

1. Execute the following command to create a new IAM user:
   aws iam create-user --user-name {{IAM User Name}} --path {{IAM Path}}
2. Verify that a success response is received that includes the ARN of the user created
3. Execute the following command to create an access key for the user:
   aws iam create-access-key --user-name {{IAM User Name}}
4. If successful, make sure that the AccessKeyId and SecretAccessKey are copied and stored in a secure place!

Add the IAM user to the group

1. Execute the following command to add the user created above to the group created earlier:
   aws iam add-user-to-group --user-name {{IAM User Name}} --group-name {{IAM Group Name}}
2. The above command doesnt provide any output but success can be verified using the following:
   aws iam list-groups-for-user --user-name {{IAM User Name}}

Create a KMS key for Vault

1. Execute the following command to create new Symmetric Key:
   aws kms create-key --description "v12n Production Vault Unseal Key"
2. Verify that a success response is received that includes the ARN and KeyId of the key created
3. Record the KeyId in a secure place!
4. (Optional) Create an alias for the key using the following command:
   aws kms create-alias --alias-name alias/{{KMS Key Alias}} --target-key-id {{KMS KeyId}}

Next Steps

All of the above can be completed through the AWS console as well, but who doesn't love a command line?

In my next post I'll cover migration of my Vault server across to use AWS auto-unseal.