Preparing Vault for integration with vRA

In my previous post I set out a high-level solution for integrating VMware vRealize Automation and HashiCorp Vault together to ensure that newly provisioned workloads get secure, unique passwords. Now it's time to get down in to some of the details. In this post I'm focussing on Vault.

As I called out before, I'm starting with an existing Vault server and not covering off design or installation. And although Vault has a UI, I'm not going to be using it. Everything will be done via the CLI in this post and some will then be converted in to API calls in the next post for vRO to use.

No more dallying, let's get started...

Enabling a K/V Secrets Engine

Secrets Engines in Vault are used to store, encrypt or generate data. Vault includes a number of secrets engines for common use-cases that can be enabled on paths to create what looks like a virtual filesystem. To store username / password pairings for local accounts on my workloads, the Key/Value engine is the most logical fit. It's intended for storing arbitrary data and can be enabled in two versions (V1 and V2), with the most important difference between the two being that V2 keeps a history of the secrets (and other metadata), useful perhaps if you want to see the previous passwords that have been used for an account.

As everything is path-based in Vault, you have to choose a mount point for the secrets engine and, especially with K/V, have a plan for structuring the data under that.

I chose to mount the K/V engine at the path "hosts/". Under that I planned to have a new sub-level for each new workload. Under each workload there would then be a secret called "credentials". That secret's data would include key-value pairs for each account to be secured. Conceptually it looks like this:

The only bit that needs to be created by hand is the engine itself. Assuming that you have the necessary access to Vault then the engine is enabled using the following command:

If you're new to Vault then the only two bits of that to be concerned about are:

1. -path=hosts (this tells Vault where I want the Secrets Engine mounted)
2. kv-v2 (this tells Vault that I want a Key/Value engine running version 2)

By default virtually no-one will have access to "hosts". We'll need to change that later.

Installing and Enabling a Password Generation Plugin

There are a number of options that I could have selected for password generation. I could execute it on each workload and "push" the credentials in to Vault. Or I could have vRO generate passwords and place them in Vault. There are pros and cons to each. The option I selected to go with was to use a plugin for Vault.

Seth Vargo is an engineer at Google Cloud but used to work for HashiCorp. He has created and maintains a plugin for Vault that is capable of creating high entropy passwords and passphrases. Sounds perfect!

sethvargo/vault-secrets-gen

A Vault secrets plugin for generating high entropy passwords and passphrases. - sethvargo/vault-secrets-gen

GitHubsethvargo

Instructions are provided in the repo's README to install and use the plugin so I won't repeat them here. I opted to mount it at the default path of "gen".

Turning On AppRole Auth

There are a plethora of authentication methods available and built-in to Vault: LDAP, username & password, GitHub, Kerberos and Okta to name but a few. AppRole is another and seems like a perfect fit for this use-case.

AppRole is intended to be used by machines or apps and allows defined roles to be assigned. In addition, there are a number of constraints that can be set on the role to control when and where it can be used as well as controlling what constraints are applied to tokens issued. Hopefully this will become apparent as I step through the configuration. First of course we have to enable the method and define the path that we want to mount it at in Vault.

That mounts the AppRole auth method at "/auth/approle". (Optionally I could have specified a different path by adding -path=<new path>.) We'll configure a role for vRO (vRA) in a moment. Before doing that though we have to create a policy that will be assigned.

Creating a Policy for vRO

Policies control what paths in Vault are accessible and what operations can be executed against them. Policies are mapped to authentications as part of configuration of the authentication mechanism. An example of how this works with LDAP authentication would help.

1. Authentication is made to Vault by a user using LDAP credentials.
2. Vault verifies those credentials with LDAP and obtains information about the user, including group memberships.
3. The LDAP configuration within Vault is used to map the user to one or more policies. A unique token is created and the matching policies are attached to it.
4. Vault returns the token to the user.

So, what's in the policy and how it's "assigned" to vRO will dictate what operations vRO can execute in Vault. Clearly we want to grant vRO only enough access to achieve our goals. In summary that means that vRO must be able to:

• Use the vault-secrets-gen plugin to create new passwords
• Create and manage policies
• Create and manage secrets in the path "hosts"
• Manage authentication methods (specifically creating new AppRoles)

It may not sound like much but some of these are fairly sensitive operations. The challenge therefore is creating a policy that only permits limited scope for these operations. As a first pass, I started with this:

This got me started, but it's still too broad. The policy path allows vRO to manage any policies (including its own). The auth path is also too broad. All we need is to allow vRO to grant access to new workloads and tidy up after itself. To address these issues I made the paths more specific and reduced the capabilities to only what was needed:

Now that we have identified a policy configuration, it needs to be created in Vault. The command itself is simple enough but first the policy configuration from above needs to be written to a text file, for example vrealize.hcl.

This takes the configuration defined above and creates it in a policy called "vrealize".

Creating a Role for vRO

To authenticate using AppRoles, there are two pieces of information needed. The first is the ID of the role and the second is a Secret ID that is associated with the role. Provide both of those and (if validated) Vault returns a token to be used as if you had authenticated using a username and password. The following example is copied straight from the Vault documentation:

You can think of the Role ID as a username and the Secret ID as a password if it helps, but that's not entirely accurate. An AppRole has one, and only one, Role ID. By default it's a unique guid but it can be changed to a more predictable value if required. In contrast, there could be one or many Secret IDs associated with a role. Each Secret ID inherits some properties from the role it is associated with and can have some properties associated with it at creation time.

For the role I'm creating for vRO, I want to restrict its use so that only vRO uses it. Also, I want to make sure that any issued tokens are short-lived so that if they do "escape" then they can't be used for much elsewhere. Vault provides a means to control much of this via options configured on the role. Let's cut to the chase, here's my vrealize role creation command:

Some of these are defaults, but I've explicitly set them anyway. The purpose of each setting is explained in the Vault API documentation but the important ones that I wanted to call out are:

• secret_id_bound_cidrs - this restricts the use of Secret IDs (and hence authentication) to a specific address, namely vRA / vRO
• secret_id_num_uses - allows unlimited use of the Secret ID (in the future I might implement a mechanism to rotate the Secret ID for extra security)
• token_max_ttl - Limits the life of the authentication token to 1 minute (again I might shorten this in the future)
• token_policies - this is where the AppRole and Policy are linked together, whenever vRO authenticates to Vault it will be assigned the "vrealize" Policy
• token_no_default_policy - all authenticating users are automatically assigned the built-in "default" policy, this setting prevents that
• token_num_uses - limits the number of times that the issued token can be used (I may fine-tune this number further)

Having created the role, you then need to extract the Role ID from Vault:

And create a Secret ID:

Both of these I will then store in vRO for it to authenticate with Vault.

Vault Operations for vRO

Up to this point everything that I've done has been a one-time operation. I don't have to repeat any of the above steps for my integration to work. Of course it's good practice to rotate the Secret ID for vRO regularly - I may even automate it!

In this section however I will have to take the steps outlined as CLI commands below and turn them in to API calls for vRO to use them. I'm not going to explain how I did this in detail now, the Vault documentation contains API examples for most things.

If you recall from my previous post, my solution overview set out the highlevel process that I will automate through vRO. Here it is again:

Now I need to go a little deeper and call out the specific operations needed and the order they'll be needed in.

1. The first order of business is authentication, we've seen it before but for completeness here it is again:

2. With the Token extracted from that result we can now create a new Policy specifically for a single workload. This policy (named vro-<hostname>) permits read access to the credentials secret for a single host:

3. Next we need to create a role. This is similar to the vRO role from earlier but with a couple of small differences:

• Any Secret IDs associated with this role can be used to authenticate only once, i.e. Secret IDs can not be re-used
• Tokens issued are valid for only 10 seconds
• Tokens can be used for only 5 operations (I may restrict that further)
• Of course, the role is linked to the Policy created above

That's enough to allow a single session to retrieve some credentials. The beauty part of this is that if someone gets the timing write and uses the Role ID and Secret ID to retrieve the passwords then the workload itself won't get them and the stolen data is useless.

4. The role will have been given a random ID. Nice and secure, but I haven't quite worked out a reliable method to get complex and unique Role IDs to individual workloads yet. Instead, I have opted to update the Role ID to match the hostname:

5. Next, we need to create a Secret ID. As with the Role ID I haven't been able to make it unique for individual workloads yet so I've set a custom Secret ID that matches the hostname:

6. Now that access is sorted, the next stage is to create high-entropy passwords for each workload. The following command does that:

7. Take the password and place it in a secret for the workload to read:

That's it done for the vRO side of things.

Vault Operations for Workloads

For the individual workloads, assuming that they have Vault installed (or I use CURL for API access to Vault), then we only need to execute two commands:

1. Authenticate to Vault:

2. Extract the password and apply it:

What's next?

As I mentioned, some of those CLI commands need to be converted in to API calls for vRA / vRO. I also need to create a cloudConfig script to be executed on every workload. All of that is to come in the next post.