Creating unique passwords in Vault for new vRA workloads

In any IT environment the use of weak passwords, commonly used passwords or the re-use of the same passwords again and again is not advisable. In my own lab I have in the past been guilty of all of these security sins. Maybe it's not so bad because I'm the only human user of my lab, but there's an important principle involved and an opportunity exists for me to automate some security in to my lab and share my experiences for others to build on.

In the multi-cloud world we want to be able to provision (create) infrastructure and application platforms rapidly and reliably. Often that means taking a pre-defined template, making a copy and customising it for our purposes. All of this should be done on-demand for whomever needs to consume the infrastructure or platform using a mechanism that is managed and operated for that purpose. This is where vRealize Automation fits in.

VMware vRealize Automation

vRA is VMware's infrastructure automation platform, available as either (Software-as-a-Service) SaaS or to be deployed in your own environment. It provides a number of capabilities to define cloud templates and then build and deploy from them across multiple clouds. With it we can start to standardize service delivery, but we can also extend and orchestrate that delivery in to other processes and technologies. For example, HashiCorp Vault.

HashiCorp Vault

If vRA is a master of automation and orchestration, Vault is a master of secrets and data encryption. A "secret" is a piece of data that you want to keep secure and control access to. It could be anything, including a set of credentials. Of course there are other tools out there that manage credentials, but the quality of Vault's API and the extensibility potential of the product as a whole are compelling.

My fellow automator of things, Mark Brookfield, presented about the integration of Vault and vRA 7 at a VMUG event some time ago and we've been talking about it ever since. Add to that a number of close colleagues who have joined HashiCorp in the last 18 months and you have the beginnings of my motivation for selecting these two tools for the job.

Of course, "because I can" features heavily too, but my main driver in tackling this was to further the treatment of my lab as a production environment (and learn something along the way). Let's start with a problem statement and define the issue I've tried to solve a bit more clearly.

The Problem to Solve

Imagine if we have a simple VM template created that is used to create new workloads from. To build that template we have to install the guest OS and make a few customisations (perhaps) to prepare for it to be cloned. As part of installing the guest OS a password is required to be set for the "root" or "Administrator" user. In my experience this password is often:

1. Not very complex
2. Re-used in other templates or when templates are updated
3. Known to multiple individuals
4. Stored in a readily accessible locations
5. Stored in another system

The net result is that if nothing else is changed then a simple clone of that template results in an insecure workload that has a known set of credentials, aka a security vulnerability.

Even if we implement processes and safeguards to mitigate against the first four of those issues above, the last one is tricky and we're still cloning from a template that contains a fixed set of credentials. In fact, in the past it was the case that to apply a unique set of credentials to every workload using some form of automation would require the initial set of credentials to be known by that automation system. In the past I have seen and implemented several platforms where that was the case and credentials were either delivered by email, stored in some sort of safe or both.

Even if we exercise great care, those new credentials are vulnerable in a number of places. A much better situation would be if the template credentials were unknown and not even needed and if the new credentials could be securely created, securely stored and passed through as few hands as possible.

If those credentials are secured away and access to them is audited then we have the makings of a secure process. This is what I'm going to target solving using vRealize Automation and Vault.

In summary, I want to achieve the following:

• The ability to clone an image / template without having the template credentials known, stored or required
• Each and every workload to have unique credentials that are set at deployment time
• Credentials for every new workload to be stored securely and accessible only to authorised users

(The panacea would be credentials that are short-lived and rotated on a regular basis as well as being dynamically set at deployment time. However, let's start by walking first!)

Solution Overview

I'm not going to cover the design or initial deployment of either vRealize Automation or Vault. I'm starting from the position of having both of these deployed and working.

At a high level, what I will implement looks like this:

1. The provisioning request for a new workload is made in vRA and is based on a simple blueprint. The blueprint will clone a single VM in a vSphere endpoint.
2. Prior to the clone operation commencing an extensibility operation is triggered in vRA that executes a workflow. That workflow connects to Vault and creates a Role, a Policy, a Secret ID and defines the path where the credentials will be placed.
3. vRA instructs vSphere to clone the VM and additionally supplies the VM with a set of cloudConfig instructions.
4. Before the new workload is powered on vRA configures a set of unique credentials in the Vault path for the new workload using another workflow.
5. Once the new workload is powered on the cloudConfig instructions are executed by cloud-init (which is already installed in the template). These instructions include the Role ID, Secret ID and address to be used to access Vault and retrieve / apply the credentials.

Key to how this all works with vRealize Automation (v8.x and Cloud) is the support for cloud-init that is built-in. Cloud-init has become the standard for customising cloud templates and instances. Most public cloud providers support it in their templates and it can very easily be installed in a vSphere template too. One of the key components of my solution was to take an otherwise vanilla template and make sure it could communicate with Vault without anything being hard-wired in to the template itself and without guest OS credentials being required. The presence of cloud-init facilitates that.

(If you're interested in how I build my templates, I have another post and a video that covers it - Integrating VMware Code Stream and Packer.)

The stage is now set and the show begins in the next post where I will cover off the following Vault topics:

• Creating a Key/Value secrets engine for storing new host credentials
• Enabling AppRole authentication for vRA to update Vault
• The required operations needed for vRA to update Vault