HashiCorp Vault KV v2 Lookups from Ansible AWX

I've been spending a little time with Ansible in my homelab of late, as you may have gathered from my previous post. Like Mark, whose posts I reference as a guide / inspiration, I have placed all of my credentials in to a secrets manager (HashiCorp Vault).

The idea behind this is that each (virtual) host in my homelab should have a unique and complex set of credentials that are required for root or administrator access. These credentials should be stored as a single source of truth for each host and accessed via some form of role-based policy. But this post isn't about why use HashiCorp Vault, it's about how we can access those credentials programmatically from Ansible AWX without having to duplicate them in another system.

Mark Brookfield's post Enabling HashiCorp Vault Lookups in Ansible AWX – Part 2 covers the creation of a new credential type in AWX and how it can be used to reference credentials in Vault. If you're familiar with Vault however, you'll realise that his solution is using a KV v1 Secrets Engine. Nothing wrong with that at all. There are reasons that you might want to use a KV v2 Secrets Engine though and there are some subtle differences with how to reference one. The plugin documentation does contain some examples that put me on the right path after a bit of trial and error but to save anyone else having to work it out I thought I'd document it here.

Firstly, let's look at how the credentials are structured in Vault.

The KV Secrets Engine has been configured with the path credentials. Under that there is a level called computers followed by a level called production. Under production we have a separate level for each host. In this example I have called them host01 and host02 but these match my hostnames in reality. Each host has two secrets under it, one called admin and one called user. They are explained as follows:

Admin - this secret contains the credentials for the built-in administrative or super user for the host. So for Linux this corresponds to the root account and for Windows it's for the local Administrator account.

User - this secret contains the credentials for a relatively unprivileged local user for the host. I add this account at build time in my templates. For Linux in particular this is important as I also disable root SSH access by default.

Each one of the actual secrets (admin or user) contains two key-value pairs. One is called username and the other is password. Each secret is versioned starting at V1 when it is first created. In the example above I have simulated a version increment when the root password has been changed after 7 days. For Ansible I will usually always want the current password, i.e. the highest or latest version of the secret. Fortunately this is the default behaviour anyway when using the hashi_vault plugin.

Compared to the KV V1 lookup for Ansible, we need to make a couple of minor changes to what we request from Vault. Let's say that we had used the same structure above but as a KV V1 engine, of course we wouldn't have versions but the lookup would be as follows:

ansible_become_password: "{{ lookup('hashi_vault', 'secret=credentials/computers/production/{{ inventory_hostname }}/admin:password')}}"

This would return the root password for use by Ansible as the "become password". However, with KV V2 there are two minor challenges that we must take account of.

1. Per the Vault documentation we must insert "data/" between the Secrets Engine path and the rest of the secret path.
2. A KV V2 secret contains metadata pertaining to the version etc. so we must ignore that and focus on the value that we actually want.

The first is easy, instead of the secret path being, for example:

credentials/computers/production/host01/admin

We should use the following:

credentials/data/computers/production/host01/admin

The second challenge can be worked out with some knowledge of how V2 secrets are structured and a little bit of experimentation. Instead of asking for the password field directly we have to extract the secret data first and then select our desired field (password) from that. This ends up with a lookup as follows:

ansible_become_password: "{{ lookup('hashi_vault', 'secret=credentials/data/computers/production/{{ inventory_hostname }}/admin:data')['password'] }}"

Which works perfectly!