vExpert

SSL Validation in Python Virtual Environments for HashiCorp Vault lookups in Ansible AWX

When using a Python virtual environment (venv) and HVAC to perform HashiCorp Vault lookups from Ansible, you may run in to SSL verification errors.

Michael Poore

• 3 min read
SSL Validation in Python Virtual Environments for HashiCorp Vault lookups in Ansible AWX
Photo by Kelly Sikkema / Unsplash

That title is a bit of a mouthful! But necessarily so.

I've been working with Mark Brookfield to get Ansible AWX pulling in host credentials from HashiCorp Vault in my homelab. Mark wrote two posts about this process back in lockdown 2020 and I followed them at the time to reproduce his solution. Here they are:

Recently I had cause to redeploy my AWX server and needed to re-create the integration to Vault from scratch. The two posts linked above are still very relevant but while following the first of them I hit upon something that wasn't covered that I had to solve previously and forgot to write about. So I thought I'd do it now...

Wherever possible I like to use signed SSL certificates and not rely on self-signed, untrusted certificates. Besides being a good practice to get in to, most of my customers use signed SSL certificates and one of the reasons for running a homelab is to understand the challenges that they face day-to-day.Without using any additional steps the process in the first post above will require you to either:

  1. Skip SSL certificate verification of the certificate being used by Vault; or
  2. Configure Vault to use a certificate signed by a trusted authority

If neither approach is followed then you end up an error similar to the following when you try to run a playbook that uses HVAC to pull credentials from Vault:

FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'requests.exceptions.SSLError'>, original message: HTTPSConnectionPool(host='vault.v12n.io', port=8200): Max retries exceeded with url: /v1/auth/approle/login (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))"}

As I said before, I try to emulate the same sorts of configurations and challenges that my own customers face and for most of them option 1 is a complete no-go. The second option can be adopted in more than one way as we could either buy Vault a certificate that is signed by a known certificate authority, or configure the Python virtual environment for AWX to trust an internal certificate authority. In case you hadn't already guessed, this post is about the second option.

Following Mark's article, when you use pip to install psutil, hvac and hvac[parser] some other dependences are installed / configured. One of these is certifi, which is a small package that provides Mozilla's CA bundle to be used with HTTP(S) requests. The error above is caused when certifi doesn't trust the certificate supplied by Vault. Fixing it is pretty simple, we just have to add our trusted root and intermediate CA certificates to the CA bundle. Let's go!

💡
I created my virtual environment in the directory /opt/my-envs/v12n so the path below reflects that. You will need to account for any differences in where you created the virtual environment.

Use your favourite editor (vi, vim, nano, emacs etc.) to edit the following file:

/opt/my-envs/v12n/lib/python3.6/site-packages/certifi/cacert.pem

Simply go to the end of the file and paste in the base64 encoded root and intermediate CA certificates that you need and save the changes.

Root and Intermediate certificates added to the end of the cacert.pem file

Now when you run the docker cp commands per Mark's article the virtual environment will "trust" your Root and Intermediate CAs.