Working with Fernet Decryption Keys in Red Hat OpenStack 12

Working with Fernet Decryption Keys in Red Hat OpenStack 12

With Red Hat OpenStack 12, Fernet keys are now the default. In this guide, I’d like to show you how to not only rotate your decryption keys on the controller, but how to debug any issues you may encounter. This is really meant to be a guide to managing fernet decryption tokens as a whole.


Let Me Count The Ways

I’ve posted previously about how to rotate fernet decryption keys using Ansible. As that post is pretty thorough, I’ll move ahead with what is essentially the “supported” method in RHOSP 12. This method uses Mistral, the OpenStack workflow engine. Official documentation on this method exists here – but I’d like to go into further depth on this topic, as I believe it’s warranted here.

Lay of the Land

Let’s start out with the fact that Fernet tokens are enabled by default in RHOSP 12. This means that, if you rely on UUID tokens for some reason, you’d have to manually change your token provider in Keyston back to it. I’d highly recommend against this, unless you have a specific reason for it. Fernet offers many advantages, and this looks to be quite stable and usable in RHOSP 12.

I’d also like to mention that Keystone, like most of the other services, is containerized in RHOSP 12. This means that you’ll need to change the way you manage it for certain operations. I’ll go through these and give you color where needed, but I’m sure most people will be quite familiar with this methods used herein by now.

How many keys?

Let’s have a look inside the docker container running keystone, to see how many keys we have on the server by default:

We can see from this listing that there are two keys installed by default. This doesn’t, however, show us what the default number of keys is.

We can find the default number of keys but issuing the following command on one of our controllers:

This tells us that the default number of active keys is 5, as defined by max_active_keys in /etc/keystone/keystone.conf. While we now have two active keys, one primary and one staged, rotating keys will give us up to 5 active keys before they’re recycled.

Validating your currently active key

I like to validate my key before messing with it. Here’s how you do that:

As the highest key is your active key, let’s get a dump of it from the docker container running keystone:

Next, let’s get keystone to issue a token for us from the director:

Now, let’s turn to python to decrypt this token, using our key obtained from the controller – interactively.

Let’s first import the modules we need to get started:

Let’s put the server key into a variable called ‘skey’, define the Fernet key with it and put the token keystone issued us into the variable ‘token’. Note, I’ve put an ‘=’ at the end of the token, as this is base64 encoded and the output of the openstack client strips that out.

Now we’ll decrypt that token using the key with the decrypt() method:

If this returns, you’re good to go – if not, you’ll see something like this:

Finally, we can pull out the actual UUID token value like so:

Rotating the keys

Ok, let’s get to rotating using the the openstack client. We’ll use the openstack workflow execution create to run the RHOSP 12 provided workflow for rotating your tokens:

The important fields here are State and ID. Let’s use the ID field to look at the status of the workflow:

As you can see, our status here is RUNNING. Using the ID, let’s have a look at the mistral task list:

We can now see that all but one of the tasks has completed – deploy_keys. If we’d like to go further down the stack and look at the tasks and their results, we can do so using the task ID’s. For instance, let’s take the first task (rotate_keys) and use its ID to see what the results were:

We get actual output of the keys and their values. How cool is that?

We can use this method to debug failures for, not only keystone key rotation, but and mistral task we run in RHOSP. This is a useful powerful tool. I encourage you to look at some of these other tasks, the output is quite large on some — giving you a good amount of detain into the inner-workings of this workflow.

After a short while, we can now see that the workflow has succeeded:

Let’s go into the controller and have a look at the keystone container:

This is great – we now have three keys on our controller. If you need more info on ferenet decryption keys and their meanings, please check out my previous post on rotating keys with Ansible.

Test and Verify

So, we’re looking good here in terms of rotation. But do we know if these keys will decrypt our tokens correctly? Using the method outlined previously in this document, you can test this by trying to decrypt our previous token with key 1, as this was our previously primary token. It should still be able to decrypt any tokens in the wild that were encrypted with it. Additionally, you should now be able to get a new token, and decrypt it using key 2, as this is now the primary token. Key 0 is our staging token, as always, and will be rotated to the top of the order once another rotation happens.


So we know how to rotate our keys, validate them and debug our mistral workflows. All this and OpenStack goodness to boot. I hope this gives you some tools to further your knowledge on this subject, as well as enhance your scripts and own code in the future.

Leave a Reply

%d bloggers like this: