Fernet Key Rotation on Red Hat OpenStack Platform with Ansible

Fernet Key Rotation on Red Hat OpenStack Platform with Ansible

Fernet tokens in Keystone are fantastic. Enabling these, instead of UUID or PKI tokens, really does make a difference in your cloud’s performance and overall ease of management. I get asked a lot about how to manage keys on your controller cluster when using fernet. As you may imagine, this could potentially take your cloud down if you do it wrong. Let’s look at what fernet  keys are, as well as how to manage them in your Red Hat OpenStack cloud.



  • A Red Hat OpenStack Platform or TripleO director-based deployment
  • One or more controller nodes
  • Git command-line client

What are Fernet Keys?

Fernet keys are used to encrypt and decrypt fernet tokens in OpenStack’s Keystone API. These keys are stored on each controller node, and must be available to authenticate and validate users of the various OpenStack components in your cloud.

Any given implementation of keystone can have (n)keys based on the max_active_keys setting in /etc/keystone/keystone.conf. This number will include all of the types listed below.

There are essentially three types of keys:


Primary keys are used for token generation and validation. You can think of this as the active key in your cloud. Any time a user authenticates, or is validated by an OpenStack API, these are the keys that will be used. There can only be one primary key, and it must exist on all nodes (usually controllers) that are running the keystone API. The primary key is always the highest indexed key.


Secondary keys are only used for token validation. These keys are rotated out of primary status, and thus are used to validate tokens that may exist after a new primary key has been created. There can be multiple secondary keys, the oldest of which will be deleted based on your max_active_keys setting after each key rotation.


These keys  are always the lowest indexed keys (0). Whenever keys are rotated, this key is promoted to a primary key at the highest index allowable by max_active_keys. These keys exist to allow you to copy them to all nodes in your cluster before they’re promoted to primary status. This avoids the potential issue where keystone fails to validate a token because they key used to encrypt it does not yet exist in /etc/keystone/fernet-keys.

An example follows, showing the keys that you’d see in /etc/keystone/fernet-keys, with max_active_keys set to 4.

Upon performing a key rotation, our staged key (0), will be the new primary key (2), while our old primary key (1), will be moved to secondary status (1).

We have three keys here, so yet another key rotation, will produce the following result:

Our staged key (0), now becomes our primary key (3). Our  old primary key (2), now becomes secondary key (2), and (1) remains a secondary key.

We now have four keys, the number we’ve set in max_active_keys. One more final rotation would produce the following:

Our oldest key, secondary (1), is deleted. Our previously staged key (0), is moved to primary (4) status.  A new staged key (0) is created. And finally our old primary key (3) is moved to secondary status.

If you haven’t noticed this by now, rotating keys will always remove the key with the lowest index, excluding 0 — up to your max_active_keys. Additionally, note that you must be careful to set your max_active_keys configuration setting to something that makes sense, given your token lifetime and how often you plan to rotate your keys.

When to rotate?

The answer to this question would probably be different for most organizations. My take on this is simply: if you can do it safely, why not automate it and do it on a regular basis? Your threat model and use-case would normally dictate this, but I think about regular key rotation as a best-practices security measure. You always want to limit the amount of sensitive data, in this case fernet tokens, encrypted with a single version of any given encryption key. Rotating your keys on a regular basis creates a smaller exposure surface for your cloud and your users.

Rotating Fernet keys

So you may be wondering, how does one automate this process? You can image that this process can be painful and prone to error if done by hand. While you could use the ferent_rotate command to do this on each node by hand, why would you? Let’s look at how to do this with Ansible, Red Hat’s awesome tool for automation. If you’re new to Ansible, please do yourself a favor and check out this quick-start video.

We’ll be using an awesome Ansible role, created by my fellow Red Hatter Juan Antonio Osorio (Ozz), one of the coolest guys I know.

Let’s start by logging into your Red Hat OpenStack director node as the stack user, and create a roles directory in /home/stack:

Go into the roles directory, install git and clone Ozz’s repo containing the role we’ll use to rotate keys:

Next, we’ll create an Ansible playbook that uses this role, called rotate.yml

We need to source our stackrc, as we’ll be operating on our controller nodes in the next step

Using a dynamic inventory from /usr/bin/tripleo-ansible-inventory, we’ll run this playbook and rotate the keys on our controllers

Ansible Role Analysis

What happened? Looking at Ansible’s output, you’ll note that several tasks were performed. If you’d like to see these tasks, look no further than /home/stack/roles/tripleo-fernet-keys-rotation/tasks/main.yml:

This task runs a python script, generate_key_yaml.py, in ~/roles/tripleo-ansible-inventory/files, that creates a new fernet key.

This task will take the output of the previous task, from stdout, and register it as the new_key.

Next, we get a sorted list of the keys that currently exist in /etc/keystone/fernet-keys

Let’s set the next primary key index

Now we’ll move the staged key to the new primary key

Next, let’s set our new_key to the new staged key

Finally, we’ll reload (not restart) httpd on the controller, allowing keystone to load the new keys


Now that we have a way to automate rotation of our keys, it’s time to schedule this automation. There are several ways you could do this:


You could, but why?

Systemd Realtime Timers

Let’s create the systemd service that will run our playbook:

Now we’ll create a timer with the same name, only with .timer as the suffix, in /etc/systemd/system on the director node:

What about logging? Ansible to the rescue. Ansible will use the log_path configuration option from /etc/ansible/ansible.cfg, ansible.cfg in the directory of the playbook, or $HOME/.ansible.cfg. You just need to set this and forget it.

So let’s enable this service and timer, and we’re off to the races:

Ansible Tower

More on this later. Needless to say, I like this option best.

Credit: Many thanks to Lance Bragstad: http://lbragstad.com, and Dolph Matthews for the key rotation methodology.

Leave a Reply

%d bloggers like this: