Rotating Infrastructure Secrets

Using systemd and rsyslog to provide reliable service

Prior Reading Required

This guide is written with the assumption that you're comfortable with the concepts from the Native Linux Installation Guide. Examples below assume an environment that matches the examples in the linked guide; if your environment differs, some commands may not work as written.

The GRAX app requires several secret values specified in the environment (normally sourced from .env). These include a valid Postgres connection string, an administrator password, and a key base for encryption of the DB-based Secrets Store used for SFDC and storage connection secrets (SECRET_STORE_BASE). Rotation of secrets is mostly external to the GRAX app, with one exception.

Rotating Database Connection String or Administrator Password

To rotate the connection string used to connect to the Postgres database cluster manually, perform the following steps:

  1. Stop the GRAX services

    $ systemctl disable grax.service; systemctl stop grax.service;
    
  2. Update the configuration source/file with your editor of choice:

    $ vim .env
    
    [change intended key\'s value to new value and save file]
    
  3. Start the GRAX services

    $ systemctl enable grax.service; systemctl start grax.service;
    

If you have interest in automating this behavior, the automation needs to preserve or recreate the other necessary values for the configuration.

Rotating SECRET_STORE_BASE

The SECRET_STORE_BASE is used to encrypt the SFDC and Storage secrets in the database. Changing this value between reboots without proper care results in these secrets being irrecoverable and the GRAX app being unable to start properly; a manual reset of configuration information in the database is the only recovery option.

To properly rotate this value, perform the following steps:

  1. Stop the GRAX services

    $ systemctl disable grax.service; systemctl stop grax.service;
    
  2. Update the SECRET_STORE_BASE to the new value with your editor of choice

  3. Update the PREVIOUS_SECRET_STORE_BASE to the previous value with your editor of choice

  4. Start the GRAX services

    $ systemctl enable grax.service; systemctl start grax.service;
    

At this point, the GRAX app reads the configuration secrets with the old key and writes them with the new key on first boot. To clean up the extraneous PREVIOUS_SECRET_STORE_BASE after a rotation: wait two minutes, stop the GRAX services, and remove the value before starting GRAX again.