Terraform – State Files

Scaling and securing your deployments – managing remote state

Welcome to my series on Terraform, starting with the basics and moving into more advanced topics. We began with Terraform on Azure, we introduced the state file briefly.  To review, when you deploy Terraform it creates the state file to that maintains your environments’ configuration.  When deploying using Terraform to a small environment, maybe for your own personal use, you will deploy code directly from your local machine.  When doing this, the state file is created locally on your hard drive.  This is great for learning how to use Terraform, but this doesn’t scale to deploying to a real-life environment.

Once you run ‘Terraform plan’, the state file is created under the working directory from where you are deploying your working code:

When you deploy to your environment you want to place the state file somewhere secure, but also where it can be accessed by the entire deployment team.  One of the biggest advantages of Terraform is that you can reference what you have already built, enabling you to make changes or destroy an environment easily.  If you are running a local copy of your state file you run the risk of losing your device or leaving the company, taking your code and the environment configuration with you.

Using remote state is the recommended way to deploy and manage your state files.  Using remote state allows you and your team to work together to deploy your resources.  Remote state enables state locking that so once a deployment has begun, other uses cannot run Terraform against that state file. As a result, Terraform can run each time with the most up to date state file. 

By using a remote state file you encourage yourself and your team to collaborate, version control your code, and enable your team to open the door to use automated delivery using continuous integration – continuous delivery (CI/CD) to your organization.

In my video on Channel9, I walk through using the state file:

Secure your state file in Azure storage

I am going to expand and provide reference for how we can deploy a remote state file into Azure.

Azure storage will encrypt and persist your data, so it is an ideal place to store your state file for Azure deployments.  Instead of manually deploying our storage manually. I have created a script that you can use to automate this.  The script will provide you with a storage container, a storage blob and a KeyVault to store your storage key.  You can call the storage key from KeyVault as a variable from your Terraform script, or from a deployment pipeline, like Azure DevOps.

The key takeaway is that you don’t want to make your storage key visible from within your script in case it resides in a public repo, or visible to a wider group of employees that shouldn’t have access to the storage account.

This script can be run from the Azure Portal, PowerShell, the Azure CLI, or Terraform itself:  

set -e

export LOCATION=westeurope
export RESOURCE_GROUP_NAME=<YOUR_RESOURCE_GROUP_NAME>
export TF_STATE_STORAGE_ACCOUNT_NAME=<YOUR_STORAGE_ACCOUNT_NAME>
export TF_STATE_CONTAINER_NAME=<YOUR_STORAGE_CONTAINER_NAME>
export KEYVAULT_NAME=<YOUR_KEYVAULT_NAME>

# Create the resource group
az group create -n $RESOURCE_GROUP_NAME -l $LOCATION

echo "Resource group $RESOURCE_GROUP_NAME created."

# Create the storage account
az storage account create -g $RESOURCE_GROUP_NAME -l $LOCATION \
  --name $TF_STATE_STORAGE_ACCOUNT_NAME \
  --sku Standard_LRS \
  --encryption-services blob

echo "Storage account $TF_STATE_STORAGE_ACCOUNT_NAME created."

# Retrieve the storage account key
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $TF_STATE_STORAGE_ACCOUNT_NAME --query [0].value -o tsv)

echo "Storage account key retrieved."

# Create a storage container (for the Terraform State)
az storage container create --name $TF_STATE_CONTAINER_NAME --account-name $TF_STATE_STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY

echo "Storage container $TF_STATE_CONTAINER_NAME created."

# Create an Azure KeyVault
az keyvault create -g $RESOURCE_GROUP_NAME -l $LOCATION --name $KEYVAULT_NAME

echo "Key vault $KEYVAULT_NAME created."

# Store the Terraform State Storage Key into KeyVault
az keyvault secret set --name tfstate-storage-key --value $ACCOUNT_KEY --vault-name $KEYVAULT_NAME

echo "Key vault secret created."

# Display information
echo "Azure Storage Account and KeyVault have been created."
##echo "terraform init -backend-config=\"storage_account_name=$TF_STATE_STORAGE_ACCOUNT_NAME\" 
-backend-config=\"container_name=$TF_STATE_CONTAINER_NAME\" 
-backend-config=\"access_key=\$(az keyvault secret show --name tfstate-storage-key --vault-name $KEYVAULT_NAME --query value -o tsv)\" -backend-config=\"key=terraform-ref-architecture-tfstate\""

Once the script has successfully ran above, we need to tell Terraform where to store the state file, but there is also another feature that is important.  Backends.

Terraform uses a ‘backend’ to determine how the state is loaded and how commands such as ‘apply’ will execute. When deploying Terraform locally from your machine, TF assumes the ‘backend’ is ‘local.’ For non-local file state storage, remote execution, etc this type of file needs to be included to tell Terraform where to look for the state file.  The advantage of declaring the backend is that it can prevent latency in larger deployments. Enabling ‘backends’ is not required, but when working in collaboration with others and larger deployments it can solve many issues when working at scale.  The ‘backend’ feature enables you to store your state remotely and protect it with locks to prevent corruption.

For this blog we’re doing everything in Azure, so we need to declare in our ‘backend’ file that our state file will be stored in Azure. Using remote storage in Azure only on demand and is only stored in memory, so the state file will only be persisted in the Azure storage blob.  I have created a ‘backend.tf’ file that sits in the folder level as my main.tf file:

The ‘backend.tf’ file is very simple for this demo:

Another example of somewhere you can store the state file is using Hashicorp’s Consul service.  You can read more about it here.  A snippet of sample code for comparison is below:

Now that we have configured a backend.tf file and have configured our storage to securely deploy our state file to, we need to tell Terraform to where to access its state file.  This script will set the different keys for your backend configuration. You can do this by running the following command from ‘Terraform init’:

terraform init -backend-config=\"storage_account_name=$TF_STATE_STORAGE_ACCOUNT_NAME\" 
    -backend-config=\"container_name=$TF_STATE_CONTAINER_NAME\" 
    -backend-config=\"access_key=\$(az keyvault secret show --name tfstate-storage-key -vault-name $KEYVAULT_NAME --query value -o tsv)\" 
    -backend-config=\"key=terraform-ref-architecture-tfstate\"

Once this is done, you will have successfully configured Azure storage to maintain your remote state file!

Common State File Issues

State files can sometimes cause issues. I’ve listed 2 of the most common issues with state files that can occur.

Locked State File

In order to lock the state file, run ‘terraform plan’ and the state file will be locked.   Your state file will lock out any other attempts to run state against it. State locking happens automatically, if there is an error or issue with locking the state, Terraform will output the error to you and fail to execute. 

In the case that your state file gets locked out (which unfortunately can happen), there is a ‘force-unlock’ command.  Be VERY VERY careful when using this.  It should only be used when you are attempting to unlock your own state file.  If you are using a shared state file, a force unlock could cause multiple writers, causing overwrite of your state file.

Losing the State File or Importing State

Let’s say that you’ve created resources by other means in Azure (manually, ARM templates, etc), or you’ve lost your state file from another project. You can run the ‘Terraform import’ command and it will write your existing resources into your state file. You will need to reference the resource group name in Azure using it’s resource ID.  For example:

Terraform import azurerm_resource-group.rg /subscriptions/SUBSCRIPTION_ID/resourceGroups/<YOUR RESOURCE GROUP NAME>

It does not fully generate a configuration, but it is rumored that Hashicorp is looking to make this a feature in future releases.

I hope this guide has been useful, please send across any questions here or on Twitter. I have covered off other advanced topics on Terraform, links for them are below:

Terraform Modules – Deploying re-usable code

Terraform and Azure DevOps – Delivering CI/CD deployments – Link Coming Soon!

Terraform and Github Actions – Delivering code from your repo – Link Coming soon!