Terraform – State Files

Scaling and securing your deployments – managing remote state

Welcome to my series on Terraform, starting with the basics and moving into more advanced topics. We began with Terraform on Azure, we introduced the state file briefly.  To review, when you deploy Terraform it creates the state file to that maintains your environments’ configuration.  When deploying using Terraform to a small environment, maybe for your own personal use, you will deploy code directly from your local machine.  When doing this, the state file is created locally on your hard drive.  This is great for learning how to use Terraform, but this doesn’t scale to deploying to a real-life environment.

Once you run ‘Terraform plan’, the state file is created under the working directory from where you are deploying your working code:

A screen shot of a clock

Description automatically generated

When you deploy to your environment you want to place the state file somewhere secure, but also where it can be accessed by the entire deployment team.  One of the biggest advantages of Terraform is that you can reference what you have already built, enabling you to make changes or destroy an environment easily.  If you are running a local copy of your state file you run the risk of losing your device or leaving the company, taking your code and the environment configuration with you.

Using remote state is the recommended way to deploy and manage your state files.  Using remote state allows you and your team to work together to deploy your resources.  Remote state enables state locking that so once a deployment has begun, other uses cannot run Terraform against that state file. As a result, Terraform can run each time with the most up to date state file. 

By using a remote state file you encourage yourself and your team to collaborate, version control your code, and enable your team to open the door to use automated delivery using continuous integration – continuous delivery (CI/CD) to your organization.

In my video on Channel9, I walk through using the state file:

Secure your state file in Azure storage

I am going to expand and provide reference for how we can deploy a remote state file into Azure.

Azure storage will encrypt and persist your data, so it is an ideal place to store your state file for Azure deployments.  Instead of manually deploying our storage manually. I have created a script that you can use to automate this.  The script will provide you with a storage container, a storage blob and a KeyVault to store your storage key.  You can call the storage key from KeyVault as a variable from your Terraform script, or from a deployment pipeline, like Azure DevOps.

The key takeaway is that you don’t want to make your storage key visible from within your script in case it resides in a public repo, or visible to a wider group of employees that shouldn’t have access to the storage account.

This script can be run from the Azure Portal, PowerShell, the Azure CLI, or Terraform itself:  

set -e

export LOCATION=westeurope

# Create the resource group
az group create -n $RESOURCE_GROUP_NAME -l $LOCATION

echo "Resource group $RESOURCE_GROUP_NAME created."

# Create the storage account
az storage account create -g $RESOURCE_GROUP_NAME -l $LOCATION \
  --sku Standard_LRS \
  --encryption-services blob

echo "Storage account $TF_STATE_STORAGE_ACCOUNT_NAME created."

# Retrieve the storage account key
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $TF_STATE_STORAGE_ACCOUNT_NAME --query [0].value -o tsv)

echo "Storage account key retrieved."

# Create a storage container (for the Terraform State)
az storage container create --name $TF_STATE_CONTAINER_NAME --account-name $TF_STATE_STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY

echo "Storage container $TF_STATE_CONTAINER_NAME created."

# Create an Azure KeyVault
az keyvault create -g $RESOURCE_GROUP_NAME -l $LOCATION --name $KEYVAULT_NAME

echo "Key vault $KEYVAULT_NAME created."

# Store the Terraform State Storage Key into KeyVault
az keyvault secret set --name tfstate-storage-key --value $ACCOUNT_KEY --vault-name $KEYVAULT_NAME

echo "Key vault secret created."

# Display information
echo "Azure Storage Account and KeyVault have been created."
##echo "terraform init -backend-config=\"storage_account_name=$TF_STATE_STORAGE_ACCOUNT_NAME\" 
-backend-config=\"access_key=\$(az keyvault secret show --name tfstate-storage-key --vault-name $KEYVAULT_NAME --query value -o tsv)\" -backend-config=\"key=terraform-ref-architecture-tfstate\""

Once the script has successfully ran above, we need to tell Terraform where to store the state file, but there is also another feature that is important.  Backends.

Terraform uses a ‘backend’ to determine how the state is loaded and how commands such as ‘apply’ will execute. When deploying Terraform locally from your machine, TF assumes the ‘backend’ is ‘local.’ For non-local file state storage, remote execution, etc this type of file needs to be included to tell Terraform where to look for the state file.  The advantage of declaring the backend is that it can prevent latency in larger deployments. Enabling ‘backends’ is not required, but when working in collaboration with others and larger deployments it can solve many issues when working at scale.  The ‘backend’ feature enables you to store your state remotely and protect it with locks to prevent corruption.

For this blog we’re doing everything in Azure, so we need to declare in our ‘backend’ file that our state file will be stored in Azure. Using remote storage in Azure only on demand and is only stored in memory, so the state file will only be persisted in the Azure storage blob.  I have created a ‘backend.tf’ file that sits in the folder level as my main.tf file:

A screenshot of a cell phone screen with text

Description automatically generated

The ‘backend.tf’ file is very simple for this demo:

Another example of somewhere you can store the state file is using Hashicorp’s Consul service.  You can read more about it here.  A snippet of sample code for comparison is below:

Now that we have configured a backend.tf file and have configured our storage to securely deploy our state file to, we need to tell Terraform to where to access its state file.  This script will set the different keys for your backend configuration. You can do this by running the following command from ‘Terraform init’:

terraform init -backend-config=\"storage_account_name=$TF_STATE_STORAGE_ACCOUNT_NAME\" 
    -backend-config=\"access_key=\$(az keyvault secret show --name tfstate-storage-key -vault-name $KEYVAULT_NAME --query value -o tsv)\" 

Once this is done, you will have successfully configured Azure storage to maintain your remote state file!

Common State File Issues

State files can sometimes cause issues. I’ve listed 2 of the most common issues with state files that can occur.

Locked State File

In order to lock the state file, run ‘terraform plan’ and the state file will be locked.   Your state file will lock out any other attempts to run state against it. State locking happens automatically, if there is an error or issue with locking the state, Terraform will output the error to you and fail to execute. 

In the case that your state file gets locked out (which unfortunately can happen), there is a ‘force-unlock’ command.  Be VERY VERY careful when using this.  It should only be used when you are attempting to unlock your own state file.  If you are using a shared state file, a force unlock could cause multiple writers, causing overwrite of your state file.

Losing the State File or Importing State

Let’s say that you’ve created resources by other means in Azure (manually, ARM templates, etc), or you’ve lost your state file from another project. You can run the ‘Terraform import’ command and it will write your existing resources into your state file. You will need to reference the resource group name in Azure using it’s resource ID.  For example:

Terraform import azurerm_resource-group.rg /subscriptions/SUBSCRIPTION_ID/resourceGroups/<YOUR RESOURCE GROUP NAME>

It does not fully generate a configuration, but it is rumored that Hashicorp is looking to make this a feature in future releases.

I hope this guide has been useful, please send across any questions here or on Twitter. I have covered off other advanced topics on Terraform, links for them are below:

Terraform Modules – Deploying re-usable code

Terraform and Azure DevOps – Delivering CI/CD deployments – Link Coming Soon!

Terraform and Github Actions – Delivering code from your repo – Link Coming soon!

2 thoughts on “Terraform – State Files”

  1. Hi April..

    Thank you for the in-depth explanation of the Terraform content on Azure. I’ve been following the content on multiple channels and found it very interesting.

    I have a doubt though regarding the script above(to create the storage account and key-vault) which i am using via AZ’s powershell module. Post creating the storage account, I have to create context around it which is not allowing the proper container to be created around that context. I did some research around it, applied some Wait-events as well post storage account as well as the context around it, still the container is not being created. When i do retried multiple-times it resulted in a container after some-time(no specific timestamp elapsed). Is it something which you can advise on.

    Or should i be using the bash version only to create the account and containers as mentioned above.

    # Connect to Azure ACCOUNT

    set-variable -name LOCATION -value eastus
    set-variable -name RESOURCE_GROUP_NAME -value rg-terraform-state
    set-variable -name TF_STATE_STORAGE_ACCOUNT_NAME -value sthermothtfstate
    set-variable -name TF_STATE_CONTAINER_NAME -value tfstates
    set-variable -name KEYVAULT_NAME -value kv-tf-state
    set-variable -name KEYVAULT_SECRET -value tf-kv-state-secret
    # Create the resource group
    New-AzResourceGroup -Name $RESOURCE_GROUP_NAME -Location $LOCATION

    echo “Resource group $RESOURCE_GROUP_NAME created.”

    Error :
    New-AzStorageContainer: The specified resource does not exist.

    # Create the storage account
    $storage= New-AzStorageAccount -ResourceGroupName $RESOURCE_GROUP_NAME -Location $LOCATION `
    -SkuName Standard_LRS `
    -Default $azcontext

    echo “Storage account $TF_STATE_STORAGE_ACCOUNT_NAME created.” $storage

    # Waiting for the storage account to setup and refresh
    Wait-Event -Timeout 10

    # Retrieve the storage account key
    $ACCOUNT_KEY=(Get-AzStorageAccountKey -ResourceGroupName $RESOURCE_GROUP_NAME -Name $TF_STATE_STORAGE_ACCOUNT_NAME)| Where-Object {$_.KeyName -eq “key1”}
    $constr=’DefaultEndpointsProtocol=https;AccountName=’ + $TF_STATE_STORAGE_ACCOUNT_NAME + ‘;AccountKey=’ + $saKey + ‘;EndpointSuffix=core.windows.net’

    echo “Storage account key retrieved.”

    Wait-Event -Timeout 10
    # Create a storage context to be consumed for storage container
    echo “Now creating context of the storage account.”
    $ctx=New-AzStorageContext -ConnectionString $constr

    echo “Context created for the storage account and is as follows:” $ctx

    # Waiting for the storage account context to setup and refresh
    Wait-Event -Timeout 10

    echo “Creating storage container now”
    # Create a storage container (for the Terraform State)
    New-AzStorageContainer -Name $TF_STATE_CONTAINER_NAME -Context $ctx -Permission blob

    echo “Storage container $TF_STATE_CONTAINER_NAME created.”

    # Create an Azure KeyVault
    New-AzKeyVault -Name $KEYVAULT_NAME -Location $LOCATION -ResourceGroupName $RESOURCE_GROUP_NAME

    echo “Key vault $KEYVAULT_NAME created.”

    # Store the Terraform State Storage Key into KeyVault
    $Secret = ConvertTo-SecureString -String ‘$ACCOUNT_KEY.Value’ -AsPlainText -Force
    Set-AzKeyVaultSecret -VaultName $KEYVAULT_NAME -Name $KEYVAULT_SECRET -SecretValue $Secret

    echo “Key vault secret created.”


  2. Hello! You can do it with PowerShell, but yes, I did find the bash version easier.

    # Create resource group
    az group create –name $RESOURCE_GROUP_NAME –location “West Europe”
    # Create storage account
    az storage account create –resource-group $RESOURCE_GROUP_NAME –name $STORAGE_ACCOUNT_NAME –sku Standard_LRS –encryption-services blob
    # Get storage account key
    ACCOUNT_KEY=$(az storage account keys list –resource-group $RESOURCE_GROUP_NAME –account-name $STORAGE_ACCOUNT_NAME –query [0].value -o tsv)
    # Create blob container
    az storage container create –name $CONTAINER_NAME –account-name $STORAGE_ACCOUNT_NAME –account-key $ACCOUNT_KEY
    echo “storage_account_name: $STORAGE_ACCOUNT_NAME”
    echo “container_name: $CONTAINER_NAME”
    echo “access_key: $ACCOUNT_KEY”


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s