Phases of Terraform adoption

As more of your organization adopts Terraform, your infrastructure provisioning workflows will need to change and adapt. The workflows that are suitable for individual practitioners may not scale to larger enterprises. This article will help you plan your organization's Terraform adoption strategy and presents workflow considerations that you should keep in mind to support future scale. This article focuses on challenges faced by larger organizations, but we recommend implementing each practice as early as you can to help you scale smoothly.

Adopt

An individual practitioner can establish strong foundational practices that support future scale and make Terraform operations predictable and secure.

Use version control

Store your Terraform configuration in a version control system, such as Git, just as you would with your application code. Terraform configuration files are code, and will benefit from the same features as your application in a version control repository such as versioning and easier code reviews.

You can connect your VCS provider to HCP Terraform to automatically initiate Terraform runs and view speculative plans that let you preview your infrastructure changes in your pull requests.

Reuse code with modules

Terraform modules group resources that you usually deploy together, letting you define reusable units of infrastructure code. For example, when you create a VPC in AWS, you may also need to create subnets, the route table, the internet gateway, security groups, and more. Instead defining the individual resources and configuring the relationships between them every time you need a new VPC, you can use the VPC module, which you can customize using input variables to quickly create the required infrastructure. The public Terraform module registry offers many modules that encode best practices for common use cases.

You can also create your own modules to deploy the specific infrastructure required by your services. Even a small three-tier application may require many Terraform-managed resources. A module lets you contain that complexity, turning each deployment of the application stack into a short, readable, and reusable configuration. The following Terraform configuration references a local module stored at ./modules/appstack that takes in two arguments named web_instance_count and api_instance_count:

module "appstack" {
  source = "./modules/appstack"

  web_instance_count = 2
  api_instance_count = 1
}

output "web_instance_ips" {
  value = module.appstack.web_ips
}

Follow our tutorials to learn how to use and develop modules and explore the public Terraform module registry.

Use secrets storage

Your configuration may rely on sensitive values, such as provider credentials. Although you can mark certain variables as sensitive to prevent displaying them as plaintext in run output, a more robust solution is to use secrets storage such as HashiCorp Vault

Vault securely stores sensitive information such as credentials and provides granular access control. You can integrate Vault into your Terraform configuration using the Vault provider. If you deploy your infrastructure to a major cloud provider, such as AWS, you can also generate short-lived credentials with Vault or use dynamic provider credentials, which prevents having to store credentials.

Vault also integrates into many popular CI/CD solutions such as GitHub, Jenkins, and CircleCI. Vault provides a central system to store and access data, which lets CI/CD pipelines push and pull secrets programmatically.

Collaborate

Multiple developers working on the same codebase introduces a new set of challenges, but solutions such as remote state backends help ease collaboration and coordinate execution.

Use remote state storage

As more team members work on Terraform configuration, you should implement remote state storage to support collaboration. HCP Terraform and remote backends implement several features to help you safely manage your Terraform state:

• Storage: Remote state storage lets you manage infrastructure collaboratively and securely. Different state stores may also support additional features for state management, such as encryption, versioning, automated backups, redundancy, and more.
• Locking: Some remote state storage options support state locking. State locking prevents concurrent Terraform operations on single state files.
• Execution: HCP Terraform and Terraform Enterprise support executing Terraform operations in stable, remote environments.

Since state files may contain sensitive data, refer to your backend documentation and, if supported, use state encryption. HCP Terraform and Terraform Enterprise both automatically encrypt state, and AWS, GCP, and Azure backends can implement encryption as well.

As your team grows, you may run into the risk of concurrent operations on state files. If supported by your remote storage solution, use state locking to prevent unpredictable outcomes or corrupted data. HCP Terraform and Terraform Enterprise support state locking by default, but other state storage implementations require additional configuration. For example, the AWS S3 remote backend requires that a DynamoDB table for state locking.

Get started with HCP Terraform and learn how to securely store your Terraform state.

Implement code reviews

Implement good code practices for your Terraform configuration, including using pull requests for code changes and performing proper code reviews. Code reviews can prevent introducing errors into your infrastructure configuration. They also help team members share their knowledge of the code base and enforce coding standards.

Use the integrations offered by your version control system to help with your code reviews. For example, HCP Terraform's VCS integration generates speculative plans for each pull request, showing the exact changes that Terraform will make to your infrastructure.

Automate deployments with CI/CD

A CI/CD pipeline offers a consistent process for shipping new features and fixes. By storing your Terraform configuration in version control, you define a single source of truth for your infrastructure configuration and can automate your deployments. You can configure a CI pipeline to automatically start a Terraform plan and apply operation for any changes to your code.

Terraform integrates with many automation solutions. If you do not have an existing CI/CD workflow, HashiCorp's Setup Terraform GitHub action sets up and configures the Terraform CLI in your Github Actions workflow.

Scale

As Terraform usage expands across your organization, you will need to decide how to define boundaries of infrastructure ownership.

You will also need to decide on a cloud deployment strategy based on your organization's practices and needs. Possible approaches include using a single account in a single cloud provider, a hybrid or multi-cloud approach, or to divide up resources across accounts by environment. Regardless of your implementation, Terraform lets you manage your infrastructure with a consistent workflow.

Adopt modules across your organization

We recommend using modules early in your Terraform adoption process to support consistent infrastructure configuration. As your Terraform usage scales, a central module registry helps teams find and use your modules rather than rewriting the same code.

Terraform supports multiple module distribution options, but we recommend that you use a native Terraform module registry such as HCP Terraform or Terraform Enterprise. These both use the module registry protocol, which is the Terraform-specific protocol to discover metadata about modules available for installation and to locate the distribution package for a selected module.

If you cannot use a native module registry, there are other source options such as Git repositories or AWS S3.

Modules also help teams establish infrastructure configuration standards. For example, you can write a module to create a database used by your application that includes all of the defaults that your architecture requires. The module can define the database size, type, and handle all of the required networking. This ensures that module consumers provision infrastructure in line with your organization standards and requirements.

Since modules define their own inputs, you can decide which parameters are configurable by the user. For example, you might want to allow them to change the size of the cluster, but not let them change the engine type.

Read the recommended patterns for creating modules.

Divide infrastructure responsibility

It is common for different teams to focus on different parts of your organization's infrastructure. For example, the networking team may manage the VPCs, while the application team only needs to know where to deploy their application and focuses on configuring servers and databases. In this scenario, there is a division of responsibilities but the application team still needs to access data about the networking resources for their own configuration.

Terraform lets you reference data about other resources in your configuration without having to manage them in the same state file, allowing you to maintain distinct areas of ownership and infrastructure decoupling. You can use data sources to query a provider for more data about a particular resource, or reference output values from another state file using the remote state data source. HCP Terraform lets you explicitly grant access to your workspace state file to only the workspaces that need it, reducing access to potentially sensitive data. You can also use the tfe_outputs data source to access the outputs of another HCP Terraform workspace.

Consider multiple IaaS accounts

Many Terraform users start by deploying to a single account in their cloud provider. This makes sense when you are managing only a few resources. As your Terraform adoption matures, managing thousands of resources across several cloud providers can become very complex, slow, and hard to secure. One strategy is to split your managed resources into multiple accounts in a way that makes sense to your organization. For example, you may want an account per deployment environment, such as one for development and one for production.

Govern

As your teams grow, a common operational challenge is deciding how to enforce your organization's standards and practices. Using codified, automated policy enforcement with Sentinel or OPA ensures consistent application of your standards.

Govern infrastructure through policy

You can use policy as code to ensure your infrastructure meets your organization's security, governance, and cost requirements. You can configure your workflows to automatically run policy checks as part of your Terraform operations and set conditions for how to handle policy failures. Soft enforcement lets prompts a user to approve an operation that fails a policy check, and hard enforcement blocks the operation entirely.

You can define policies that set standards for both your infrastructure configuration itself, and for the workflows around configuration deployment. Some examples of policy rules you can define include which ports are open in a firewall, the permitted sizes of virtual machines, or that deployments cannot take place on Fridays. In HCP Terraform and Terraform Enterprise you can use either OPA or Sentinel for your policy definitions.

Learn how to write a Sentinel policy for a Terraform Deployment and how to detect infrastructure drift and enforce OPA policies.

Next steps

This article introduces considerations to keep in mind as your organization adopts Terraform, but there are many more topics to explore. HCP Terraform provides a place to get started with many of these topics, and you can get started for free.

The HashiCorp Well-Architected Framework provides more in-depth information on how to adopt and scale your use of Terraform.