Terraform and Jenkins: IaC from Your Build

Cloud infrastructure gives you unprecedented agility and flexibility. But those benefits don't come for free; if you don't have a way to bring them under control, they threaten to overwhelm you. You need infrastructure as code (IaC) and a way to integrate it into your existing continuous integration/continuous delivery (CI/CD) workflow.

IaC creates and manages system resources through configuration files instead of manipulating hardware resources manually or with interactive tools. In other words, IaC means automating your infrastructure and treating the automated tools like source code.

Terraform is a popular open-source IaC tool. With it, you can define your infrastructure with Hashicorp Configuration Language (HCL) or JSON.

In this post, you'll see how to apply a Terraform configuration using Jenkins. If you'd like to follow along with a video, you can watch CloudBees Developer Advocate Darin Pope walk through a similar tutorial below.

Now, let's dive in.

IaC replaces the manual interaction required to configure systems and networking resources. It gives you access to automated tools that you can manage the same way you manage configuration files. You develop, test, deploy, and, most importantly, manage those configurations with version control.

Terraform uses a declarative configuration language to provision infrastructure. It supports most major cloud vendors and customizable plugins that you can use and create for proprietary private clouds and on-premises systems.

For example, instead of provisioning an EC2 instance via the AWS control panel or CLI, you could start with a Terraform configuration file like this:

resource "aws_instance" "iac_in_action" {
  ami               = var.ami_id
  instance_type     = var.instance_type
  availability_zone = var.availability_zone
  // dynamically retrieve SSH Key Name
  key_name = aws_key_pair.iac_in_action.key_name
 
  // dynamically set Security Group ID (firewall)
  vpc_security_group_ids = [aws_security_group.iac_in_action.id]
 
  tags = {
    Name = "Terraform-managed EC2 Instance for IaC in Action"
  }
}

By using Terraform to manage your infrastructure, you gain the ability to roll out new configurations, roll them back when something goes wrong, and reproduce them on demand. So, for example, you could deploy the EC2 instance to a new availability zone in minutes as part of a more extensive deployment.

Terraform is primarily a command-line tool. Deploying a new configuration via the Terraform CLI is still easier than using interactive tools, and using a platform-independent tool is still preferable to proprietary tools. But running Terraform from the command line is still a manual process.

Running it from Jenkins removes the manual step. It also provides you with a way to integrate your infrastructure deployments into your CI/CD pipeline. 

Does a new release mean you need to open a new firewall port? Make it part of the deployment. Does another release need a new compute instance? Deploy it as you deploy the new code.

Need to roll something back? Roll back the Terraform deployment to close the firewall port or terminate the instance.

Let's see how to do this.

The first thing you need is a running Jenkins server. The screenshots for this post are from a server running version 2.303.2. Since we're running a very simple tutorial, you only need a recent version with the recommended plugins.

You’ll also be using a GitHub repository with a simple Terraform build job. It’ll install the Terraform binary as part of the job, execute a simple script, and clean up after itself. 

So, if your Jenkins server has internet access, you’re ready to go!

To run a Terraform job, you need the application. One option is to install it on your build agents. Another would be a Docker container. 

However, we’re going to use TFSwitch to install Terraform as part of the Jenkins job. TFSwitch downloads Terraform for you so you don't have to install it in advance. Then it runs it with the command line arguments you passed it.

So, rather than run Terraform directly, the Git repository uses a script named terraformw.

#!/bin/bash 

# based on https://tfswitch.warrensbox.com/Continuous-Integration/

echo "Installing tfswitch locally"

# Get the installer on to your machine
wget -N -c https://raw.githubusercontent.com/warrensbox/terraform-switcher/release/install.sh

# Make installer executable
chmod 755 install.sh

# Install tfswitch in a location you have permission
./install.sh -b $(pwd)/.bin

# set custom bin path
CUSTOMBIN=$(pwd)/.bin

#Add custom bin path to PATH environment
export PATH=$CUSTOMBIN:$PATH

$CUSTOMBIN/tfswitch -b $CUSTOMBIN/terraform

terraform $*

If you’ve worked with Java builds before, you may be familiar with gradlew or mvnw. They retrieve the configured version of the build tool, removing the need to install the required version in advance, and run it with the parameters you pass to the script.

terraformw does the same thing for Terraform by running TFSwitch. 

Terraform is a self-contained binary that runs on most Linux distributions and only weighs in around 32K. So downloading it for each job doesn’t take too much time. This allows you to keep your scripts up-to-date.

TFSwitch downloads the version of Terraform based on a command line entry or a config file. Since we’re running in a Jenkins job and don’t want to have to edit the script for each version change, we’ll use a config file.

We’re looking for version 1.0.0. Here’s the configuration.

terraform {
  required_version = "1.0.0"
}

Our Terraform file will print a single message.

output "jenkins_terraform" {
  value = "running Terraform from Jenkins"
}

Let’s set up a Jenkins job.

The Git repository has a simple Jenkinsfile. It cleans the workspace, checks out the repo, and runs terraformw. 

pipeline {
  agent { label 'linux'}
  options {
    skipDefaultCheckout(true)
  }
  stages{
    stage('clean workspace') {
      steps {
        cleanWs()
      }
    }
    stage('checkout') {
      steps {
        checkout scm
      }
    }
    stage('terraform') {
      steps {
        sh './terraformw apply -auto-approve -no-color'
      }
    }
  }
  post {
    always {
      cleanWs()
    }
  }
}

We’re telling Terraform to apply the configuration and skip waiting for interactive approval before applying it. The last argument tells it not to add color to the results, since it will only garble the output in the Jenkins log.

The build requires at least one build agent that has a label of linux. If you don’t want to add a label, you can fork this repo and edit the Jenkinsfile. 

Go to the Manage Jenkins menu.

Then select Manage Nodes and Clouds.

Now, choose a node.

And add the tag.

I’m building on the controller for this ad hoc install. This isn’t a best practice. For a production configuration, you would want to set up at least one agent node.

Now, add a new pipeline job. We’ll call this one Terraform_Tutorial.

Then, configure it with the GitHub URL.

Be sure to point the job at the Jenkinsfile and the GitHub repository’s main branch.

That’s it. Save the configuration and the job is ready to go.

Now it’s time to run the job. Click on Build Now.

The job will finish quickly. When it’s done, look at the console output.

After the git clone, we see the output of the Terraform stage.

First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (terraform)
[Pipeline] sh
+ ./terraformw apply -auto-approve -no-color
Installing tfswitch locally
--2021-10-31 00:01:08--  https://raw.githubusercontent.com/warrensbox/terraform-switcher/release/install.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9159 (8.9K) [text/plain]
Saving to: ‘install.sh’
#
     0K ........                                              100%  175M=0s
#
Last-modified header missing -- time-stamps turned off.
2021-10-31 00:01:08 (175 MB/s) - ‘install.sh’ saved [9159/9159]
#
warrensbox/terraform-switcher info checking GitHub for latest tag
warrensbox/terraform-switcher info found version: 0.12.1168 for 0.12.1168/linux/amd64
warrensbox/terraform-switcher info installed /var/jenkins_home/workspace/Terraform_Tutorial@2/.bin/tfswitch
Reading required version from terraform file
Reading required version from constraint: 1.0.0
Matched version: 1.0.0
Downloading to: /root/.terraform.versions
33043000 bytes downloaded
rename /root/.terraform.versions/terraform /root/.terraform.versions/terraform_1.0.0: no such file or directory
Switched terraform to version "1.0.0" 
#
Changes to Outputs:
  + jenkins_terraform = "running Terraform from Jenkins"
#
You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.
#
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
#
Outputs:
#
jenkins_terraform = "running Terraform from Jenkins"
[Pipeline] }

There’s the Terraform message at the end. From here, you can add any Terraform job, providing the agent has the proper access to the managed resources.

In this post, we covered how to run Terraform from Jenkins. We used TFSwitch to make your jobs more flexible and easier to run from different agents. We wrapped up with a simple job that outputs a single message.

Jenkins and IaC work well together. With them integrated, you can apply infrastructure changes as part of your CI/CD pipeline. You can manage the configurations alongside your source code, too. 

Now you know how to combine your infrastructure and your pipelines. If you want, you’re ready to get started today!

This post was written by Eric Goebelbecker. Eric has worked in the financial markets in New York City for 25 years, developing infrastructure for market data and financial information exchange (FIX) protocol networks. He loves to talk about what makes teams effective (or not so effective!).