In this article, we will see how we can integrate testing into the process of building Linux docker container images. The ideas will be generic, but we will focus on using CloudBees CodeShip Pro as the continuous integration solution.
While testing container images, we can write tests of two kinds:
Tests which test the container image contents by inspecting the filesystem layers directly
Running a container and then inspecting various expected characteristics of the running container
We will also look at integrating container structure tests in our continuous integration (CI) pipeline. We will use a container structure test to run tests against the contents of the Docker image and supplement these with additional tests which will test a running container from that image.
All the code examples we discuss are available in a git repository on github.
Our container image under test
We need a container image under test. Although there are various ways one can create such an image, we will use docker's Dockerfile and build a docker image from it. Our docker image will run a nginx server with the geoip2 module installed.
Dockerfile looks as follows:
FROM centos:7 RUN yum clean all && \ yum update -y && \ yum install epel-release -y && \ yum -y install https://extras.getpagespeed.com/release-el7-latest.rpm && \ yum -y install nginx wget nginx-module-geoip2 WORKDIR /etc RUN wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz && \ wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.mmdb.gz && \ gzip -d GeoLite2-City.mmdb.gz && \ gzip -d GeoLite2-Country.mmdb.gz RUN mkdir /content/ COPY index.html /content/ COPY nginx.conf /etc/nginx/nginx.conf EXPOSE 80 VOLUME ["/etc/nginx/conf.d/"] ENTRYPOINT ["nginx"]
Considering the purpose of our image, here are some of the key features of the image's filesystem we want to test:
A nginx configuration file exists in the container at
/contentexists and has a file
The GeoLite2 database files necessary for GeoIP2 module to work are present inside the image
The container exposes the port 80 outside the container
The entry point is set correctly and it exposes a docker volume
The characteristics we want to test for a container running from this image are:
When we run a container from this image, we should get back an HTTP 200 response when we make an HTTP GET request to container's IP address
Given the HTTP header (
X-Forwarded-For) with an appropriate value, we should see logs in a valid JSON format and including GeoIP decoded information (See the
nginx.conffile for details on how the logging is set up)
Container structure tests
Container structure tests are written as YAML or JSON files and currently (as of release v1.8.8) fall into the following categories:
Metadata tests (
File existence tests (
File content tests (
Command tests (
Let's see the test file we will use to verify the existence of the different files (as stated above):
schemaVersion: "2.0.0" fileExistenceTests: - name: 'nginx basic config' path: '/etc/nginx/nginx.conf' shouldExist: true permissions: '-rw-r--r--' - name: 'nginx content' path: '/content/index.html' shouldExist: true permissions: '-rw-r--r--' - name: 'geolite2 country db' path: '/etc/GeoLite2-Country.mmdb' shouldExist: true permissions: '-rw-r--r--' - name: 'geolite2 city db' path: '/etc/GeoLite2-City.mmdb' shouldExist: true permissions: '-rw-r--r--'
Each test file must have a
schemaVersion which is currently,
2.0.0. Then we have a section
fileExistenceTests and each of the test that we want to run as objects inside it. Each object must have the three required fields - the directory or file name, (
name), path of the file or directory (
path) and whether it should be present or absent (
shouldExist). Each object can optionally have fields to check the Unix permissions set on the file (
permissions), the User ID (
uid) and Group ID (
gid) of the owner of file, and who the file is executable by (
isExecutableBy) which can be any of
To verify the properties of a docker image - entrypoint, exposed ports and volumes, we will write a metadata test which looks as follows:
schemaVersion: "2.0.0" metadataTest: exposedPorts: ["80"] volumes: ["/etc/nginx/conf.d/"] entrypoint: ["nginx"]
We can also test for image
labels, working directory (
workdir), image command (
cmd) and environment variables
You can find both the tests above in the cst subdirectory of the repository.
For the runtime tests, we will use curl to make an HTTP GET request to our container and then inspect the logs emitted by nginx to verify they contain the GeoIP information. The
curl command we will use is
curl -H "X-Forwarded For: 126.96.36.199" http://nginx with the
X-Forwarded-For header set to a certain IP address. This simulates the likely scenario that our nginx server is running behind a load balancer.
To verify the GeoIP information in the nginx access log, we use the following program written in Python:
#!/usr/bin/env python import subprocess import json data=subprocess.check_output(["docker", "logs", "jet-nginx-HttpTest", "--tail", "1"]) j = json.loads(data) log_keys = j.keys() expected_fields = ['geoip_country_code', 'geoip_city', 'geoip_country_name', 'geoip_timezone'] for f in expected_fields: assert f in log_keys
docker logs to get a line of output from the
nginx container, attempt to serialize the data as a JSON object and then look for the GeoIP related fields in the log.
Setting up CloudBees CodeShip CI files
With the building blocks in place, let's now look at the files needed for building our project in CodeShip CI pro. We need two files: first, the
codeship-services.yml file which defines the services (as containers) we will use in the steps defined in the
Next, we look at the
codeship-services.yml file's contents. We define four services in this file. First, we have the
nginx: build: image: amitsaha/nginx-geoip2 context: . dockerfile: Dockerfile.image_under_test
We specify that for this service, we will build the image
amitsaha/nginx-geoip2 from the Dockerfile specified at the path
Next, we define the
cst service where we specify the
container-structure-test image that we want to pull from Google Container registry:
cst: image: gcr.io/gcp-runtimes/container-structure-test depends_on: ["nginx"] volumes: - ./cst/filesystem_tests.yaml:/test-config/filesystem_tests.yaml - ./cst/metadata_tests.yaml:/test-config/metadata_tests.yaml - /var/run/docker.sock:/var/run/docker.sock
In addition, we specify three files to volume mount inside the container, two of them being tests that we want to run and the third the docker daemon socket, so that we can communicate with the docker daemon running on the host to run the tests on the docker image that we built in the previous step. We specify a dependency on the
nginx service so that the image is built before the tests are attempted to be run.
Next, we specify a service to build a docker container which will have the docker client installed so that we can run the runtime test where we need to inspect the logs of the
docker_client: build: image: amitsaha/docker-client context: . dockerfile: Dockerfile.docker_client depends_on: ["curl"] volumes: - /var/run/docker.sock:/var/run/docker.sock
Lastly, we define the
curl service which we will use to run the HTTP GET test using
curl: image: pstauffer/curl:latest depends_on: ["nginx"]
Next, we have the
codeship-steps.yml file where we define the steps to run the various tests. First, we have the container structure tests:
- service: cst name: Run container structure tests command: test --image amitsaha/nginx-geoip2 --config /test-config/filesystem_tests.yaml --config /test-config/metadata_tests.yaml
The container structure tests framework can be used to run the tests either via the binary or via the published docker image. Here, we have to use the docker image - whose entry point is set to
container-structure-test (the binary). The arguments we supply to the binary are:
test:: Sub-command specifying that we want to run the tests
--image: Specifies the docker image we want to run the tests on
--config: Specify the tests that we want to run and can be specified multiple times
Next, we run a test to make sure the nginx configuration of the image is valid:
- service: nginx name: Test nginx configuration command: -t
Next, we test that the nginx server has started correctly and it responds to an HTTP GET request with a 200:
- service: curl name: HttpTest command: 'curl -H "X-Forwarded-For: 188.8.131.52" http://nginx'
The final test tests the GeoIP decoding feature of the GeoIP2 module:
- service: docker_client name: Test GeoIP data in logs command: /test_nginx_log.py
If we go back to the above listing of the Python program we use to test the GeoIP information in the logs, you will see that we have this line:
data=subprocess.check_output(["docker", "logs", "jet-nginx-HttpTest", "--tail", "1"])
The container name looks a bit suspicious, although not random. The container name turns out to be
jet-<service-name>-<test name>, where the
test name are as defined in the
codeship-steps.yaml file. Here we wanted to read the logs from the
nginx container, so we derive the name of the container using the above pattern. Note that, this is not documented in the CodeShip CI docs.
Trying it out
If you have created a fork of my repository and then setup CodeShip Pro CI using the fork, you should be able to trigger a build and see the tests being run successfully. Alternatively, you could also use CodeShip's jet cli to run the tests locally on a clone of the demo repository using
jet steps command.