Testing Linux Docker Images in CloudBees CodeShip CI

In this article, we will see how we can integrate testing into the process of building Linux docker container images. The ideas will be generic, but we will focus on using CloudBees CodeShip Pro as the continuous integration solution.

While testing container images, we can write tests of two kinds:

• Tests which test the container image contents by inspecting the filesystem layers directly

• Running a container and then inspecting various expected characteristics of the running container

There are various open source tools which aim to provide a framework for testing Linux container images. container structure tests, inspec, serverspec and dgoss are a few of those examples.

We will also look at integrating container structure tests in our continuous integration (CI) pipeline. We will use a container structure test to run tests against the contents of the Docker image and supplement these with additional tests which will test a running container from that image.

All the code examples we discuss are available in a git repository on github.

We need a container image under test. Although there are various ways one can create such an image, we will use docker's Dockerfile and build a docker image from it. Our docker image will run a nginx server with the geoip2 module installed.

The Dockerfile looks as follows:

FROM centos:7
RUN yum clean all && \
    yum update -y && \
    yum install epel-release -y && \
    yum -y install https://extras.getpagespeed.com/release-el7-latest.rpm && \
    yum -y install nginx wget nginx-module-geoip2
WORKDIR /etc
RUN wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz && \
    wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.mmdb.gz && \
    gzip -d GeoLite2-City.mmdb.gz && \
    gzip -d GeoLite2-Country.mmdb.gz
RUN mkdir /content/
COPY index.html /content/
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 80
VOLUME ["/etc/nginx/conf.d/"]
ENTRYPOINT ["nginx"]

Considering the purpose of our image, here are some of the key features of the image's filesystem we want to test:

• A nginx configuration file exists in the container at /etc/nginx/nginx.conf

• A directory /content exists and has a file index.html

• The GeoLite2 database files necessary for GeoIP2 module to work are present inside the image

• The container exposes the port 80 outside the container

• The entry point is set correctly and it exposes a docker volume /etc/nginx/conf.d/

The characteristics we want to test for a container running from this image are:

• When we run a container from this image, we should get back an HTTP 200 response when we make an HTTP GET request to container's IP address

• Given the HTTP header (X-Forwarded-For) with an appropriate value, we should see logs in a valid JSON format and including GeoIP decoded information (See the nginx.conf file for details on how the logging is set up)

Container structure tests are written as YAML or JSON files and currently (as of release v1.8.8) fall into the following categories:

• Metadata tests (metadataTest)

• File existence tests (fileExistenceTests)

• File content tests (fileContentTests)

• Command tests (commandTests)

Let's see the test file we will use to verify the existence of the different files (as stated above):

schemaVersion: "2.0.0"
fileExistenceTests:
- name: 'nginx basic config'
  path: '/etc/nginx/nginx.conf'
  shouldExist: true
  permissions: '-rw-r--r--'
- name: 'nginx content'
  path: '/content/index.html'
  shouldExist: true
  permissions: '-rw-r--r--'
- name: 'geolite2 country db'
  path: '/etc/GeoLite2-Country.mmdb'
  shouldExist: true
  permissions: '-rw-r--r--'
- name: 'geolite2 city db'
  path: '/etc/GeoLite2-City.mmdb'
  shouldExist: true
  permissions: '-rw-r--r--'

Each test file must have a schemaVersion which is currently, 2.0.0. Then we have a section fileExistenceTests and each of the test that we want to run as objects inside it. Each object must have the three required fields - the directory or file name, (name), path of the file or directory (path) and whether it should be present or absent (shouldExist). Each object can optionally have fields to check the Unix permissions set on the file (permissions), the User ID (uid) and Group ID (gid) of the owner of file, and who the file is executable by (isExecutableBy) which can be any of owner, group, other or any.

To verify the properties of a docker image - entrypoint, exposed ports and volumes, we will write a metadata test which looks as follows:

schemaVersion: "2.0.0"
metadataTest:
  exposedPorts: ["80"]
  volumes: ["/etc/nginx/conf.d/"]
  entrypoint: ["nginx"]

We can also test for image labels, working directory (workdir), image command (cmd) and environment variables env.

You can find both the tests above in the cst subdirectory of the repository.

For the runtime tests, we will use curl to make an HTTP GET request to our container and then inspect the logs emitted by nginx to verify they contain the GeoIP information. The curl command we will use is curl -H "X-Forwarded For: 8.8.8.8" http://nginx with the X-Forwarded-For header set to a certain IP address. This simulates the likely scenario that our nginx server is running behind a load balancer.

To verify the GeoIP information in the nginx access log, we use the following program written in Python:

#!/usr/bin/env python
import subprocess
import json
data=subprocess.check_output(["docker", "logs", "jet-nginx-HttpTest", "--tail", "1"])
j = json.loads(data)
log_keys = j.keys()
expected_fields = ['geoip_country_code', 'geoip_city', 'geoip_country_name', 'geoip_timezone']
for f in expected_fields:
    assert f in log_keys

We use docker logs to get a line of output from the nginx container, attempt to serialize the data as a JSON object and then look for the GeoIP related fields in the log.

With the building blocks in place, let's now look at the files needed for building our project in CodeShip CI pro. We need two files: first, the codeship-services.yml file which defines the services (as containers) we will use in the steps defined in the codeship-steps.yml file.

Next, we look at the codeship-services.yml file's contents. We define four services in this file. First, we have the nginx service:

nginx:
  build:
    image: amitsaha/nginx-geoip2
    context: .
    dockerfile: Dockerfile.image_under_test

We specify that for this service, we will build the image amitsaha/nginx-geoip2 from the Dockerfile specified at the path Dockerfile.image_under_test.

Next, we define the cst service where we specify the container-structure-test image that we want to pull from Google Container registry:

cst:
  image: gcr.io/gcp-runtimes/container-structure-test
  depends_on: ["nginx"]
  volumes:
    - ./cst/filesystem_tests.yaml:/test-config/filesystem_tests.yaml
    - ./cst/metadata_tests.yaml:/test-config/metadata_tests.yaml
    - /var/run/docker.sock:/var/run/docker.sock

In addition, we specify three files to volume mount inside the container, two of them being tests that we want to run and the third the docker daemon socket, so that we can communicate with the docker daemon running on the host to run the tests on the docker image that we built in the previous step. We specify a dependency on the nginx service so that the image is built before the tests are attempted to be run.

Next, we specify a service to build a docker container which will have the docker client installed so that we can run the runtime test where we need to inspect the logs of the nginx service:

docker_client:
  build:
    image: amitsaha/docker-client
    context: .
    dockerfile: Dockerfile.docker_client
  depends_on: ["curl"]
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

Lastly, we define the curl service which we will use to run the HTTP GET test using curl:

curl:
  image: pstauffer/curl:latest
  depends_on: ["nginx"]

Next, we have the codeship-steps.yml file where we define the steps to run the various tests. First, we have the container structure tests:

- service: cst
  name: Run container structure tests
  command: test --image amitsaha/nginx-geoip2 --config /test-config/filesystem_tests.yaml --config /test-config/metadata_tests.yaml

The container structure tests framework can be used to run the tests either via the binary or via the published docker image. Here, we have to use the docker image - whose entry point is set to container-structure-test (the binary). The arguments we supply to the binary are:

• test:: Sub-command specifying that we want to run the tests

• --image: Specifies the docker image we want to run the tests on

• --config: Specify the tests that we want to run and can be specified multiple times

Next, we run a test to make sure the nginx configuration of the image is valid:

- service: nginx
  name: Test nginx configuration
  command: -t

Next, we test that the nginx server has started correctly and it responds to an HTTP GET request with a 200:

- service: curl
  name: HttpTest
  command: 'curl -H "X-Forwarded-For: 8.8.8.8" http://nginx'

The final test tests the GeoIP decoding feature of the GeoIP2 module:

- service: docker_client
  name: Test GeoIP data in logs
  command: /test_nginx_log.py

If we go back to the above listing of the Python program we use to test the GeoIP information in the logs, you will see that we have this line:

data=subprocess.check_output(["docker", "logs", "jet-nginx-HttpTest", "--tail", "1"])

The container name looks a bit suspicious, although not random. The container name turns out to be jet-<service-name>-<test name>, where the service-name and test name are as defined in the codeship-steps.yaml file. Here we wanted to read the logs from the nginx container, so we derive the name of the container using the above pattern. Note that, this is not documented in the CodeShip CI docs.

If you have created a fork of my repository and then setup CodeShip Pro CI using the fork, you should be able to trigger a build and see the tests being run successfully. Alternatively, you could also use CodeShip's jet cli to run the tests locally on a clone of the demo repository using jet steps command.

• Container structure test

• Demo repository

• CodeShip Pro docs