Test Mobile Apps with Jenkins Workflow and AWS Device Farm

In his blog post Continuous Delivery with CloudBees Jenkins Platform and AWS Lambda , Cyrille Le Clerc introduced the CloudBees AWS CLI plugin. However, what was not mentioned in that post is that this plugin also includes Build Wrapper support for the Jenkins Workflow plugin . Jenkins Workflow Build Wrapper support for the AWS CLI plugin simplifies integration of AWS CLI functionality with your Jenkins Workflows jobs. In this post, we will explore how you can use this integration to build one Jenkins Workflow that will build an Android app inside a Docker container, then upload the app and test package to the AWS Device Farm , and then test the app against a pool of real devices in the AWS Device Farm cloud.

The AWS Device Farm allows you to test your Android and iOS mobile apps against real devices provisioned and managed by Amazon in the AWS cloud. The Device Farm supports a number of different test types and for this post we will be using Android instrumentation tests provided by the AWS Device Farm sample app for Android . In addition to a typical web based console for interacting with the AWS Device Farm, Amazon also offers a full-featured CLI for the Device Farm. If you aren’t already familiar with the AWS Device Farm, take a moment to look over the AWS Device Farm Getting Started documentation and the AWS Device Farm CLI documentation.

The OSS CloudBees Jenkins Workflow plugin is the go to Jenkins plugin for enabling Continuous Delivery with Jenkins. The Workflow plugin also has excellent integration with Docker via the Docker Workflow plugin . Jesse Glick discussed this capability in his blog post Orchestrating Workflows with Jenkins and Docker . Basically, using Docker containers as ‘build environment wrappers’ completely changes the management of tool-chains needed to deliver software quickly. And when you have a tool-chain that is as volatile as what is needed to support Android software delivery, the importance of optimizing the management and delivery of these tool-chains cannot be understated.

The CloudBees AWS CLI plugin makes it easy to integrate a secure AWS CLI environment into your Jenkins jobs. And even better, it has out of the box support for Jenkins Workflow via the wrap step. We will also utilize the CloudBees AWS Credentials plugin that is integrated with the AWS CLI plugin to manage the AWS credentials we will use to interact with the AWS Device Farm CLI.

Now we are going to create a new Multibranch Workflow job that will:

1. Build an Android app and instrumentation test package in a Docker container using a Docker image that has all the necessary tools.

2. Set-up AWS Credentials with an ID for use with CloudBees AWS CLI Plugin.

3. Use the CloudBees AWS CLI plugin to Create an AWS Device Farm Project and Device Pool.

4. Upload the Android app and test package to AWS.

5. Schedule an AWS Device Farm test run and retrieve the results when the tests are complete.

We will use the Jenkins Workflow Docker inside step to build an Android app inside a container that has all of the necessary tools. No need to pre-configure any tools on the Jenkins controller or have a statically configured Jenkins agent - beyond having a Docker enabled Jenkins agent. In this case, we are using a Docker image for our Jenkins agent that has SSH, JDK 7 and Docker-in-Docker (DIND ).

Now we need a Docker image for actually building the Android app. There are a number of different docker images available for Android builds on Docker Hub. But if you look at how these images are tagged, you will notice that they typically tagged as latest or with the underlying JDK being used. However, we want to make it easier to interchange and update the tool-chains we are using. Therefore I am using a custom image based on jacekmarchwicki/android and explicitly tagging it with the version of the primary build tool, the Android SDK in this case, included. As you will see in the Workflow script below, I am using the Android SDK version as my image tag: kmadel/android-sdk:24.3.3 . Using a new version of the Android SDK would only require pushing a new image with the necessary updates, say kmadel/android-sdk:24.3.4 , then creating a new branch of the Android GitHub project, and finally updating the Jenkinsfile to specify this newly tagged image: docker.image('kmadel/android-sdk:24.3.4').inside . (NOTE: The use of a Jenkinsfile alongside your source code, that will be automatically picked-up by a Workflow job, requires a Jenkins Multibranch Workflow project .) By using the Docker Workflow inside step, updating the tool-chain to build an Android app is as easy as one GitHub commit!

node('docker') {
    //build Android app in Docker container
    stage 'Build App'
    //checkout AWS Device Farm Sample Android App from GitHub
    git 'https://github.com/kmadel/aws-device-farm-sample-app-for-android.git'
    //tell docker to pull the android skd image, 
    //start that as a container, 
    //and then run the proceeding block of workflow steps
    docker.image('kmadel/android-sdk:24.3.3').inside {
        sh './gradlew assembleDebug assembleDebugAndroidTest'
    }
    //stash successful build, wrapped with the dir step to simplify includes
    dir('app/build/outputs/apk/') {
        stash includes: '*.apk', name: 'app-test-pkg'
    }
}

So, now we have a built and stashed an Android app and an instrumentation test package ready to upload to the AWS Device Farm later in the flow.

You will need AWS credentials (Access Key ID and Secret Access Key) before you can use the AWS CLI Plugin wrap step. And as always, you shouldn’t use your root AWS user, rather you should create an AWS Device Farm specific IAM user with only the necessary permissions to work with the AWS Device Farm. Now, using the credentials of this new Device Farm specific IAM user, we will add a new set of Amazon Web Services Basic Credentials to our Jenkins controller and use the Credentials advanced settings to provide an ID, AWS-DEVICE-FARM-CREDS , that we will use to populate the credentialsId parameter of the AmazonAwsCliBuildWrapper .

Here is the Workflow snippet of using the AmazonAwsCliBuildWrapper with the wrap step:

wrap([$class: 'AmazonAwsCliBuildWrapper',
        credentialsId: 'AWS-DEVICE-FARM-CREDS', 
        defaultRegion: 'us-west-2']) { … }

Also note that we have set the defaultRegion to us-west-2 , because at the time of this writing the AWS Device Farm was only available in that region.

Before we can upload the Android app and test package to the AWS Device Farm, we will need an AWS Amazon Resource Name (ARN) for the Device Farm project that we will want to schedule our test run against. This Workflow job is parameterized with the Device Farm project ARN, but we will create a new project if the parameter is not provided. Here is the Workflow snippet to create a new Device Farm project and to capture the returned projectArn :

      writeFile file: 'create-project.json', text: '{"name": "Jenkins Workflow AWS CLI Device Farm Demo"}' 
      wrap(
        [$class: 'AmazonAwsCliBuildWrapper', 
        credentialsId: 'AWS-DEVICE-FARM-CREDS', 
        defaultRegion: 'us-west-2']) {
            sh 'aws devicefarm create-project --cli-input-json file://create-project.json > createProjectOutput'
        }
        //get project arn from output
        def createProjectOutput = readFile('createProjectOutput')
        def jsonSlurper = new JsonSlurper()
        def projectObj = jsonSlurper.parseText(createProjectOutput)
        projectArn = projectObj.project.arn

We are directing the output of the AWS CLI command to a file so that we can capture the project ARN from the response. By default, the response will be returned as JSON; so we will use the wonderful Groovy JsonSluper to parse it to an object to get the project ARN.

NOTE: Currently you can’t delete Device Farm projects (via CLI, API or the AWS Device Farm web console) and AWS allows you to have an unlimited number of Device Farm projects with the same name. Therefore, if you run this job multiple times without providing an existing Device Farm project ARN, then you will end up with a lot of projects with the same name.

In addition to the Device Farm project ARN, we will also need a Device Pool ARN to actually schedule the test run. The AWS Device Farm includes a globally available Top Devices curated pool that includes a number of popular Android devices (if you are interested, the ARN for this device pool is arn:aws:devicefarm:us-west-2::devicepool:082d10e5-d7d7-48a5-ba5c-b33d66efa1f5 ). For this flow, we are going create a private device pool (and to speed up what are otherwise some fairly long running tests, we will only include one device in that pool).

   //create device pool with just one device for demo purposes - Samsung Galaxy S6 (Verizon)
    wrap([$class: 'AmazonAwsCliBuildWrapper'... {
        sh """
            aws devicefarm create-device-pool --project-arn ${projectArn} --name android-device-pool --rules '{"attribute": "ARN", "operator": "IN", "value": "[\\"arn:aws:devicefarm:us-west-2::device:9E515A6205C14AC0B6DCDBF3FC75BC3E\\"]"}' > createDevicePoolOutput
        """
    }
    ...
    devicePoolArn = devicePoolObj.devicePool.arn

For the sh step above, we are using triple double-quotes to make it easier to escape our AWS CLI command and because we need to include the projectArn variable that we retrieved from the create-project response. If we didn’t have to inject a parameter then we could have just used triple single-quotes.

Now that we have a project ARN, we can upload the Android app and test package that we stashed earlier in the build stage. We are going to upload these two artifacts in parallel, capturing the ARN that is returned in the responses.

    unstash 'app-test-pkg'
    parallel(
        uploadApp: {
            wrap([$class: 'AmazonAwsCliBuildWrapper'... {
                sh "aws devicefarm create-upload --project-arn ${projectArn} --name app-debug.apk --type ANDROID_APP > createUploadAppOutput"
            }
            ...
            uploadAppArn = createUploadAppObj.upload.arn
            uploadAppUrl = createUploadAppObj.upload.url
            ...
            sh "curl -T app-debug.apk '${uploadAppUrl}'"
            waitUntil {
                //wait until upload is complete
                wrap([$class: 'AmazonAwsCliBuildWrapper'... {
                    sh "aws devicefarm get-upload --arn ${uploadAppArn} > getUploadAppOutput"
                }
                ...
                def uploadStatus = getUploadAppObj.upload.status
                if(uploadStatus == 'FAILED') {
                    error 'Upload App Failed'
                }
                uploadStatus == 'SUCCEEDED'
            }
        }, uploadTests: {
            wrap([$class: 'AmazonAwsCliBuildWrapper'... {
                sh "aws devicefarm create-upload --project-arn ${projectArn} --name app-debug-androidTest-unaligned.apk --type INSTRUMENTATION_TEST_PACKAGE > createUploadTestOutput"
            }
            ...
            uploadTestArn = createUploadTestObj.upload.arn
            uploadTestUrl = createUploadTestObj.upload.url
            ...
            sh "curl -T app-debug-androidTest-unaligned.apk '${uploadTestUrl}'"
            waitUntil {
                //wait until upload is complete
                wrap([$class: 'AmazonAwsCliBuildWrapper'... {
                    sh "aws devicefarm get-upload --arn ${uploadTestArn} > getUploadTestOutput"
                }
                ...
                uploadStatus = getUploadTestObj.upload.status
                if(uploadStatus == 'FAILED') {
                    error 'Upload Test Failed'
                }
                uploadStatus == 'SUCCEEDED'
            }
        }, failFast: true
    )

The create-upload command does not actually upload the file. Rather, it returns a pre-signed S3 PUT URL to be used to upload the file. Here we use a simple curl command and upload the app and test package files to the S3 URL that was returned from the create-upload response. We also capture the ARNs for both the app and test package uploads as we will need these, in addition to the project ARN and device pool ARN, for the schedule-run command. Finally, we use the waitUntil step to ensure that the files were uploaded successfully before we schedule the test run.

Now that we have uploaded the app and test package, we can schedule a run to execute those tests.

  stage 'Schedule Test Run'
  wrap(
      [$class: 'AmazonAwsCliBuildWrapper'... {
          sh "aws devicefarm schedule-run --project-arn ${projectArn} --app-arn ${uploadAppArn} --device-pool-arn ${devicePoolArn} --test type=INSTRUMENTATION,testPackageArn=${uploadTestArn} > scheduleRunOutput"
  }
  ...
runArn = scheduleRunObj.run.arn

The instrumentation tests that are included with the AWS Device Farm Sample App for Android take over 16 minutes on average for just one device. So, before we poll for the results of the scheduled run, we will tell the job to sleep for 14 minutes:

sleep time: 14, unit: 'MINUTES'

Now we are ready to check on the status of the run we scheduled:

    def getRunOutput
    def runResult
    waitUntil {
        //wait until upload is complete
        wrap([$class: 'AmazonAwsCliBuildWrapper'...{
            sh "aws devicefarm get-run --arn ${runArn} > getRunOutput"
        }
        ...
        runStatus = getRunObj.run.status
        runResult = getRunObj.run.result
        runStatus == 'COMPLETED'
    }
    echo getRunOutput
    if(runResult != 'PASSED') {
        error "Test Run ${runResult} - see output above for details"
    }

Upon waking up from the sleep step, we will once again use the waitUntil step to poll for the scheduled run results until we get a returned run status of COMPLETED . Upon completion, we can use the run result to verify if the run PASSED or not, and in this case, fail the job for any other response.

As you can see, the combination of Jenkins Workflow, Docker, the CloudBees AWS CLI Plugin and the AWS Device Farm is quite a powerful combination for the building and testing of mobile software.

However, there is definitely room for improvement.

If you look at the entire Workflow script for this job you will definitely see that we are breaking some basic DRY principles - specifically around capturing the response output from the AWS CLI commands. Jenkins Workflow includes support for a Git repo based Global Library as documented by Nigel Harniman - Jenkins Workflow - Using the Global Library to implement a re-usable function to call a secured HTTP Endpoint . In this case, we could create a global library function that would encapsulate the parsing and capturing of the AWS CLI responses.

Also, there is quite a bit more we could do to complete the entire software delivery pipeline for this Android app. For example we might deploy the app to a beta testing platform and once it has been manually verified, we could allow human testers to push a button to deploy it to Google Play. Again, this example just scratches the surface of the Continuous Delivery awesomeness that you can achieve with Jenkins Workflow!

Kurt Madel
​Solutions Architect
CloudBees