One of the biggest goals of a continuous development workflow is to find and fix bugs earlier in the development cycle. Automatically smoke testing every deployment helps developers quickly validate new changes and catch bugs before they make it to production.
In this post, I'll walk through setting up automated post-deployment smoke tests in a Codeship pipeline using Assertible. I've put together an example Ruby app that is deployed to
production environments on Heroku and validated with Assertible.
Continuous Delivery with Codeship
Codeship is a continuous integration and delivery service that allows you to build, test, and deploy code after every commit. Codeship makes it easy to get started by removing the need for
yaml configurations and complex scripts. This, combined with the Assertible deployments API, gives developers an easy way to set up test automation for any web app.
Let's start by outlining the steps of the pipeline:
Push code and build it with Codeship
Deploy the application to Heroku
Smoke test the app with Assertible
The primary goal of this process is to create a completely automated build, deployment, and smoke-testing pipeline that will catch application errors during development by testing the app on a live
Connect a repository
The first step is to connect Codeship to a repository on GitHub. To do this, create a new project in your Codeship account and enter the Git clone URL of your repo:
Now every time code is pushed to the connected repository, the changes will be pulled and built by Codeship. Learn more about connecting repos in the docs.
Create a deployment pipeline
A deployment pipeline is specific to a branch in your repository, like
controller. Multiple pipelines can be set up to handle deployments from different branches; for example, the Ruby app is deployed to production from the
controller branch and to staging from any
wip-* branch. The deployment pipeline is initiated only after all of the build and test steps have completed.
The example Ruby application is set up with the following pipelines:
For commits on the
controllerbranch, deploy the app to production.
For builds on a branch that start with
wip-*, deploy the app to a staging environment.
In Codeship, this is called [Wildcard Branch Matching](https://documentation.codeship.com/basic/getting-started/deployment-pipelines/#wildcard-branch-deployment-pipelines) and is extremely useful for handling deployment of the same app to multiple environments.
Testing a web app on
staging environments is an effective first line of defense to find bugs early in the development cycle, before they make it to production. The process outlined above provides automatic deployments to an isolated staging environment where the release can quickly be checked and validated.
Learn more about Managing Multiple Environments for an App in the Heroku docs.
!Sign up for a free Codeship Account
Smoke Testing with Assertible
With a deployment pipeline in place, you can set up Assertible to smoke test the application immediately after it's deployed. This approach has several benefits:
Quickly validate potential releases
Catch bugs earlier in the development cycle
Tighten the feedback loop between QA and developers
Prevent wasting time on testing broken applications
The Assertible Deployments API can be integrated with the deployment pipelines to run an automated test suite against both the
Define and test a web service
The first step is to define your web service in Assertible. Log in to your Assertible Dashboard and enter the URL of your web service or import your Swagger Spec. A test will automatically be created for the URL, and you can add more tests for other critical endpoints.
For the Ruby API example, I've created a test for each endpoint that will validate the core functionality of the app and fail if anything is critically wrong.
Ideally, the smoke tests are fast and check core behavior to identify a faulty deployment. The assertions made against the Ruby API include:
If any of these assertions fail, you will be immediately notified of a bad release.
Run tests after a deployment
Now that the web service is defined, we can go back to the Codeship deployment pipelines and add a second step to run the smoke tests. The Assertible deployments API makes this possible by tracking releases of an application and initiating tests against new deployments.
Add a Custom Script step to the deployment pipeline that makes a
POST request to Assertible:
This step is called after the first step has successfully deployed the application to Heroku. You can use a simple
cURL command to send the details of the release to Assertible, which will then initiate the tests for that service.
CI_COMMIT_ID(provided by Codeship)
Testing deployments on staging
Assertible supports the concept of environments for a web service. An environment is a separate URL where the same application may be deployed, like a
The example Ruby API has two environments:
The example application is deployed to both
production. Tests can be run against a specific environment by specifying the
environmentName in the
A Complete Deployment Pipeline
Everything is now set up for Codeship to build the code, deploy to Heroku, and run smoke tests against the application. As code is pushed to the repository, you will hopefully see passing tests in Assertible and a green build in Codeship:
Bonus: Status Badges
Once your continuous delivery and post-deployment testing pipeline is set up, you'll want to show off how all your builds and tests are passing.
Codeship and Assertible both offer status badges you can use in a
README, documentation, or anywhere else to let your team and users know the status of your web services.
There are many great resources to learn more about setting up and extending this pipeline. Check out some of the articles below:
Do you have any questions, comments, or corrections? Hit me up on Twitter. I'd love to hear about your deployment and testing workflow. To get started building and testing your code, sign up for a free Assertible account and Codeship account.