The Butler and the Snake: Continuous Integration for Python by Timo Stollenwerk, Plone Foundation

This is the first in a series of blog posts in which various CloudBees technical experts will summarize presentations from the Jenkins User Conferences. This first post is written by Félix Belzunce, solutions architect, CloudBees.

At the Jenkins User Conference/Europe, held in Berlin on June 25, Timo Stollenwerk of Plone Foundation presented how the Plone community uses Jenkins to build, test and deliver Python-based software projects. Timo went through some of the CI rules and talked about the main tools you should take a look at for implementing Python CI.

For open source projects implementing CI, the most important thing besides version control and automated builds is the agreement of the team. In small development teams, that is an easy task most of the time, but not in big teams or in open source projects where you need to follow some rules.

When implementing CI, it is always a good practice to build per commit and then notify the responsible team as to the outcome. This makes the integration process easier and avoids “Integration Hell.” The Jenkins dashboard and the Email-ext plugin could help accomplish this. Also, the Role-based Access Control Plugin could be useful to set-up roles for your organization, so your developers can access the Jenkins dashboard while being sure that nobody can change their job configuration.

Java developers usually use Gradle, Maven or Ant as automated build tools, but in Python there are different tools you should consider, like Buildout, PIP, Tox and Shining Panda. Regarding testing and acceptance testing, I have listed below some of the tools that Timo mentioned.

Due to Python’s nature static analysis has become somewhat essential. If you plan to implement this in your organization, I recommend reading this article, which compares different tools for Python static analysis, some of them which Timo also mentions.

Regarding scalability, when you are running long builds you could start facing some issues. A good practice here is not to run any build in your master and to let your slaves do the job.

If you have several jobs involved in launching a daemon process, you should ensure that each job uses unique TCP port numbers. If you don’t do this, two jobs running on the same machine may use the same port and end up interfering with one another. In this case, the Port Allocator Plugin can help you out.

The CloudBees Long Running Build Plugin and the NIO SSH Slaves Plugin could also be helpful if you want to restart a build (in the case that Jenkins crashes) without starting from scratch or if you want to increase the number of executors attached to your Jenkins master while maintaining the same performance.

In the release process, Timo explains that the Jenkins Pipeline plugin could be combined with some specific Python tools like zest.releaser or devpi.

Get Timo’s slides and (when videos are posted) watch the video of his JUC Europe session.

Félix Belzunce
Solutions Architect

Félix Belzunce is a solutions architect for CloudBees based in Europe. He focuses on continuous delivery. Read more about him on his Meet the Bees blog post.