Introduction to Jenkins X Pipeline

Editor’s Note: John Ha also contributed to this blog post. 

A brief history of continuous integration in Jenkins

When someone asks me what Jenkins X is, I typically say, “Jenkins X is Jenkins continuous integration and continuous deployment (CI/CD) re-imagined for a cloud native world on Kubernetes.” But what does this mean?

Jenkins is the world’s most popular and deployed general purpose automation server. In 2016, CloudBees created and released the first iteration of the Jenkins Pipeline plugin. This plugin was the first step toward creating more comprehensive workflows for continuous integration in Jenkins. Since then, the plugin has been updated to support a domain specific language (DSL) in the form of the Declarative Pipeline syntax. This syntax provides a guided and prescriptive way of authoring your Jenkins pipelines for continuous integration.

Over the last few years, the world has begun to embrace Kubernetes and cloud native infrastructures built on top of it. Each cloud provider now has native support for running Kubernetes workflows. Kubernetes enables you to create applications and services that scale up and down gracefully based on traffic and resourcing.

As organizations have started to move their applications to the cloud, they’ve also been trying to determine how to use Jenkins in a more cloud native way. However, while the Kuberentes plugin for Jenkins allows you to use Kubernetes to scale your Jenkins agents, your masters do not automatically scale.

Cloud native, to put it simply, is a concept that implies your applications are designed to run in smaller pieces (often called micro-services) and to be deployed by using orchestration systems like Kubernetes.

Enter stage left, Jenkins X

Jenkins X provides pipeline automation, built-in GitOps and preview environments to help teams collaborate and accelerate their software delivery at any scale. Jenkins X fully embraces Kubenetes by automatically scaling your CI workflows to support many developers working concurrently to commit, preview and promote code.

Michael Neale’s blog post about Jenkins X gives a wonderful overview of how Jenkins X works and how simple it is to get started. There has been one significant development to Jenkins X since that post: Jenkins X Pipeline.

The Jenkins X Pipeline

Before the Jenkins X Pipeline was introduced, Jenkins X ran pipelines by default by having a Jenkins master running in a pod, waiting for triggers from git. This workflow used a Jenkins pipeline (via a Jenkinsfile) and supported only a subset of the Jenkins Pipeline workflows. Testing has also shown that pipelines do not run as fast as the Jenkins X team would like them to, causing the team to work on developing a pipeline syntax and execution engine that would natively support Kubernetes-based pipelines.

This discovery and development is what has led to Jenkins X Pipeline. The Jenkins X Pipeline is both a cloud native execution engine and a syntax. Jenkins X pipeline execution is built on top of Tekton Pipelines. CloudBees engineers worked closely with Google, Red Hat, IBM, Pivotal and others in planning the Tekton and Jenkins X pipelines, and both pipelines are now part of the Continuous Delivery Foundation. Tekton is the low-level pipeline execution engine for Kubernetes, and Jenkins X pipeline builds on top of that to provide user-facing features and workflows for cloud native pipelines.

All Jenkins X pipeline steps run in containers and all agents are ephemeral Kubernetes pods. This means that Jenkins X can now use Kubernetes cluster autoscaling to handle load spikes dynamically (that is, as more developers commit code and submit pull requests, Jenkins X scales to support preview environments and pipeline runs). This architecture diagram shows how Jenkins X pipelines work in Kubernetes.

Comparing the Jenkins X Pipeline Syntax to the Jenkins Declarative Pipeline Syntax

The following example shows a Declarative pipeline in Jenkins.

pipeline {

 agent {

   docker {

     image ‘maven:3-jdk-8’

   }

 }

 env {

   SOME_VAR = ‘some-value’

 }

 stages {

   stage(‘Build’) {

     steps {

       echo ‘Building!’

       sh ‘mvn -B clean package’

     }

   }

   stage(‘Deploy’) {

     steps {

       echo ‘Deploying!’

       sh ‘./deploy.sh’

     }

   }

 }

}

In Jenkins X Pipeline, these same stages and steps look like this:

pipelineConfig:

 pipelines:

   release:

     pipeline:

       agent:

         image: ‘maven:3-jdk-8’

       env:

         - name: ‘SOME_VAR’

           value: ‘some-value’

       stages:

         - name: Build

           steps:

             - command: ‘echo’

               args: ‘Building!’

             - command: ‘mvn’

               args:

                 - ‘-B’

                 - ‘clean’

                 - ‘package’

         - name: Deploy

           steps:

             - command: ‘echo’

               args: ‘Deploying!’

             - command: ‘./deploy.sh’

You can also review the Jenkins X integration pipeline to see a real world Jenkins X Pipeline that runs each time that a new commit is applied to Jenkins X.

What does this mean for the future of Jenkins?

Jenkins Pipeline will continue to run as it does today and support all its syntax: both Scripted and Declarative. Providing a purely YAML-based pipeline syntax for Jenkins X helps ensure that Jenkins X pipelines are always consistently formatted and easily versioned. It also adds a way for development environments, such as Microsoft VSCode, to be able to have Jenkins X pipeline extensions in the future to validate your syntax in real-time. Finally, you will be assured that your pipelines will run as expected since your syntax must be valid YAML and must use the supported functions in Jenkins X pipelines.

Just like Jenkins Pipelines with Jenkinsfile, your Jenkins-X.yaml file is stored as code, within your source code repository along with the code for your app or service. Ultimately, Jenkins X pipelines are the cloud native form of Jenkins pipelines, with mirrored support for most Jenkins pipeline functions and cloud native constructs for Kubernetes. The team are working daily to add more functionality to Jenkins X pipeline, and you can contribute your ideas in the Jenkins X GitHub project.

CloudBees is dedicated to continuing to improve the Jenkins and CloudBees Jenkins Distribution experiences, while helping create a clear path to Jenkins X from Jenkins for those workflows that make sense to move into Kubernetes. General purpose automation and Jenkins Pipelines are first class citizens, along with Jenkins X.

To learn more about Jenkins X and creating a Spring Boot application, watch this video by James Rawlings. 

 

Additional resources