Template Catalog: Welcome to the Pipeline as Code Family

Pipeline Template Catalogs joins forces with Shared Libraries (Figure 1) to become another member of the Pipeline as Code family. Both use external source control repositories that enforce consistency, best practices, and enterprise rules without overhead across an organization’s  Jenkinsfiles. The Pipeline Template Catalog extends the Shared Libraries to ease the collaboration between different personas who are involved in the Continuous Integration (CI) and Continuous Delivery (CD) within the same organization. Please note that this feature is only available for CloudBees CI. 

Figure 1: Illustration of the Shared Library and Pipeline Template Catalogs fusion which enables Pipeline as code.

Some of the benefits of using Template Catalogs are:

• It provides developers with a simplified and directed experience for configuring Pipeline in “domain specific” terms that make sense for an organization. Thus, developers can focus on shipping products.

• It categorizes Pipelines per projects and teams. Similar-looking jobs that only differ in a few aspects. When you change a template, all the uses of that template get updated instantaneously.

Shared Libraries and Pipeline Templates Catalogs make an interesting synergy. Jenkins administrators use the Shared Library to host the Jenkinsfile code (and resources) which is fed by the parameters provided by a template from a Pipeline Template Catalog. In this way, developers would not need to maintain any Jenkinsfile (and Groovy) for their project repos.

This demo presents a template that model the pipeline Continuous Integration process in the Acme Corporation.  The pipeline defines Maven Apps of the corporation than are deployed as Docker containers. It integrates CloudBees CI and GitHub Enterprise via webhooks. Builds are managed by Kubernetes agents. Finally, successful builds create an image that embeds the project artifacts and it gets uploaded to Dockerhub (Docker-in-Docker).

To run this demo there are a set of requirements listed here. In particular, the following items were used:

• CloudBees CI on modern cloud platforms 2.204.2.2 (Kubernetes plugin 1.21.4 and CloudBees Pipeline: Templates Plugin 3.3)

• GitHub Enterprise server. Authentication is done in CloudBees CI via Jenkins Credential: user and password (API token) 

• DockerHub as Container Registry. Authentication is done in the agent side via Kubernetes secret

There are three personas involved in this demo

1. The Kubernetes administrator who creates the Kubernetes objects as well as defines the Kaniko pod agents yaml. 

2. The CloudBees Jenkins administrator who codes the Pipeline Home and configures the CloudBees CI.

3. The Developers who set values for the designated template by a team and/or a project

Not only the Template Catalog and Shared Libraries are complementary to each other but also compatible in their code structures (Figure 2). This compatibility is why they can be integrated into a single point of control: “The Pipeline Home”. The Pipeline Home, for the purposes of this demonstration, is a repository which hosts both Shared Libraries and Pipeline Template Catalogs and it can be found at cloudbees-oss/pipeline-home-demo.

(root)
+- catalog.yaml # Catalog definition
+- vars
|   +- foo.groovy # for global 'foo' variable
|   +- foo.txt # Help for 'foo' variable
+- templates             
|   +- bar               
|       +- Jenkinsfile # Configuration file for bar
|       +- template.yaml # Template’s parameters for bar
+- resources
|   +- agents
|       +- k8s # Agent k8 resources description
|              +- agent.yaml # Pod Agent description
|   +- cli
|       +- pipeline-template-catalogs
|              +- catalog.import.json # Import definition for the Catalog  
|   +- pocs          
|       +- templates
|           +- bar           
|              +- Jenkinsfile # POC for bar

Figure 2: Source code structure for “Pipeline Home”: Integration of Template Catalog and Shared Libraries into a single repository. Note: The above diagram represents only the elements that were used for this demo, for example, the src folder is omitted.

Figure 3: A Blue Ocean Jenkins UI capture showing the POC job stages

I recommend creating a Proof of Concept (demo: /resources/pocs/templates/docker-java-maven-app/Jenkinsfile , Figure ​​​​​3) before writing the template itself. If you find any issue with loading the POC into CloudBees CI, you might refer to Pipeline Development Tools for debugging.

For the POC the repo https://github.com/jenkins-demo/simple-app was imported to a GitHub Enterprise server.

The principles for this POC design are:

• The Shared Library is loaded dynamically for easing the portability to other instances

• The agent is load as libraryResource from the Shared Library /resources/agent/k8s folder which hosts the Kubernetes agent pod templates available for the CD/CI server. The agent is deployed into a different namespace than the controller node. It makes uses of pvc for the maven cache which speeds up build times

• The parameters are defined as environment variables  in the pipeline and they are encapsulated into a yaml object (configYaml)

• Docker Images are tagged by the commit SHA from the project checkout. Alternatively, they can be tagged as the latest

Once the POC behaves as expected we are ready to move to the next step in our Pipeline as Code journey.

Figure 4: A Blue Ocean Jenkins UI capture showing the Pipeline Template job stages

For this demo, a new pipeline multibranch template (demo: docker-java-maven-app-m-ghe , Figure 4) was added into the CloudBees catalog template example.

The principles for these templates design are:

• The pipeline code is moved to the Shared Library vars and it is categorized per project type (demo: mavenK8Project)

• GitHub Enterprise is used as branchSource which automates the webhooks creation for the continuous integration process (Figure 5)

Figure 5: GitHub Enterprise Integration with CloudBees CI via webhooks

• As this demo uses Maven as a building tool, markerFile: pom.xml is defined. Meaning that any branch containing a pom.xml at the top level will be recognized as a buildable Jenkins project (for this reason the controller branch is not built for the simple-app)

• The form input validation (Figure 6) comes from the Shared Library (gitUrlValidatorstr and​​​ strNoSpacesValidator) and it happens before the call to the Pipeline mavenK8Project.

Figure 6: A Blue Ocean Jenkins UI capture showing a Pipeline Template job build that fails because the Git Repository Name value parameter is invalid according to the gitUrlValidator validation.

• CloudBees CI does not provision any agent until the build passes the Run stage which means that it has not been ABORTED (Figure 7) when code changes come from the Template definition (Jenkinsfile and/or template.yaml). At the moment of writing this demo, Suppress automatic SCM triggering is not available for Templates but it might be in the future to tackle this concern. Update: Since CloudBees Pipeline: Templates Plugin 3.6 additional configuration options for Multibranch Pipelines are exposed, including Suppress automatic SCM triggering (NoTriggerBranchProperty) so this workaround is not longer needed.

Figure 7: A Blue Ocean Jenkins UI capture showing a Pipeline Template job build  that is ABORTED because git changes come from the template definition

UPDATE: Since CloudBees Pipeline: Templates Plugin 3.6 multibranch pipeline templates in pipeline template catalogs are now able to configure a branch property strategy in template.yaml, including Suppress automatic SCM triggering. As a result, the Skip Run stage would not be not longer needed as long as the multibranch section of the template.yaml is defined as follows:

...
multibranch:

  branchSource:

    github:

      apiUri: "https://github.beescloud.com/api/v3"

      repoOwner: "support-team"

      repository: "${ghe_repo}"

      credentialsId: "${ghe_cred}"

    strategy:

      $class: DefaultBranchPropertyStrategy # All branches get the same properties

      props:

        - $class: NoTriggerBranchProperty # Suppress automatic SCM triggering

  markerFile: pom.xml # Optional if you are using maven

  scanRepositoryInterval: 2 hours

Although Pipeline Catalog can be loaded via GUI (Pipeline Template Catalogs option in the left-hand side menu), this demo has opted for importing it remotely via Jenkins CLI. Browsing to $CBCore_URL/cli list all the available commands for the CloudBees CI instance, including pipeline-template-catalogs. The Jenkins CLI HTTP mode was used. Firstly, JENKINS_USER_ID and JENKINS_API_TOKEN were defined as environment variables. Then, the call to the CLI command pipeline-template-catalogs happens, passing the templates-definition.json (demo).

$> java -jar jenkins-cli.jar -s $CB_Core_controller_URL pipeline-template-catalogs --put < templates-definition.json

{ "message" : "Successfully imported Pipeline Template Catalogs.", "status" : "SUCCESS" }

Figure 8: Terminal showing the usage of pipeline-template-catalogs

To validate that the templates from a catalog have been correctly loaded/updated into CloudBees CI all following requirements must be met:

1. Pipeline Template Catalog > Import Log must show Creating (or Updating) Pipeline job template imported from the source code.

Figure 9: Pipeline Template Catalog Import logs in CloudBees CI UI 

1. Go to Catalog Name (it should be shown as Healthy) and all the templates should be listed there

Figure 10: Pipeline Template Catalog items in CloudBees CI UI 

1.  The Template job should be listed under the Create a New Item list of options

 Figure 11:  Create New Item Menu in CloudBees CI UI 

Now it is time for developers to configure the respective template per type of project.

Figure 12:  Configuration form of the job based for the demo template in CloudBees CI UI 

As a result of the successful build, the Docker images are stored in the desired registry (Docker Hub, for this demo).

Figure 12:  Dockerhub showing the image repository for simple app with its list of tags

The Pipeline as Code demo was inspired by the simple-app project of Adrien Lecharpentier so thanks for this idea. Also, thanks to Ryan Campbell for encouraging me to write this post. Finally, to the CloudBees Pipeline Team for its review.

• Pipeline Template guide from the CloudBees documentation

• Using Kaniko with CloudBees CI guide from the CloudBees documentation

• Kaniko project Docs

• Share a standard Pipeline across multiple projects with Shared Libraries from the Jenkins Blog

• Jenkins Kubernetes plugins examples from the Jenkins Kubernetes plugin documentation