Scaling Jenkins Horizontally with Jenkins Operations Center by CloudBees

The recent Jenkins survey [1] showed that 82% of users consider Jenkins critical for their organization.

Approximately 91.5% of Jenkins support 2+ projects and 42% support 10+ projects. Furthermore, 30% of respondents worked for companies with 100+ developers.

So what do these numbers mean for Jenkins administrators toiling away to keep the agile factory running?

Admins are constantly adding more and more teams to the software factory. Thus, admins are in the business of making their instances resilient to failures and scaling them to onboard more teams.

Jenkins is scaled by adding build nodes to the Jenkins instance while beefing the machine that runs the Jenkins master. Said differently, admins scale their Jenkins master vertically. Although, Jenkins scales well vertically, there is a limit on what can be done on one box [2].

And as we know, horizontal scaling trumps vertical.

Jenkins Operations Center by CloudBees makes it possible to scale Jenkins horizontally. The product (affectionately known at CloudBees as “Jockey”) is the operations dashboard for Jenkins in the organization.

The Jenkins Operations Center server (which is just a Jenkins master with some plugins on it) effectively becomes the central “cloud” provider for build nodes for all downstream masters attached to it.

Downstream masters rent a build node when they want to run a build. Thus with Jenkins Operations Center, admins can add a new master when a team comes onboard or just create a new master if performance dips on a particular Jenkins master.

About 32% of organizations have more than one master floating around (usually one master per group) and about 65% of these have more than one build node. Since the Jenkins Operations Center server consolidates build nodes, it drastically cuts down CAPEX costs entailed as admins can move away from the model where each group buys more build nodes to bolster their master.

The Jenkins Operations Center server becomes the central launching pad to jump into and browse jobs on downstream masters. There is SSO setup, facilitating easy access to downstream jobs.

One other point, the Jenkins Operations Center by CloudBees server can optionally push down security realm and authorization roles to downstream masters. Thus, it effectively becomes the single point of security configuration, as opposed to setting up configurations on each master separately. To take it further, you can also configure plugin centers per master and manage them from the Jenkins Operations Center server.

Watch some videos [3] to see a quick overview of the product and how to set it up.

I believe Jenkins Operations Center by CloudBees is going to fundamentally change how admins architect scalability and resiliency for their Jenkins instances. No more single point of failure due to an overloaded Jenkins master!

There is more (Jenkins Enterprise by CloudBees)…

We have also released a new version of Jenkins Enterprise by CloudBees (November 2013 release). The release has two main customer RFEs: a) capability to create templates per folder and general improvements in the Templates plugin, b) CLI support for the Role-based Access Control plugin and the ability to quickly setup security with pre-defined roles. If you want to know more, there is additional information in the user guide and release notes. [3]

Next steps:

I suggest reading through the excellent tutorial [5] on how to set up Jenkins Operations Center by CloudBees. Note: this sets up a highly available Jenkins Operations Center server and highly available downstream masters. Download the free 30-day evaluation copies for Jenkins Operations Center and Jenkins Enterprise [6].

Key Takeaways:

  1. Scale Jenkins masters horizontally making CI infrastructure resilient to failures
  2. Share build nodes between masters. Reduce CAPEX by sharing instead of buying build nodes
  3. Share security configuration between masters in the organization

References:

[1] Jenkins 2013 survey results (link). All graphs in this blog are from the survey
[2] View Kohsuke’s presentation for JUC 2013 where he talks about the X1K goal of making it feasible to run 1k executors on one master (link)
[3] Jenkins Operations Center videos: two-minute introductioncomprehensive overviewattach a client masterattach a shared slave
[4] Jenkins Enterprise by CloudBees release notes and user guide
[5] Jenkins Operations Center by CloudBees tutorial
[6] Free 30-day evaluation links for Jenkins Enterprise by CloudBees and Jenkins Operations Center by CloudBees

 

- Harpreet Singh
Senior Director, Product Management
CloudBees
www.cloudbees.com

Harpreet has 15 years of experience in the software industry. Prior to CloudBees, he was at Oracle and Sun for 10 years in various roles, including leading the marketing efforts for Java EE 6 and GlassFish 3.1. He was also product manager for Hudson, launching it within Sun’s GlassFish portfolio.
 
 
 
 
Blog Categories: 

Comments

Hi,<br /><br />The operation center instance is the operations dashboard and acts as the central authority in the cluster. Can it be a client master or slave at the same time?

It cannot also be a client master. You *could* create Jenkins jobs on it directly, but this is definitely discouraged. You could also run a slave agent on it, but this is also discouraged, for the same reasons running builds on any Jenkins master is discouraged (security and performance).

Add new comment