How Big is Big

Stephen Connolly's picture

I caught a fish this big
“I caught a fish this big” by Mark Hanna

When we talk to Jenkins users, about half of the people we talk to will say something like:

“We have a really big Jenkins instance”

This is a statement that always intrigues me… quite often when we probe a little into exactly how big their Jenkins instance is, we find answers like “more than 500 jobs” and “five or ten build agents”. Now some people are probably laughing at the thought of such an instance as “really big”, but that is never my reaction.

Growing a Jenkins instance is, in some ways, very much like growing a company. When I started in CloudBees, there were very few employees… in fact we all were listed on the Company Team page… everyone knew everyone else.

When your Jenkins instance starts off, you only have a few jobs, maybe one or two build agents. Everyone knows what every job does. Everyone knows what every build agent is for.

After a while, the company grows… there are more people… the Company Team page no longer lists everyone… not because people have stopped being important to the company, but because those kinds of pages should not list all 60 or 70 people. It’s still a village though. Everyone knows everyone else to see, even if not personally.

In the same way, as the Jenkins instance grows, we see people start to use things like Views to customise what they see. Everyone knows who owns each of the jobs, even if they don’t know exactly what they are for. Everyone knows who owns the build agents, etc.

At some point in the growth of a company, it will transition from the “village” phase to the “town” phase. This is the hypothesised Dunbar’s number… which is rumoured to lie somewhere between 100 and 250. This can be a breaking point for some companies. Thankfully, the CloudBees executive team have all gone through this before and as soon as they put me back on the Company Team page any potential problems will have been resolved (joke!).

As you grow your Jenkins instance to the 500+ jobs range (which will typically require 10+ build agents) you start to hit the same issues that a growing company hits. No longer will Views solve organising all your jobs. You need to start putting jobs in folders. You probably will need naming conventions. You certainly will need more complex permissions systems. In short, you feel pain.

To grow your Jenkins instance from 500 to 1,000+ jobs will require you to start to solve some of these problems, and until you realise that there are various solutions to these problems, it can feel almost as if you have hit some fundamental limit of scalability for Jenkins.

So while 500 jobs or so and 10 build agents or so is not a big instance, to the person who has grown their instance to this point, it can feel really really big.

But what is a really big instance then?

Well obviously I have the answer. I researched the subject as part of the preparation for my Jenkins World 2016 talk: “So you want to build the worlds largest Jenkins cluster”.

Jenkins instances have a built in “phone home” facility that reports usage statistics to the Jenkins community home page. The reporting of statistics is anonymised, only includes some basic information and can also be turned off by the Jenkins administrator. But there are enough instances reporting that we can at least put a measure on actual really really big instances.

For the latest reported statistics (covering the month of August 2016):

  • There were 61 instances with more than 8,000 jobs. The instance with the largest number of jobs had 67,563 jobs.
  • There were 19 instances with more than 450 agents. The instance with the largest number of agents had 6,794 agents.

Those are considerably bigger numbers than 500 jobs or 10 build agents and getting to those sizes will have required a different management strategy from that used when initially growing.

  • The instances with many build agents probably have some sort of automation to create their build agents. The agents are probably all nearly identical and created from some sort of template image or recipe.
  • The instances with many jobs are probably using some sort of automation to create the jobs. This may be as simple as using something like the GitHub Organisation Folders plugin so that all the organisation’s GitHub repositories are scanned, with jobs being auto-created for each branch of each repository with a Jenkinsfile (as well as Pull Requests). Or it may be more complex automation using the CloudBees Templates plugin or the Job DSL plugin. But in any case, there are very few “special snowflake” jobs.

So, in summary, today in 2016 a “really big Jenkins instance” is probably one with either more than 10,000 jobs or more than 1,000 build agents.

Blog Categories: 

Add new comment