Inside the 5th Annual DevOps and Jenkins Community Survey with Kohsuke Kawaguchi and Kristen Baskett

Brian Dawson: Hello, this is Brian Dawson with the DevOps Radio and today we're going to talk to you about the Fifth Annual DevOps and Jenkins Community Survey conducted by CloudBees and here to talk about that with me, I have Kohsuke Kawaguchi and Kristin Baskett, both of CloudBees. Hello, Kohsuke.

Kohsuke Kawaguchi: Hello.

Brian: So why don't we start out by introducing yourselves, telling our listeners a bit about you. Now, I know really Kohsuke, not to embarrass you, but you need no introduction. That said, I'm still going to ask you to tell us a bit about your role with Jenkins and your role with CloudBees.

Kohsuke: Hey, thanks. So, yeah, and I'm Kohsuke. I'm the creator of Jenkins. I've been working on this project for a long, long time now, 15 years or so. I am currently a chief scientist at CloudBees and what that means is I get to work on some fun projects.

Brian: Awesome. Thank you. Kristin, can you go ahead and introduce yourself, tell us a little bit about your background?

Kristin Baskett: Absolutely. Hello everybody. I'm Kristin Baskett. I’m a solutions marketing manager here at CloudBees and I've had the pleasure of working on this year's Jenkins Community Survey along with Mr. Dawson, our wonderful host who's been working on the survey for the last couple of years. So it's been a really fun projects and I can't wait to talk about it.

Brian: Cool. Thank you, Kristin. Thanks for the plug. So there's a lot of interesting data in the survey as we've all discussed, and I'll dig into that in just a moment, but before we do, as we know this is the DevOps and Jenkins Community Survey and for the first four years, we had called it the Jenkins Survey. So I wanted to start with this opportunity to ask the creator of Jenkins how has Jenkins enabled DevOps? How is Jenkins relevant to DevOps today?

Kohsuke: Right. I think when I started what became Jenkins, I think originally the forecast was much narrower. You know, most people just generally focused on using desk automations. But as the things that people do kept expanding, that's I think I can say we can all see that in this report as well, people started putting a little more interesting workload, automating a lot more and then so, you know, I think one of the reasons I think that Jenkins stayed relevant for so long and continues to do so is because you know, I think Jenkins is kind of always busy whoever you are, wherever you are, or wherever you're going. So I think that still remains true and yeah, so maybe for me, that's kind of the key pillar.

Brian: Okay. Well, and so the Jenkins project was founded at this point some years ago. I guess we just had our 10 year anniversary, or the Jenkins community had its 10 year anniversary and this is , I think we'd all agree, a rapidly changing landscape, especially in regards to tooling. How does Jenkins stay relevant? What is Jenkins doing to embrace or drive the future of CD and DevOps?

Kohsuke: Yeah. I mean, as I said, _____ well, 10 seems to be the only constant. I think that's actually true in Jenkins itself, right and you can see it in some of the like _____ the way the community evolves or community around itself, you know, this whole idea around plugins, for example, as very much this idea that everyone can kind of innovative in the direction that they want it and collectively we are able to explore this space very rapidly. And I think that's still true, although I think the _____ has changed somewhat. Now we are no longer _____. I don't think the _____ more plugins is the focus of the new innovations. I think we're taking on the bigger efforts. So, you know, things like that Jenkins X, that's a major project. That's taking Jenkins in a very interesting new direction with a new set of people, new use cases, new set of audience, etcetera. We also recently started _____ the _____ foundation. That's another sort of like a new, I think a new evolution and then I hope among many things, I hope they improve the collaboration and the _____ and so on with the other tools in this space. You mentioned this earlier  you said _____ important change _____ from just Jenkins to DevOps Survey and I think it's reflective of the fact that the span of this practice now, it goes beyond one tool. So I think this will be actually relevant in that sense.

Brian: Right.

Kohsuke: So I think those are just a few of the examples of the evaluation that's currently happening. There's a lot of interesting efforts going on.

Brian: And to comment myself, I'm excited, really excited about what CDF will do for Jenkins and the community overall. I think one thing that we saw is in people communities as well as commercial vendors creating best of breeds, point solutions, which offers a lot of power in terms of flexibility, for a lot of people it also adds complexity. 'Cause, not necessary all these solutions, do all these solutions talk together well. So the whole idea that the Continuous Delivery Foundation can increase the interoperability for Jenkins and the other best-of-breed tools I think is pretty powerful and pretty promising.

Kohsuke: Actually because it's funny, thanks for saying that. In some ways, like, some of the things for me that the CDF is actually inspired by you, Brian.

Brian: Wow.

Kohsuke: You talk about this continuous delivery as a practice, not in terms of like tools and how to use Jenkins and then that was, I mean, going back to the days that you came on board, that's something that you did and then that was something, for me, that was very new. Over time, I started to understand and appreciate the need to up level the message. I feel like the practitioners, the users of these tools, like they know that these efforts, _____ and continuous delivery maters, but I often, when I talk to them, they often feel like their organizations are not rallying behind them. Like, they are not, they are failing to leverage that. They have to fight that.

Brian: Yes.

Kohsuke: And I felt like what Brian, you are going after is you're trying to talk to people who don't take this for granted, take that this is a good practice that's granted and then you need to explain the business value and so on and so forth.

Brian: Right.

Kohsuke: So that's for me, like another part of the CDF and the stuff.

Brian: I guess you triggered a couple of things. One, you described my approach to continuous delivery in DevOps in a way that is better than me, in a way that I didn't recognize, but I also think that's what's so powerful about CDF. I'm going to shift gears a little bit and, for our audience or for our listeners, again, we're talking about the fifth annual or 2019 DevOps Community Survey. If you haven't seen it, there should be a link in the show notes. Absolutely go visit CoudBees.com. Download it. There is a very rich and informative infographic, interactive infographic, along with a powerful opening from Kohsuke that I think really helps kind of frame this idea that some of us are doing well, some of us aren't. This transformation to continuous delivery in DevOps is a challenge and we're all in it together. So to jump into the survey a bit, let me go ahead and move over to Kristin. Kristin, we had talked earlier and you said there were a few things in the survey that really jumped out at you as being interesting. I'm going to want to ask both of you, but why don't I start with you, Kristin? What did jump out at you?

Kristin: Sure. So just to back up a little bit and set the stage. You know, when we put these surveys together, like you said, it's the fifth year, so I had the pleasure of digging through the last four years of data when we were writing this survey, and the decision this year to open it up to include more DevOps topics really surfaced some pretty cool data. So in previous years, we talked a lot about the technology, a lot about how practitioners are using Jenkins and one of the things that I see in my role in solutions marketing is there's a lot of best practices that arise out of this community and I spent a good bit of time actually working with a professional services team and we found that these best practices, these people and process pieces of DevOps are really, really strong in the Jenkins community, and that bore out in the survey results. So, those are the two areas that I want to talk about. And before we go there, Brian, do you want to talk a little bit about how we defined our high velocity practitioners versus kind of the general survey respondents because I think that'll help.

Brian: Yeah, absolutely. You read my mind, Kristin. So to talk a bit about that, as we talk about the survey data or as you as a reader go through the survey, I think where you'll find a lot of value is the fact that we've defined a group called high velocity practitioners that we've compared to the general population in the survey. And this was to allow us to really kind of suss out and identify where the people that are succeeding, what is it that they're doing. So we can all walk away not only with a better understanding of the maturity of the industry, but with some clues and guidance on how to do better ourselves. So this high velocity practitioner group represents the top 25 percent of the general population in the areas of deployment frequency; depth of automation of their continuous delivery process or their software delivery lifecycle; deployment approach, such as big plane deployment versus incremental canary, blue, green, et cetera; and level of cross-functional collaboration. So we identified this group, this very similar I think if you look at the Dora report, very similar to the elite performers. And these are people that have outcomes where they measure in the top 20 percent that are indicative of successful implementation of continuous delivery in DevOps. So yeah, thanks for bringing that up, Kristin, because I think that was important, and I'm actually going to take this opportunity to redirect a little bit and ask you to address what was another sort of linchpin or underpinning of the report. And this is the section that is titled Defending Against DevOps Washing, where we found that 67 percent of people report practicing DevOps, which is a 20 percent absolute improvement over those in 2017, which is really positive. But what's interesting, and I'd like you for you to comment on, and Kohsuke if you have any comment, chime in, we found that while 67 percent claim they're practicing DevOps, only 49 percent claim they're practicing CD. So, you know, I asked, do you know, you know, what are your thoughts as to why this is? Do you believe that people can practice DevOps without practicing CD?

Kristin: Yeah, I think that there is a lot of varying descriptions of what it is to practice DevOps and I think it's worth noting that DevOps kind of being the very attractive buzzword that it is, everybody wants to align with that in the same way that they wanted to align with Agile. How I see it is to practice DevOps well, you really need to be doing CD and so it's an interesting gap in that respect that there's a lot of people who proclaim to practice DevOps, but aren't practicing CD and that's an interesting gap. Another interesting thing about the data was the prevalence of the practice of CI. So this is something that we see gaining a lot of traction. So within the Jenkins community, CI is really pervasive. Eighty-one percent are essentially claiming to be practicing CI this year. So that's a really positive sign for those practicing DevOps that they are practicing CI. But there's certainly some room for improvement in CD, but it's also worth noting that CD is picking up too. So in 2017, 38 percent said they were practicing CD and so we're up 10 plus percent this year and that's a significant growth trend.

Brian: Yeah, that is I think really positive uptick. Kohsuke, you got a comment?

Kohsuke: Yeah. I think I'd be saying more or less the same thing but in part, you know, because these terms are fuzzy and so I think some of that, some of the answer probably fix that, but I guess one thing I'd point out is there are lots of other software development domains where the concept of DevOps doesn’t really apply like if you're producing a packaged software, then by definition, you don't have ops. Like if you have a mobile app, that's the main thing you're doing, you still don't have any of the ops in there, at least on the mobile side as well. For me when I see like a 67 percent is practicing DevOps, I don't see it as, oh, we need to come use the rest of the 33 percent. I think it's important if we have that diversity of the ecosystem. But like Kristin was saying, the number has grown substantially over the past few years. _____ indicates it's continued more, so I think the _____ and then this _____ are probably doing it. So I think more and more people are working farther and farther in their journey and then some people are pushing things pretty far.

Brian: That actually leads me up well to go another section in the survey, particularly your comment about some people not having ops, Kohsuke. So when we asked about how do you unlock the flow of value, i.e., how are you automating, what have you automated? We divided automation of your pipeline into what we'll call kind of upstream development and downstream delivery. And it's interesting that we found 95 percent of people automate their build and test activities, and another positive improvement that's up 74 percent from last year. However, we found that only 64 percent automate their downstream activities. Now, huge improvement over last year. We went from 18 to 64 percent of people that are automating the downstream activities, but Kohsuke, I'm going to be curious if you have a comment when we compare the number 84 percent are doing CI, 49 percent claim to be practicing CD, but only 64 percent, 31 percent less downstream activity is being automated than upstream activity. What are your thoughts? My first thought is, well if you're truly doing CD, then you're downstream activities such as deployment, smoke testing, et cetera, should be automated as well. You have any thoughts on that?

Kohsuke: Automation isn't a binary thing. In my own experience, it's more like kind of like a never-ending thing. So, maybe some people thought, well, maybe different people _____ a different threshold and they needed _____ _____ kind of _____ but the, you know, I think when I look at that page,  the report, what I see is there's a huge uptick in the downstream automation activities, whereas the upstream is kind of saturated. Like almost everyone has basically automated the basic things like building and testing. So that, I think, it certainly consistent with the CM. _____ what I observe out there. I guess testing actually has a lot of different aspects, releasing, deployment, more like a way more downstream automation and _____ existing _____ automation. All that seems to be a lot of focus.

Brian: I'll add an observation that really drove this stat home for me in the past couple of weeks and that is, as some of you may know, and of course, Kristian and Kohsuke, as you know, CloudBees recently acquired Electric Cloud and Electric Cloud is a leader in what the analysts call the CD continuous delivery release automation space or continuous delivery application release automation space. And one of the things I found in having a conversation with Electric Cloud is that a significant percentage of their buyers and users are actually release automation and operations teams. And a lot of the value proposition is that Dev has adopted agile and continuous delivery and they're moving faster than ops, so what we can offer with what is now CloudBees Flow, is the ability for operations to come up and operate on the same cadence. Now, the interesting indicator is, more so than I had even understood, people were automating up until the point that they needed to deploy, and then there was more of still yet a throw it over the wall to operations. Generate binaries, put in a binary repository and then ops does whatever they do from there. So I thought that interesting.

Kohsuke: I mean it kind of resonates to me. Back in the early days, in most places, where I used to work, that led to Jenkins, the engineering, like the software developers and QA are like different teams and then the way initially this automation like I got this stronghold is like say, they each automated their own thing and they connected those automations. At that point, it sort of became more and more, they started evolving so like a co-evolving, more of a one practice. So when I see like a devs things, it’s automation on the inside and obviously automation on the inside side and then when the automation connects I think _____ the automation _____ I think people _____ realize that, it’s sort of like keeping the distinction so explicit and visible isn't necessary or that useful.

Brian: Well, I hope putting my CloudBees hat on, I hope having Jenkins CloudBees Core and Electric Cloud in the same house helps us blur that line of distinction and connect those two things. Building on that point, Kristin, I wanted to go to you and ask you to talk a bit about, so we've talked about automation, the change in automation, but our top 25 performers seem to have successfully achieved automation in some specific areas. Do you think you can talk to that a big, Kristin?

Kristin: Sure. So, you know, our high velocity practitioners, that top 25 percent, we found some interesting things that they are automating at a greater rate than kind of the average group. And the first one of those is governance. So, these high velocity practitioners are automating governance at 30 percent versus the average of 18 percent, and that kind of shows you how the mindset is shifting a bit. So we're seeing it shift from, “I'm automating very CI specific things” to, “I'm automating more of the life cycle. I’m automating outside of what the traditional developer bubble” and kind of moving towards the business, and I think that's an important distinction. The other thing kind of along those same lines is we're finding high velocity practitioners automating security at a much higher rate, so these folks are automating security at 38 percent versus the average of 23 percent. And when we talk about security, one of the really interesting things that also came out of the survey, not in this section. Now I'm on a tangent, but one of the really interesting things that came out of this is you know, those people who are integrating security practices, it's not actually slowing them down any. And so you know, it's not impacting their time to release. So what we found was practitioners with integrated security practices, they're deploying once per week or more and those high velocity, that high velocity group was at like 88 percent versus an average of 56 percent. So that's another instance where we see this high velocity group really kind of glomming onto the bigger business idea. And then the last section that I found really interesting about what these guys are automating, they're automating a lot of release and so we saw this with the previous data plan as well where CD is picking up 10 plus percent. What was interesting about the CD activities that are picking up though was that 64 percent of automated release, up from 18 percent in 2017, and that number is just at the survey average. And so that's an enormous gap that's being filled with the adoption of CD, and that's pretty exciting. And we see that in the high velocity practitioners too where they’re up at like 84 percent of them are automating release. So, if I were to pick like one area of this report that's truly transformational with the adoption of CD, I mean, this is it. These high velocity practitioners in the Jenkins community at large is really adopting release automation and that's pretty cool to see.

Brian: Awesome.

Kohsuke: Yeah, that makes sense. I mean, when you go to these big conferences, you see all these I guess _____ _____ and all the other scans and so on all over and they are all talking about, you know, the _____ thing like integration is the software development process. And so, people are having that, people are taking that and creating, that's resulting in the higher performance. I think it makes sense.

Brian: I think we're starting to see an interesting theme here and, you know, a lot of stuff is aligning. As Kristin called out, well high velocity performers are taking on some more of the business processes, which I think we don't run into when we just implements CI/CD within the confines of development, but as we truly move to implement CD, through what as you called out Kohsuke, were traditionally organizationally siloed or separate teams, we have to take on new concerns, and some of those are usually heavier _____ right, security, governance and high velocity performers have taken that on and tackled it. It also ties to a theme that you brought up earlier with CDF. It's for this to really take hold and drive organizations to excellence, we have to acknowledge that we have to get out of our technical development and operational silos, and as you said, what did you say, and educate and convince the people that don't take it for granted.

Kohsuke: I mean, the security, I think that's a great example. It used to be the ops things, like the part of the reason the company steals this process and getting stuck over the wall is that was a point in _____ happens and _____ _____. Now if you have both sides automated, it's kind of _____ if you're a two year old that you _____ think the other points _____ not mention.

Brian: So, actually a good place to move on from there, Kohsuke, is to talk this idea, when we spoke earlier, you called out some of the things in the report that jumped out to you. And one of the things you brought up in terms of high velocity performers is how they are or aren't scaling the implementation of these practices. Do you want go ahead and talk to that a bit?

Kohsuke: Right. Yeah. So, let me see. I think in one point of the report where it talks about how to scale up the CI/CD beyond one team and that being a key, let's see, here, so that seems to be the key, one of the key differences between the high velocity practitioners and the rest of the pack and that this is making that point and it's close to me because when I go out, when I go out and talk to these software development teams, that usually seems to be the kind of like the main focus of the strategy goal. It's not just – I mean, those are kind of fundamentally, well, there are some technology challenge but more often than not, it's not technology challenges. It's uniquely organizational. And that's something like I had zero appreciation when I started Jenkins. Like I thought _____ this _____ out there, like anyone can use it and be successful. But it turns out that in order to make a truly bigger impact, like beyond one team, then you need to think about all sorts of other things like the consistency and the uniformity of the process, governance and stuff like that.

Brian: Yeah, that is an awesome point. And I'll call out, hey, you, as I know, I think you're aware, Kohsuke, that I often go out and talk about kind of this four quadrants of DevOps maturity. And for listeners, there's team level implementation of these practices as I've defined it, work group level and then enterprise level, sort of most of your organization. And I am consistently finding that people are now reasonably equipped to overcome the challenges of going left to right in automating their SDLC and implementing these practices. But it's scaling it and propagating across the organization where people consistently fall over and run into trouble.

Kohsuke: Yeah, and that's kind of like the journey map that the people really want to see and compare notes with others in the same boat. I feel like because they have these people either by themselves or they have people under them that can solve the technical challenges.

Brian: Right, but how do you solve the other challenge? So Kristin, can you talk to me a little bit about what you saw under people and processes. Cause as we're hearing, the tools and the technology are only one part of it. I think we also have to address people and culture, process and practices, and the survey query data bit. Can you comment on what we found?

Kristin: I think we found some really interesting things, actually. So that's why I'm so excited that we kind of broadened the scope of the survey this year. The first and biggest thing that we saw was really around collaboration, where our high velocity practitioners, nearly all, 99 percent, and that's not a stat that you see often, are collaborating to plan upcoming work. And this is compared to kind of the superset group where the average is 74 percent. So the big takeaway there is collaborate, collaborate, collaborate. And of course, where you can find tools and technology that helps support that collaboration, even better. The other thing that we found that was pretty interesting was high velocity practitioners spend a lot of time prioritizing their metrics. And kind of getting back to the comparison here between our high velocity practitioners and what Dora is doing, we're seeing a lot of the same trending here. So 84 percent of our high velocity practitioners are capturing software delivery process metrics, and this is up over the average of 66 percent for the average group this year. And so by and large, you're seeing the majority of this top 25 percent, hearing a lot about really understanding the DevOps metrics, the work that they're doing. And furthermore, there's the motivation aspect there. So it's 50 percent of our high velocity practitioners are using Jenkins to automate their metrics reporting and that's up over 35 percent of average.

Brian: Wow. And you know, something brings to mind there, it's probably overly used, but the Drucker quote, but it is absolutely relevant, especially at this change over industry maturity, and that's, you can't improve what you can't measure. So it sounds like the data is sussing that out, that probably part of why high velocity performers have been able to do better is they're doing better at capturing metrics and presumably using those to improve.

Kristin: Yeah, absolutely. And we saw that too, actually. So we saw that of those high velocity practitioners, those who were tracking for more metrics, are actually experiencing less unplanned work. So that's bringing their unplanned work down by eight percent. So turns out, Drucker was right.

Kohsuke: And then this is like using data to drive these sorts of development process, that's something I lately am very interested in and I _____ conference talk, so it's exciting to see this number going up and I actually see _____ more like not _____ in this area and then more exciting things that can be done. So I think this is the interesting part is for the investors, so the 19 and onward.

Brian: No, and then so as we get ready to wind down here, before we move into wrapping up, I just want to give an opportunity to ask each of you, starting with Kohsuke, is there anything else in the report in particular that jumps out at you that we haven't covered yet?

Kohsuke: So, on the lighter note, you know, this is like almost highest, like a well-produced _____ survey report done ever, so thanks for Kristin and Brian and others that put this together 'cause I know there's a lot of work that went into this and I _____ like _____ what goes in the sausage making process, so that was awesome. I mean, the very first community survey we've done was just basically Excel charts printed out as a PDF. So, this came a long, long way. So congrats on that.

Brian: Thank you. Thank you. And Kristin, is there anything else that in particular jumps out at you that you want to call out?

Kristin: Sure. I mean, no additional data point to share, but I do want to drive home how pervasive and influential the teams who are using Jenkins are becoming, whether it be the uptick in governance, the kind of bleeding over into the DevSecOps space, that huge uptick that we saw in automated release, you know, it's a pretty big deal and with the acquisition of Electric Cloud, we've really seen the importance of release automation. They are kind of the go-to solution for the ARO and ARH space, so release orchestration and automation. And you know, only good things to come out of that partnership.

Brian: Awesome. Awesome. And so, you know, I will call out kind of pointing towards a conclusion, and then handing it off to you, Kohsuke, to send it home, where the report ends up is identifying a couple of things, key things, and that's that high velocity practitioners, the top 25 percent, deploy more often than the average or the general population which as we understand, means they can iterate faster. They can deliver value faster. They can get feedback faster and better improve. You know, and in cases of failures, they're also able to recover faster. So, by deploying more frequently, iterating more, they gain a very clear advantage in the market. We also found, as Kristin called out, that they're spending less time on unplanned work so they can increase their productivity, they can reduce their waste and something we often don't talk about is you know, we become developers, we become software practitioners to solve problems. And when we unleash the process so that we can get things to market faster, what we've consistently found is that you have more satisfied developers. So with that, what I'd like to do, Kohsuke, is hand it to you to tell our listeners, tell us what this survey, what this report means and how they can use it to their benefit.

Kohsuke: For me, one of the most exciting things about a report like this is, it helps people in the sense of this accomplishments in their own journeys, right, like I mean, if you only talk hear about what the vendors are picking and the kind of you know, I think there's a certain element of mongering, like fear mongering. And I kind of, that's a part of it that I do not like. So whereas like when you look at how other people in this journey is doing. You can clearly see like everyone is, like we are all making forward movement in wherever we are on the journey. And I think that I just kind of wanted to pause and congratulate all of us on the back that hey, you know, that _____ like amazing things are happening around the world and what we are doing is actually making an impact in those tangible ways and I think this number will hopefully help you convince the people around you that what you've been doing matters. I think that's kind of how I wanted to end this.

Brian: Well, thank you Kohsuke. That is I think a very important, often overlooked, point. So I appreciate that and I appreciate your time today on DevOps Radio, Kohsuke Kawaguchi, the man who needs no introduction and Kristin Baskett, and thank you to all of our listeners for joining us today.

Kohsuke: Thanks, Brian.

Kristin: Thank you.

Like what you’ve heard today? Don’t miss out on our next episode. Subscribe to DevOps Radio on iTunes or visit our website at CloudBees.com. For more updates on DevOps Radio and industry buzz follow CloudBees on Twitter, Facebook, and LinkedIn.