Eli Lilly’s Nick Liffen Embarks on the Jenkins Journey

Brian Dawson: Hello. I'm Brian Dawson. Thank you again for joining us on another one of our special editions of DevOps Radio where we are live at DevOps World | Jenkins World 2019 at Moscone in SF. I have here with me Nick Liffen. I did not confirm how to pronounce it.

Nick Liffen: You nailed it.

Brian: Awesome, as I usually do.

Nick: Everyone always says Liffen so I'm actually very surprised you got it right. I'm very impressed.

Brian: Thank you, so we're starting off on the –

Nick: On the right foot.

Brian: All right, good, good. So you are a software engineering team lead at Eli Lilly?

Nick: Yep. So I have been at Lilly now for just over three years and I came into Lilly as a software developer, software engineer and quickly moved into product ownership.

Brian: Oh interesting.

Nick: So I moved into product ownership towards like Heroku, Artifactory, and most recently Jenkins so I am defined as the product owner for Jenkins and was responsible for standing up the Jenkins service. And more recently I've moved into a software engineering team lead so I oversee a team of eight or so developers and we focus on standing up Jenkins Shared Libraries, Jenkins Pipelines so within that DevOps space.

Brian: So let's dig into that for a minute. Product owner, I'm sure that name wasn't – that title wasn't adopted haphazardly. I'm going to take a guess and correct me that it's different than being a Jenkins admin or a tool admin. You guys I'm guessing it's about a tool providing a capability and an experience, right?

Nick: Yeah, so let me set some context. I think if I explain a bit about our team that helps explain. So I work for a team called Cirrus, which is our immersion technologies team, and our team is responsible – we're a central team and we face off to our business-facing IT teams and we're responsible for standing up modern technologies, modern tools, modern platforms, helping to disrupt and emerge different processes. So we have so many different tools. We actually have like 42 tools, something like that within our team –

Brian: That's a number we've been trying to capture. That's – and there are people with more but 42? Wow.

Nick: And we have a headcount of about 20 people so we have in our team tech leads, we have product owners, and we have ops. So a product owner is responsible for overseeing the overall design of the product and also it could be a new product and they're responsible for standing up that service.

Brian: Okay I love it and there you're right. So if someone's coming from a traditional mindset we hear product and product owner, we don't traditionally translate it to open source or other internal or dev tools so that's interesting and I assume that there's a power – words matter, there's a power –

Nick: Responsibility.

Brian: Okay.

Nick: So I think you have that power as a product owner to decide the roadmap of your tool so how are you going to offer. So within Lilly, when we were standing up Jenkins, the product owners and myself had the responsibility – power is a strong word. I think it's really you have the responsibility to make sure that you're standing up a solution to meet your end customers' needs.

Brian: And I misused power so that's important as well. I think there's power in the selection of – we use the word product. In other words, language informs actions.

Nick: Sure, yeah.

Brian: Right and if something isn't – if we stop just calling it a tool, which means I get something, I install it, and I had it off to some guys to use –

Nick: Operations team, yep.

Brian: But rather I own an experience and what I'm doing is designing – I'm either providing an experience or service or capability; it's not just dropping off a tool.

Nick: And I think that's so important nowadays, like in my mind a lot of tools' platforms software is no longer being considered that one-time, okay I need to stand it up and then hand it over to our operations team. I think especially in an enterprise, in a large company, the mindset is starting to shift towards everything is being treated as a product because you need to drive that experience not just to start with but things change. We live in a world, especially in IT, where every hour now you're seeing there seems to be a new tool to do the same thing so we're starting to drive that product mindset, especially within our team where you can't just think about sort of standing it up and that's it. You are responsible. You have the power to make sure that you're meeting that experience.

Brian: Awesome. I love it. I'm glad we stumbled into that in the conversation. So something I knew coming in I wanted to ask you about is, as a product owner of Jenkins, is your experience with an investment in Jenkins and I'd like to start on that path by understanding your take on pipeline as code. To what extent have you guys applied it at Eli Lilly and to what benefit and then whatever else you'd like to share?

Nick: Yeah, no, that's a very open-ended question.

Brian: Yes, exactly. I love it. Let you do the work.

Nick: So pipeline as code in my mind is fundamental to the way that Lilly has stood up our enterprise Jenkins service. Now I'm going to go down three different groups if that's okay.

Brian: Love it, yes.

Nick: So the first route that I'm going to start down is so Lilly has three different types of personas and they are a touchpoint. So we have three different types of Jenkins users within our enterprise. We have persona one. Persona one are people who don't really have the expertise to be able to build Jenkins pipelines or in my mind they don't have the desire and that is what I'm most interested in. So teams within Lilly, they understand the value of DevOps but DevOps to them should be a commodity. It should be something that they make one request and I have my pipeline pre-built ready to go so that's persona one. We have persona two. Persona two are people who are, "Hey, we have some niche use cases where I do want to build my own Jenkins pipeline but I don't want to manage any Jenkins infrastructure or I don't want to manage any Jenkins jobs. The value for me in Jenkins is building that pipeline." So that's persona two. Now persona three is I want to do the whole shebang. I want to own my own masters, I want to own my own pipeline, I want to own my own –

Brian: I need root.

Nick: I need root. Give me everything. So there are three different personas. Now the reason why I framed up with that is pipeline as codes is fundamental, especially through use case number one, which is persona one and persona two. And I'll dig into a little bit of that now. So persona one what we have done is we have stood up a type of manage master called a controlled managed master. Now that's terminology that Lilly uses so obviously Jenkins has managed masters. We have an opinionated managed master, which is called our controlled master. Now on these masters, teams can request jobs and they get pre-built pipelines for them. So let's say where we see common deployment patterns like a team is deploying a container to Artifactory, they're deploying a Node.js application to Heroku. Teams make one request and then they get these jobs and these pipelines fully built for them and it's fully automated. All of the pipelines on our Controlled Managed Masters are fully as coded.

Now we as you go one step further, we have them all stored in GitHub and that's been I think one of the biggest wins for us is being open about processes and being open about where you find documentation, about where you find these pipelines. Having to define this code is one thing. Having to define this code and storing it within GitHub or storing it as you would in your source code management system is so important because it allows that collaboration and it allows that sort of inner sourcing.

Brian: Yeah, when that's in place. Now let me ask before you go on because I think there's more beyond a Controlled Managed Master right?

Nick: Yep.

Brian: Controlled, I'm curious to ask do you provide the persona one control goes to persona one?

Nick: Yep.

Brian: Do you provide them the right access to the repo?

Nick: Very good question, no.

Brian: Okay.

Nick: So basically –

Brian: And they don't want it and they don't –

Nick: And they don't want it because that is a commodity to them. So all they need to do is we use Service Now as our service request portal. A team could go to Service Now and say, "Hey, I want a Jenkins job. I'm deploying to Heroku and it's a Node.js application." And they say, "Hey, here's my code. Here's my Node.js application. Here's my GitHub URL." Now it will ultimately create a job on that Heroku Controlled Managed Master and it will link to our Jenkins pipeline. They have no admin rights over that job and they have no admin access over that pipeline so they can't go in and start changing the pipeline. They can't say, "My build is failing. I'm going to remove this."

Brian: Right.

Nick: So Lilly gets the value of, we know that there's been a consistent deployment pattern but secondly they get that DevOps experience at the click of a finger and I think that's really where Lilly has seen the biggest value is in truth 60 to 75% of our jobs across our whole enterprise Jenkins service lives on Controlled Managed Masters.

Brian: Oh, that's interesting.

Nick: Because teams are starting to realize DevOps is something that we need and teams think that they have the experience and they think that they want to go out and build pipelines. But the value to them is building their applications; it isn't building DevOps. It isn't building pipelines. It's building the application and then having the continuous delivery, having the unit test, having the security tests, the open-source tests just at a finger for them. And that's the culture that we're trying to promote within Lilly is we want to shift left in the way applications are built but also we want to shift left in the way that DevOps is being done and we're trying to move everyone over to these controlled masters.

Brian: That's interesting because, as our listeners will hear and you'll hear eventually later, there's been a lot of discussion and debate between how much do we need to engage teams to get their own unique needs and create understanding trust empathy, which hopefully we'll hit culture in a bit, but then how much is it that that's just too much. People just want something that works and I think it's a balance. Go ahead.

Nick: And so one thing that we do, so I mentioned controlled masters and they come in sort of a pre-built Jenkin Pipeline.

Brian: Yes.

Nick: One thing – so when we first released this service we got some really good feedback being like, "Hey, this is good, I love all these checks, but I have – I do have my own use cases. I want to introduce one or two of my own stages into this pipeline."

Brian: Yes, okay.

Nick: And so one thing that we've done is that about two months into the service we actually made a bit of a change. So halfway through our enterprise pipeline, we actually pause and we say, "Is this application team bringing their own Jenkins Pipeline? Yes." If they are we run that pipeline as well as our pipeline. So think of it as we provide the base blueprint so we provide the foundational service around you have to run these –

Brian: These major gates.

Nick: Correct. Think of them like we have six or seven gates that are mandatory that you have to do them, not because we are here to enforce them because they're a value to you. You want Synk running your open source packages. You want a coding stance so you want ES linked to them. These are our mandatory gates. But we understand and we sympathize that you may have your own gates that you want to run so we will also pause our pipeline halfway through, see if you're – if you're not bringing a Jenkins file we will just move on. If you are bringing a Jenkins file we will run the stages within your pipeline so you almost get a little bit of the best of both worlds.

Brian: Okay.

Nick: Does that make sense?

Brian: Yes. No, it absolutely does and in fact, I have to get more up-to-date on the project but that very much fits with the Jenkins community project that CloudBees was helping drive called Blue Ribbon where you basically are using pipeline as code to create stubs, right?

Nick: Yep. I think a good way to think about it is like Lego blocks. So like what you want – you want teams to almost be able to have their base foundation and then simply be able to add Lego blocks to say, "Hey, there's this bit of code. I want to drag this in and there's this bit of code."

Brian: Right, I want the yellow block, the blue block, the square block, yeah.

Nick: And then you start to – as teams start to be like, "Okay, I can more accommodate this service because yes, I have to run the managed stuff but I can also put my own flavor," and that's what we call our controlled managed master.

Brian: Okay and there is – I'm going to go on a quick change, I love the Lego just to resonate on that, there is DIY developers, the visual of it is they just want a box of Legos with no instructions oftentimes, figure it out, right? That's your persona three.

Nick: And developers don't want to sit there and read 10 pages of documentation. They want an easy and quick to use service that they can be like, "Okay that one, that one, that one, that one, plug in, run DevOps," and that is persona three. You’ve absolutely nailed it. That is indeed persona three. So if you do want to use our pipelines you are more than happy to or if you're persona three and you're like, "Hey, I want to do my own thing. Get out of my way," what they want is they still want the shared libraries and –

Brian: Still want modules, they still want things that snap together and stuff, right.

Nick: So even though our shared libraries are the base and the foundation of our Controlled Masters they can also be used, they're an optional process on the self-controlled and the freestyle master.

Brian: Awesome. You know what reminds me that we're doing this interview live is the smell of popcorn waft in. You smell that?

Nick: And I love popcorn; I'm not going to lie.

Brian: I do too. So I'm curious what tools or processes have been instrumental for Eli Lilly to undergo what we'll call a DevOps transformation? Is it fair that we're sharing what you guys are going through or in the midst of is a DevOps transformation?

Nick: I think transformation is a fantastic word. I think we are nowhere near finished. I don't think we've – I don't think we're ever going to be finished.

Brian: Right.

Nick: I think the DevOps transformation certainly is always going to sort of run on and even when you think you're getting there something new will come along to be like, "We can do better and better," and I think that is actually the whole – DevOps is a mindset. DevOps is a process and like you should really always be trying to improve your DevOps process to a point that you're 80% done what can you be doing differently.

Brian: Right and that bar should keep moving. So that's going to lead to a question that I'm going to put on hold for a minute, but I'm going to take the path to say so we talked a bit before this interview and I'm going to ask when we talk about a successful transformation or getting underway what's most important, people/culture, process practices, tools and technology?

Nick: It's a fantastic question and I would love to say them all. If I was to pick one, people and culture is so important. You can stand up the best tools, you can stand up the best technologies in the world. If you don't have the people to stand up the technologies or even consume those technologies what is the purpose of standing up those tools and technologies? And yet you do see companies go out there and be like, "We have this DevOps toolchain and it consists of these 20 fantastic tools," but fundamentally they don't have the people or the education to be able to actually consume these services. So one of the things that we do to like – I work for a Fortune 500 company that we mentioned has been around for 145 years.

Brian: 145? How many people by the way? Stay on track but…

Nick: So we have a total of 38,000.

Brian: Okay, to give scale is important.

Nick: And of course Lilly has about 1,000 in IT. Culture is always going to be something that is going to be difficult and it's going to take time to do. I would love to say that we have the best culture out there to promote DevOps. I think we are getting there. I think this is a journey that we're on and this journey that we're on to stand and I can give a few examples.

Brian: Yeah.

Nick: So recently our team focuses on continuous education and continuous training so alongside standing up the tools, Cirrus is fundamentally – its purpose is to help stand up tools and help evangelize them. We focus on the education side of things so recently we ran a Jenkins day and this is something that has actually been really successful and what a Jenkins day is is we had CloudBees come in and we had done a bunch of presentations in the morning and this was to internal Lilly people so anyone in Lilly was able to come to the session, we gave about a month, two months' notice and the whole purpose of the day was to help train and help educate our enterprise on hey, this is Jenkins, this is how we've stood it up; let's give you time to learn and understand why we've done things but also give you time to ask questions. So we had the whole morning focused on us giving presentations on this is what we've done, this is how we've stood it up. CloudBees did a presentation on the external market, where Lilly is in that journey, some of the good, some of the bad, and then we spent the afternoon doing a hands-on workshop so everyone in the audience got their own managed master, started to create jobs, and out of that day purely because we spent the time focusing on education and focusing on culture we actually increased by about 6% just –

Brian: Wow, your adoption rate and maybe –

Nick: Yeah.

Brian: Yeah, yeah.

Nick: And I think that's – 6% might not sound like a lot but I think that's a tremendous –

Brian: It's statistically significant based on the sample size.

Nick: I completely agree. So I think that it's really important if a company is early in their journey and they're listening to us and being like, "Okay, I can relate," I would really focus on education. I think people and culture, if you can build the people and culture the tools and technologies will come because you're setting your people up for success and your people will then go and stand up the tools and start consuming the tools.

Brian: I agree. Now I am precariously diplomatic –

Nick: Please.

Brian: But I can also see the perspective. So I spoke to a guest earlier today that had what would sound on the surface to be a completely contrasting opinion and that's – and it's a very engineering-centric approach and that's that you know what, the cognitive disconnect to address the soft science of people or culture without a foundation of tools and technology it's not going to happen and likened it to saying, "You know what, before people can learn how to drive and change lanes and merge and navigate we've probably got to build a highway first."

Nick: I 100% agree. If you don't have that person though how is that person going to build the highway?

Brian: Right.

Nick: How is the person going to build the highway? And I completely understand that and it's a very good point but what you don't want it to do is you don't want to try and skill your people out, your people up, without the needed training materials, without the needed process, without the needed – it's like you don't want to train people on Jenkins on day one.

Brian: Without having a place –

Nick: Where they can train and they can fail. So you have to stand up tools and platforms but people have to stand up tools and platforms.

Brian: Right and then people then cyclically people are going to be the ones to get the best of it.

Nick: Correct. So I think one thing that we've done okay is we have some fantastic people at Lilly and I think like our team has stood up the needed highway, in your case, and we're now letting people drive on the highway.

Brian: Yes.

Nick: So for me, I think it is a bit of both but in my mind, you need the people.

Brian: Yeah and there's one thing that is immutable in a successful – you cannot, it's non-negotiable that at some point whatever the order of precedence, the order of operations, people have to be trained up, the people part, but you have to institute and model and ultimately practice a culture of education, learning.

Nick: Yeah and I do want to back up. If you don't have the tools and the technologies then training your people is going to be difficult.

Brian: Right. There's no there there.

Nick: I do so whenever people ask that question I think it is – for me my answer would be people, people, culture, and process, but 100% without the tools and the technologies it is very difficult to train successfully.

Brian: I think we maybe solved this conundrum a bit or we got there. So how do you know, how have you known – actually I'm going to put it in a more challenging way since we're having fun late in the day.

Nick: Please.

Brian: Well you said you're about 40% along.

Nick: Yep, I think we're about 40% along in our journey.

Brian: How do you know you're 40% along? How are you measuring that?

Nick: I think that's a fantastic question. So let's start with what zero is.

Brian: Okay.

Nick: In my mind zero in the DevOps transformation is you aren't doing your automated testing, you aren't doing any security testing, you aren't doing any continuous deployment; everything is manual. You have a lot of manual processes, you haven't got the workforce, everything is still very much monolithic compared to where you want it to be.

Brian: Okay.

Nick: Now describing 100 is near on impossible because like we've already mentioned I don't think there is 100.

Brian: I agree.

Nick: But the reason why I think we are 40% done, so we have stood up an enterprise Jenkins service that runs roughly 1,900 to 2,000 builds a week. And out of that, we have roughly I think 65% of the builds pass and 35% of the builds fail. Now where I compare that to February 2019 where we had like 60 – I can't remember the number, 60% of the builds fail and 40% of the builds pass, I think it's really interesting to see in the space between June and where are we now, August, we have actually shifted from more failed builds passed to more passed builds passed if that makes sense.

Brian: Yes.

Nick: So that's one thing. Now the reason why we aren't more, I think there's still so much more that we can do to drive a better service for our end users. One of the things is so right now we run our cloud-based Core on OpenShift and OpenShift has been fantastic, we've run it. The problem in my mind is we do run it on-premises. So our cloud-based Core, our Jenkins instance runs on-premise on OpenShift so we get the value of Kubernetes but we are limited by our computes. We are limited by how easily we can scale. So sometimes builds can take longer than what we would hope and what we would expect. Some of the things that we are fundamentally missing in my mind are some of the auto-scaling, so right now we've had to build out our underlying infrastructure to support what we could be at our maximum load.

Brian: Right.

Nick: Where I would love to be is we need to be at a place where we're at our minimum load where it's always down at zero and we auto-scale our compute up. Now we aren't there today. That's a big factor for me around what I would like to improve.

Brian: And you can access those capabilities more easily if you move from an on-prem approach –

Nick: To like an AWS. Now in my mind, if you don’t mind I'm going to carry on.

Brian: No, go ahead.

Nick: In my mind and we spoke before about DevOps being a commodity to our end users within Lilly, I have still have to manage an underlying Kubernetes infrastructure so I still have to manage an OpenShift instance. I am not an infrastructure company. I do not want to manage an underlying OpenShift and an underlying infrastructure service. I would love to start to move to a SaaS version because, to me, I want to move as far off the stack as possible. So you link it back to the question around be – us being around 40% done. I think if we could move to it a bit, the way that we're working now in a software as a service we are increasing that DevOps transformation because we no longer have to manage the underlying infrastructure and we can fully focus on building out shared libraries, building out pipelines, and enabling our business partners. We don't have to worry about managing the underlying infrastructure.

Brian: I love it and I love that analogy. It is just the same thing you're trying to do for your customers. Let them focus where they want to focus so they can work on stuff that matters to them, not stuff that doesn't, so move to the cloud, pass that off. I have to – and I know we're smelling popcorn, there are muscle test hammer machines, meatballs coming out, so I know I have to get us out of here, Nick. I think you and I could talk for a while. I first want to ask who is your DevOps hero? Do you have a person or persons that you think are important to DevOps either to your or the community?

Nick: So I don't know if I have a single person that I look at and go – I think there are so many great examples. In my mind it's the people contributing to our open-source packages, the people – in my mind too they don't take all the credit, they don't take all the fame, but they are out there helping make Jenkins better by making poor requests. And there isn't one of them, there are probably hundreds of them, there are thousands of them that are out there every day and they are trying to improve the platform even though they have their own problems, they have their own work that they need to work on. They are out there helping make Jenkins better. So who I look up to and who I want to advocate for are the people who are better contributing to our open source community.

Brian: Awesome, awesome answer and I think I'm not going to dress that up anymore, just repeat. Our heroes, and we have to pay respect, support, acknowledge the open-source contributors or the contributors to Jenkins and really open-source software that's been the underpinning of DevOps. I'm glad you brought that to light. As we get ready to wrap up do you have any final thoughts to share with our listeners to help them along their journey or to give them things to think about?

Nick: So I think if I was to do any closing thoughts, be open and what I mean by that is if you are – if you're listening to this, if you're watching this and you are early on in your Jenkins journey and you're about to stand up possibly an enterprise Jenkins service or looking to stand up CloudBees, one of the biggest things that I think has helped us is making sure that our customers, we've brought them a lot in the journey to start with. We spent about three months doing research on the different personas and helping understand the personas has really helped us build a tailored and personalized Jenkins service. So be open with the people that you want to build the Jenkins service for.

Brian: Awesome, great advice. Hey Nick, it was a joy talking to you.

Nick: Absolute pleasure.

Brian: Thank you for joining us at DevOps Radio, thank you for joining us at DevOps World | Jenkins World, and thank you for sharing your knowledge.

Nick: Thank you. It's been so much fun.

Brian: All right. Thank you, everybody, for joining us. See you next episode.