Ari LiVigni - Open Source, Open Shift, Open Sesame

Sacha Labourey: Welcome at the DevOps Radio even so once again we're also on YouTube which is great. I'm here with Ari LiVigni from Red Hat. Welcome Ari.

Ari LiVigni: Thank you. Good to be here.

Sacha: So Ari you've been at Red Hat for now three years?

Ari: Three-and-a-half years.

Sacha: And so tell us what you're doing at Red Hat.

Ari: So I'm focused on Atomic OpenShift – our products – but more in using those products to help deliver a faster moving host using OpenShift in Jenkins to actually have a real pipeline – eating our own dog food to deliver our own products. So that goes for Atomic Host, also will be for RHEL (Red Hat Enterprise Linux) and that's our plan is to use our products that we're telling customers to use – we will also be using the same products as well.

Sacha: That's very cool. So did you start from the get-go on this or is that something that happened late in the process and they realized they needed to improve delivery? How did it happen?

Ari: Yeah, when I came in I was in another group and we were focused on kind of spreading CI – just CI itself into the company and getting more cultural adoption. Over time we saw that all these homegrown tools that we have in-house – and they are open-source tools but they really – we should be using the same tools that we are telling customers to use to deliver their products – we should be using them and we actually should have it so that we have high quality and we're delivering our releases much faster than we had been. So that was kind of a cultural shift within the company.

Sacha: So did you get resistance from some of the teams to that change?

Ari: Yeah, there was definitely – you know, people resist change wherever it is. It kind of came down from higher up with Matt Hicks and other folks in the company of kind of doing these initiatives to get momentum and that has helped us a lot to kind of change the culture and get people to adopt it a lot more.

Sacha: And so for our listeners Matt Hicks is really the engineering mind behind OpenShift at Red Hat, right?

Ari: Yes. Yeah. He started on OpenShift originally when it was 2.0 and it had some other parts. Now it uses Kubernetes and Docker and what it is today.

Sacha: Yeah, yeah. All right. And so let's talk a bit about what you did before being part of that team because you're saying that you were part of a different team that was trying to spread CI across your organization. So in that role you probably had to deal with a lot of different teams. So, from a cultural standpoint what did you see?

Ari: Yeah, I saw a lot of people didn't understand what CI was: they kind of equated it with automation or automation frameworks and that was – I think it was evangelizing what CI is, and now eventually what CI/CD is together and that took a lot of time. And we had to kind of foster it with things like – use Jenkins Job Builder or other things like that to help foster the automation of using CI and CD in Jenkins. Now obviously we have the pipeline and we're using that and we've embraced that and it's been great for us.

Sacha: So you're using the Jenkins Pipeline.

Ari: Yes. Yeah, in fact a lot of the work when we started was these monolithic jenkinsfiles but now we've broken them down into shared libraries and we're using that and it's been very helpful. We're also spreading that to other teams in the organization so that they can leverage our own libraries or add to them themselves without us having to do all the work.

Sacha: I see. So what are some of the challenges you're facing that you'd love to see solved?

Ari: I mean there's definitely still those pockets of different groups – or like Fedora, for example, of different areas that still kind of have their own homegrown tools and it's kind of the thing where we have to kind of show them how it works, enable them and say, "Here's some use cases that we're using with our products. You can use these same use cases for yourselves." So it's definitely a culture shift still, especially the CD part – that's still kind of new for folks – so we're making progress and it's working out pretty well.

Sacha: Okay. And so OpenShift is heavily based on containers.

Ari: And Kubernetes, yes, orchestration container.

Sacha: And so do you face some specific problems related to containers as part of CI? I hear sometimes about container – in containers for build and testing – so any specific thing you want to talk about here?

Ari: I say the biggest thing is kind of getting out of the mindset of we don't have to run all these things on a VM or bare metal, we can streamline this and run this in a container. Obviously there's certain things in OpenShift that we're still actually figuring out ourselves of how things work. We're actually – the good part of this too is in using our own products we're finding issues before the customers are finding issues to kind of say, you know, we're using this how you want customers to do this and we're running into these things and they get fixed and rolled into the next version of the product which is pretty cool.

Sacha: So have you been able to influence the road map of OpenShift through some of the work you're doing and things you don't have and you wish you had?

Ari: Yeah. We've been pretty lucky so far actually that a lot of the features we have been running into of things that we want are actually already road-mapped into the next release. I would say some of the things that where we've had some impact on are the OpenShift plugins in Jenkins – the OpenShift Sync plugin and the Pipeline plugin – I’m kind of saying like we'd like to see this kind of feature in that plugin and that's worked out for us because people within Red Hat maintain those plugins.

Sacha: So I hope it's good on the audio but I can tell you right now, here, there is a concert next door and we can barely hear each other so it's pretty –

Ari: Yeah, trying to focus.

Sacha: Yeah. Yeah, it's hard to focus. So what can you share around some of the processes you're using? Do you have specific processes that applies to all of the teams or is that specific to each team? How do you go after that?

Ari: Yeah, I mean we're really taking kind of industry standard CI/CD model of those processes and we're just using the tools to carry them out. And Jenkins has been instrumental in that and coupled with OpenShift is a huge win for us.

Sacha: Okay. How big is the environment you're using for CI?

Ari: So, right now we actually have a cluster – a OpenShift cluster that's set up in our Centos environment – it’s a public-facing cluster – I don't know how many nodes are underneath it. I think it's basically like about a handful of nodes that maintain all of that for us. And then we have our own tenant inside there for our project for all the work that we're doing and then we have Jenkins with a persistent volume underneath that running, to do all the work. And we're maintaining that Jenkins master, not in a Docker file, but actually in OpenShift's S2I image model where everything's in our source code with all the other pipeline code, our development code – everything is in the same place. So that makes it very easy for us to change things on the fly and roll them out for ourselves.

Sacha: Yeah, and maybe some of our listeners don't know that but everything that Red Hat does is open source – 100%, right?

Ari: Yes.

Sacha: It's the biggest fully open source company and so I guess it simplifies a lot of processes around access rights and so on, right?

Ari: Yeah. I mean having everything up in GitHub, helps to be able to maintain it, have people contribute, whether they're within the company or outside makes a huge – you know, it's additional people to help – additional minds to help foster that technology.

Sacha: All right.

Ari: I can't hear anything.

Sacha: So have you – it's crazy. I don't know who organized this conference.

Ari: I don't know who did this.

Sacha: Yeah.

Ari: Who do we blame?

Sacha: So were you able to measure some improvements since you moved to CI to quantify those improvements?

Ari: Sorry, can you say that one more time?

Sacha: All right. Were you able to quantify some of those improvements?

Ari: Yeah, I would say we're able to now find issues in our Atomic host much faster and we're able to also, with a certainty, basically on a dime be able to release a version of Atomic host much faster which is our intent. There's definitely more improvements we want to make – there's always more to do or more to improve upon – but now we have kind of a template of how to do that and how to improve upon it. So we're adding more tests, we're shifting a lot of the testing that was done by QE into this pipeline to find an issue sooner and then make something release it that's more stable and at a much faster rate.

Sacha: Okay. Since introduction of CI did you change anything to the way you recruit engineers? Do you ask different questions – qualification question or you didn't really change the way you recruit engineers?

Ari: I would say what we've tried to do is adapt ourselves to what their model is of, I submit a PR, I want to see my PR get tested, I want to see that feedback. I think if you get that feedback to a developer they see the value in it much faster and they appreciate that and then over time they're like, "Okay, now I get how this works." So we kind of moved to them a little bit because I feel that – we felt that that would be a more adoptable solution, that here the developers see the work and they see the value in it.

Sacha: All right. So what's the next challenge you would like to resolve?

Ari: So this is kind of – what we're doing – this project – is kind of a pilot and we want to spread this to many other products within Red Hat. You know we have new releases coming up and we want this to be the model that we use for those products.

Sacha: All right, that makes sense. So we're at Jenkins World. You're going to have a talk I suspect?

Ari: I submitted a talk – I got waitlisted – but – there were just too many good talks.

Sacha: Oh yeah. So what's going to be interesting for you at Jenkins World? What kind of talks?

Ari: For me the shared library stuff – there was a talk today on it that was great – Declarative Pipelines. I'd like to see more on the visual editor as well. Some of Jesse's talks on the using Jenkins less and the development of plugins with plugins for pipeline – that was really interesting to me. So just really kind of absorbing all the new features and all the new enhancements that are in Jenkins or that are coming down the road – next gen.

Sacha: I see. All right. So you've been in DevOps for a long, long time. You love that space?

Ari: Yes.

Sacha: So maybe ask some closing thoughts. Where do you think DevOps is going and what do you think is going to be the next big thing in DevOps?

Ari: Yeah, I think it's – the model of like a lot of the testing and a lot of the automation is going to move more towards the development side – more developers. And I think QE is going to be more of kind of doing – I wouldn't say they're out of business but they're actually going to be doing the more complex customer use cases and focusing on that stuff. And I think by just having the developers more in tune with a CI/CD process that will enable operations to feel more at ease in what they're rolling out. I know we use our own products within Red Hat for our own engineers and that will put them at ease to know like they've been tested, they've been run through a CI/CD pipeline and there's confidence they're stable that they're going to be – perform well in our environment. So I think that just gets faster over time and more efficient.

Sacha: All right, that's great. Thanks Ari.

Ari: Thank you so much. Appreciate it.

Read More >>