Inside Some Key Jenkins Plugins for Google Cloud Platform with Craig Barber

Brian Dawson: Hello. Welcome to DevOps Radio. Thank you for joining us for this episode. I'm your host, Brian Dawson, and today we're going to be talking with Craig Barber who is a Google Cloud engineer, who specializes in OAuth integrations as well as development community partnerships and frankly, as well as wearing a number of other hats, but we'll let Craig tell us about that. So why don't we go ahead and introduce him? So Craig, are you there?

Craig Barber: I am.

Brian: So, you know, I probably didn't do a complete job at describing all of the things you do at Google. So to start, maybe you can explain what you work on, what your role is over there at Google.

Read More >>

Craig: Sure, happy to. So my official title is software engineer, but as you mentioned, that doesn't quite cover it. I do wear a multitude of hats in my role. It's a pretty unique role within the company. And my team's name is Cloud Graphite and if your listeners haven't heard of that, that's okay. We're not a first-party product, but essentially what the team does is we ensure that customers are able to use their favorite open-source tools and platforms that they're already using on GCP.

And so what that looks like from our end is we provide dedicated engineering resources to build out the bridges between GCP and those tools and those manifest as open-source contributions, usually upstream to those tools themselves. And we also do a bit of education and evangelism to make sure that the community around these tools and platforms are aware of the integrations that we're publishing and supporting. 

And it's also partnership building. We want to make sure that we have great relationships with a lot of the open-source companies that are stewards of these tools, as well as the communities around these tools as well. And so what that looks like us participating directly in some of the foundations that are housing these projects as well as us going out to conferences, giving tech talks and engaging the community and having very tight relationships with our collaborators and the open-source community as well.

Brian: Awesome. Okay, that sounds like an actually cool job.

Craig: I love it.

Brian: What kind of skill set would one need. If I want to, when I grow up, be like Craig, what skill set would I need to do your job?

Craig: Well, to land a gig as a software engineer at Google, you need to go through the gauntlet, which is what I like to call it. And that's essentially the day-long marathon of interviews and the best way to practice for that is to pick up a copy of Cracking the Coding Interview by Gayle Laakmann McDowell. This topic is going to be discussed ad nauseam, but as far as the other aspects of getting a gig at Google, you know, doing well in your university courses, having relevant work experience and all of that. But my particular role is kind of unique in that I'm not just your typical engineer cranking out code.

I do a lot of speaking engagements. I do a lot of education-related activities. I do authoring of material that's going to help customers onramp onto our supported integrations easier. So it's a mixture of good public speaking skills, ability to have good relationships and being social with community members, and also a good amount of writing skills as well.

Brian: Awesome. Thank you for sharing that. You know, technical evangelist at Google, and I know that's a component of your role, but technical evangelist working on open source at Google, I'm sure there are going to be a bunch of listeners that are like, "That's the job I want. That is a cool job." So thank you for sharing that.

So you guys develop a suite GCP or Jenkins Plugins for GCP. Correct?

Craig: Yeah, that's correct.

Brian: Can you tell us a bit about what plugins are in that suite and what problems they're solving or why somebody that is using Jenkins and interested in GCP should look into the plugin suite that you and your team have developed?

Craig: Happy to. So if at any point, people are curious and want to learn more about this, they can go to cloud.google.com/Jenkins and all the plugins that I'll be talking about will be available there as well as some solutions articles and YouTube videos and all that kind of good stuff.

But basically, we have four officially supported plugins that we publish that are supported and then there's one CloudBees plugin that is supported as well that we don't publish but is a crucial part of the story of Jenkins on GCP. So I'll go with the GCP officially published plugins first. The first one is our GCP plugin, and that enables customers to run their agent workloads on VMs and the main value add that it provides is it allows Jenkins customers to essentially configure a cloud.

And when I'm saying a cloud, I mean a Jenkins cloud in GCP allows them to specify what kind of VM configurations they want to use. If they have an OS template that they want to specify, they can specify that there. And the way it'll work is, when the Jenkins master goes to provision executors to run the jobs, it will actually dynamically provision VMs on GCP and install the Jenkins agent on there and monitor the jobs that are running on there and then when the master decides to de-provision those notes, it will go ahead and clean up those VMs as well.

So it takes a lot of the manual heavy lifting that would otherwise be required for customers to use GCP as the compute platform to run their Jenkins workloads on.

The analog to this plugin is the one that CloudBees supports and it's the Kubernetes plugin.

So they publish it, but we officially support it as well and that's essentially the same thing except it allows you to do your workloads running inside of a Kubernetes cluster. And specifically, we're interested in that because GCP, we believe is the best place in the world for you to run Kubernetes. So it basically runs your agent workloads as pods in Kubernetes.

Another officially published GCP published plugin is our GKE plugin, which enables customers to deploy applications that they've built inside of their Jenkins pipelines to Kubernetes clusters running inside of GKE. So for example, if you build a spring boot application and you want to first deploy that to a staging cluster, you can do that with a single build step in your Jenkins pipeline config file, deploy it to your staging cluster, do whatever SRE type things you need to approve that, and then as the final step of your build, want to actually promote it up to your prod cluster, you can use that same integration to do that too, basically enabling customers to deploy to multiple clusters in GKE from a single Jenkins pipeline using a single GCP service account credential and using cluster names rather than having to hard code in things like endpoint addresses and CA certs, which you would otherwise have to do if you were manually doing these things yourself.

Another plugin that we support is our GCS plugin, which is for Google Cloud Storage, and that enables customers to upload and download artifacts from GCS buckets during their Jenkins pipelines. It's really beneficial, especially if you're using a tool like Conoco to build your containers because Conoco natively supports taking a GCS URL as the build context parameter. So it's very easy for you in your build step to run your - let's say you're using Maven – run your Maven build and then package everything up into a TAR file and use our GCS plugin to upload that to a GCS bucket and then your next stage when you're constructing your container, you just specify to the Conoco executor the build context URL to whatever you uploaded to GCS, and then it'll go ahead and download all those artifacts, build them into a container and then if you want to publish that up onto GCR to publish your container image, you can do that.

So that's what GCS provides. It also provides a post-build step, so if you wanted as a post-build action to upload your build logs into GCS so you can have an ongoing record, a persistent record of your build logs, you can do that too. And if you format the blog URL in the right way, you actually get native integration with BigQuery because BigQuery knows how to query data that's in GCS. So if you want to be able to query at a large scale, all of your build logs going back indefinitely, it's a really nice integration.

And then the final plugin that we published is the OAuth Plugin and that kind of underpins the rest of these. It enables customers to take GCP service account credentials, store them in the credentials store and then it provides a library that all the other plugins that we publish and utilize to be able to recognize credentials of that type and then basically the value add is it does all the negotiations with Google's IM APIs for you so that you don’t have to worry about grabbing a credential, plugging it into your environment. All you need to do is when you use the build steps in your pipeline config for each of these different integrations, you just reference the ID of the credential as it's identified in the Jenkins credential store and all these plugins know how to natively work with the OAuth plugin as well.

Brian: Awesome. Wow. There is a lot there.

Craig: There was.

Brian: I hear sort of, by and large, you guys are taking tasks that people may be able to do otherwise within their Jenkins pipeline and you're making it much easier for them to do, much easier for them to leverage resources from Google Cloud as well as to deploy to Google Cloud. Is that a fair summary?

Craig: Absolutely. Our main goal is to provide a fantastic experience when customers decide that they want to use Jenkins on GCP.

So, as you mentioned, that's why taking into account workloads that they're already doing and supporting them, but also improving upon them and taking a lot of the manual labor and heavy lifting aspects out of their hands so that they don't have to worry about that, but also by providing integrations with first-party services that we think are key differentiators for the platform at large. So for example, GKE, we view as a key differentiator industry-wise because you're not going to get as good of an experience running a Kubernetes cluster as you will in GKE.

So if we can provide a great experience both running your Jenkins masters and agents in GKE if you want, but also deploying two applications running within GKE clusters, if we can make both of those experiences first-class in industry, then we think that'll be a great differentiator for us and provide a lot of value for our customers.

Brian: Awesome. So are any particular tenets that you guys provide or mantras sort of driving principles that you guys follow to achieve your objectives of a first-class customer experience?

Craig: Absolutely. So, first of all, we want to make sure that we are not in a position of prescribing to customers the way that we think they should be doing things. We want to be responsive to what customers are already doing.

And this is something that Google at large has gotten some flak for in the years past and it's something that my team, the Cloud Graphite team, is the remedy for. So the Cloud Graphite team's whole purpose is to meet customers where they already are. So if you're a customer that already has lots of people-hours invested into your Jenkins infrastructure, which translates into, you know, a lot of money for the company that has been investing into their infrastructure – we don't want to come to you if you're considering using GCP as your compute and storage platform. We don't want to come to you and say, "Oh no, all of that is bad. You just need to toss that all out the window. Here's the official Google way of doing things." That is the exact opposite of our approach. We want to come to you and say, "Hey, we noticed that you like using Jenkins. Well, we provide a fantastic experience for Jenkins and we're going to provide a very easy and smooth migration path for you so that you can get out of the business of running a data center, which not a lot of companies are very good at, and just focus on what your core business is, your core differentiators in your business. Let us handle running the data center and we will allow a smooth transition path for you to take the infrastructure that you've already invested thousands of people hours into and seamlessly move that over to GCP.

So the number one tenet is: meet customers where they are.

The other tenet is we want to be very responsive and engaging with our customers. So when you decide to use one of our integrations, it's not simply something that we throw up into GitHub, you know, wipe our hands and call it a day. We actually provide ongoing maintenance to these integrations. We have a Slack channel that we publish through the repos, so if you happen to go to the repos for any of our plugins, you can get on our Slack channel or GCP Jenkins Slack channel.

We are very responsive to questions and issues that customers raise and we also do the engagement piece as well. So, we make sure that we meet customers where they are by going out to conventions and engaging with them and talking with them and getting a better understanding of how they use these various tools so that we can better cater the engineering investments that are on our roadmap to provide value for them.

Brian: Awesome. That actually leads me to another curiosity that I had. So you said you're going out and you're meeting with customers. You're engaging with the community. You're figuring out how they work and what their needs are. In doing that, you must have learned a lot about continuous integration, continuous delivery and I would think in particular with Jenkins. Are there any discoveries you've made or any lessons you've learned that you can share with us?

Craig: Absolutely. One of the recurring themes that I see emerging in each of these conversations across the board is that the engineers that I talk to are very interested in the latest and greatest in CI/CD tech that is being introduced and talked about in these conferences, but they have a lot of legacy infrastructure that they need to support, oftentimes for many years. And that forces us as a team to have to provide two viable paths for our customers.

One viable path is we want to make sure that when customers are ready to adopt the latest and greatest CI/CD technologies, for example, Tekton is a fantastic solution, Kubernetes native pipeline solution, if customers at some point are willing to make the leap into Tekton or let's say make the leap into Jenkins X, which uses Tekton, we want to make sure that we're right there with them, providing fantastic integration to make that a first-class experience for them.

But at the same time, we recognize that because there's been so much investment into these legacy pipelines and toolsets, that we also need to make that path for them as equally good. So we want to make sure that we can receive them where they are and make sure that their existing infrastructure and their existing pipeline infrastructure and their existing pipelines run very well on GCP.

So it's an interesting position for us to be in, but we're happy to cater to both and we consider both first-class use cases.

Brian: Okay and so what I hear is something I come across a lot, sort of two takeaways from that for me. One, for our listeners, if you know, you are dealing with legacy platforms, you see this hot new thing that you want to try, this hot new technology, but you don't have the option there to move right away, I think as Craig is sharing with us, you're not alone. It's a reality. There are still some modern practices and technologies like GCP, thanks to Craig and his team's approach, that you'll still be able to use. So, one, don't feel like you're alone. Two, you're not at a loss. You can still modernize or begin to modernize with what you have. Would you agree with that, Craig?

Craig: Yeah, absolutely. The main value that we can provide for people who are in the position of they have a bunch of legacy infrastructure that they're going to need to maintain for the foreseeable future, the main value that we can provide to them is by alleviating the need for them to run their own data centers and their own on-prem hardware. Now, some customers, especially some very large enterprise customers, are going to have to be in that position no matter what for regulatory reasons, and we want to meet them and be able to provide value to them as well, and that's why Anthos is our Kubernetes anywhere management solution.

And a big piece of Anthos is GKE on-prem or Anthos on-prem, and that's what's going to enable these large enterprise customers that have to keep the bits running on their own hardware for various regulatory reasons. That's going to enable them to do that while still taking advantage of all the great modern conveniences that Kubernetes provides. So we cater to that use case as a first-class use case for our integrations.

But for customers that don't have those regulatory constraints, and really are eager to transition as much of their infrastructure into the cloud as possible, we provide that golden path for them because it's very expensive to run a data center, run your own hardware. It provides – it necessitates a lot of expertise which really aren't the core competencies of a lot of businesses out there. Quite frankly, you know, small to medium-sized companies are not going to be able to provide hardware at the price point that is anywhere close to the price point that companies like GCP, like Google via its GCP platform can provide just because of the economies of scale and the expertise that we bring to bear. So the main value that we can provide to people in that situation is alleviating all of the headache and the overhead costs of them having to maintain their own hardware while also providing very seamless experience for them to make that transition and take those CI/CD workloads and move them onto a cloud platform that's going to have the resilience, it's going to have the scalability and also provides a pathway to integrate in the future with some fantastic first-party services that are going to open up some use cases that they might not have had available to them otherwise.

Brian: All right, thank you. Now, earlier you shared a URL to access the GCP integrations with Jenkins, the suite of GCP plugins for Jenkins. Can you repeat that URL for our listeners?

Craig: Sure. It's cloud.google.com/Jenkins.

Brian: Okay and if someone is interested in Jenkins, but they're also using other open-source tools that they want to be able to use with GCP, is there a general location that they can go to to see the output and product from your team?

Craig: Yeah, so if you're interested in other CI/CD solutions besides Jenkins, because it is a very common thing that customers will use not just one, but they'll use a smattering of others for different use cases, so it's very common that we hear from customers that they use Spinnaker for some aspects of their CI/CD, some customers use Gitlab, some customers use Jenkins or a combination therein, so another place that you can go to is Cloud.google.com/docs/CI-CD.

Brian: Oh, okay. Okay. Awesome. So, Craig, I've been asking our guests what they've been reading and asking them to share that with our listeners. Is there a particular book or text or reference that you have that you recommend our users check out?

Craig: Absolutely. There's two books that I would recommend and there's going to be a bit of a shameless company self-plug, which is probably unsurprising. But, one of my absolute favorite books, especially for customers that are wanting to dip their toes into Kubernetes. They've heard about it and they're really not quite sure where to start, my go-to book recommendation for them is Kubernetes Up and Running by Kelsey Hightower and Brendan Burns and Joe Beta. That's a fantastic getting-your-feet-wet book. It'll provide a very high-level overview of the architecture of Kubernetes and then goes into detail to each of the parameters that I've provided. If you've gone to conferences at all, I'm sure it's not a surprise to hear me say that Kubernetes is the de facto kernel of the cloud is what I like to call it. It is the extraction layer that the entire industry is coalescing around. All the major cloud platforms are supporting it.

So, if you want to help bring your organization into the future of cloud computing, that would definitely be my go-to book. And then for CI/CD, I'm going to plug another friend of mine's book, and this is Jenkins X, CI/CD for all Workloads Running Natively on Kubernetes. And this is authored by James Rawlings and James Strachan. Sorry, James Rawlings and James Strachan, two of the lead developers for Jenkins X. Definitely give that a look. Jenkins X is fantastic, opinionated Kubernetes native solution and we are excited on the Cloud Graphite team to provide integrations that enable customers to run Jenkins X and have a great experience running it on GCP.

Brian: Awesome. Thank you. So we're at about time, Craig. A lot of great information on accessing Google Cloud Platform, the way you guys service your customers and then importantly for a lot of our listeners is, you know, what plugins are available to help them work with Google Cloud Platform or GCP better within Jenkins. So thank you for that information. Before we wrap up, I'd just like to know, are there any final thoughts, tidbits of information that you'd like to share with our listeners?

Craig: Yeah. I'll just repeat the URL one more time. It's cloud.google.com/Jenkins and just to reiterate what the mandate and mission of our team is is we are first and foremost interested in making sure that customers can use the tools and the platforms that they are already invested in very well on GCP. So, Jenkins is the largest CI/CD by market share tool out there in the wild and so obviously we want to make sure that this is a great experience for our customers and we love engaging with customers, so please do come and join our Slack channel. If you have any questions or run into any issues when using any of our integrations or just running Jenkins in any way at all on GCP, we're happy to answer questions there. You can get to our Slack channel on any of our repositories for our plugins. All of them you can get to through that URL, cloud.google.com/Jenkins. And we are also very active in the conference space, so I'll be at DevOps World this year. I'll be giving one of the talks about Jenkins X on GCP. Happy to interface with customers there and chat and answer any questions.

Brian: Awesome, Craig. Thank you much for your time. Everybody, that was Craig Barber providing us valuable information on leveraging Google Cloud Platform and in particular Google Cloud Platform with Jenkins and the suite of plugins that they have created. Again, thank you Craig and thanks everybody for listening.

Craig: Thanks very much.

Brian: Bye-bye.

Like what you’ve heard today? Don’t miss out on our next episode. Subscribe to DevOps Radio on iTunes or visit our website at CloudBees.com. For more updates on DevOps Radio and industry buzz follow CloudBees on Twitter, Facebook, and LinkedIn.

Read More >>