Brock Beatty - What's In Your DevOps Wallet?

Sacha Labourey: Welcome to DevOps Radio with Sacha Labourey from CloudBees. I'm here with Brock Beatty from Capital One.

Brock Beatty: Great to be here.

Sacha: All right. It's very good to have you here. You come from Richmond, Virginia. We have an office there and I've been there many times. So it's really great. There's a little story that's going to be absolutely not interesting to anybody. There is actually Philip Morris that came from Richmond.

Brock: That's correct.

Sacha: Where I live, in an unknown city in Switzerland, actually, we have a huge Philip Morris factory. So I don't know what I'm saying exactly –

Brock: We have that in common, right?

Sacha: Yeah, exactly.

Brock: That's great.

Sacha: Anyway, Brock, you are responsible essentially for the centralized DevOps teams that's providing the complete toolset and offering to the teams at Capital One. Is that correct?

Brock: That is correct. We're focused on developer enablement and tools engineering, so operational excellence as we provide a bedrock of tooling and capabilities for our developers to ultimately leverage as they are being successful and to have more customers.

Sacha: So when did this initiative to provide this centralized offering to all teams started?

Brock: It started a couple of years ago; believe it not. Capital One is typically at the forefront of some of these more technical revolutions in the industry and that of the fintech vertical, but it's been evolved more rapidly in more recent times, and that's simply because of the emphasis on agility and time to market, and the demand of our customers to get new features in a very quick timeframe.

Sacha: What do you think made Capital One be leading the pace here, really? You've been initiating a lot of things that some other banks are not doing yet. Right?

Brock: Yeah. I tell people it's a very good story, and it's a good story about our leadership at Capital One. This is something that a lot of organizations struggle with. I speak with them at conferences all year long. It's different because Capital One's leadership pushes this down. They enable us. They challenge us. They want us to do this more frequently, more rapidly. They ask us where are impediments are, where are barriers are, how to be better. Other organizations, I hear, actually are trying to drive this revolution from middle management up, and they're having challenges to do that. So that's one thing that's great about working for Capital One. Our leaders get it.

Sacha: And when you talk about leadership, you're talking very high up the C chain.

Brock: Yeah, up to our CEO, Rich Fairbank, Rob Alexander. They highlight the importance that technology plays in our customers' lives and that of our strategic vision for the company frequently. So we take that vision and run with it, and they give us the autonomy to be successful, and that's essentially what we're doing at Capital One.

Sacha: That's amazing. Can you describe to us a bit about your infrastructure, how it looks like, where it's operating and so on?

Brock: We have a cloud first strategy, in terms of we want to operate 100 percent in the cloud. I think it affords us a certain level of flexibility and autonomy that you wouldn't see in a datacenter. But that story has been told. Effectively, we are building cloud tools on the cloud, for cloud developers to build cloud-enabled applications. That is not just a single provider, right. That's multi-cloud environments, obviously most big names you would recognize and expect us to be interacting with. But essentially we're looking for cloud developers, cloud engineers, full-stack folk that are passionate about automation, and are really trying to continue to help up us push the frontier on the cloud space.

Sacha: All right, that's great. Can you tell us about the breadth of this infrastructure, how many servers or how many teams maybe you have on this platform?

Brock: Yeah. On CloudBees Jenkins itself, we're approaching roughly 225 actual instances. That's for an infrastructure supporting a variety of different build structures, whether it's masters, JOC or your build agents. So it's fairly sizeable. I like to tell the story that we build Jenkins with Jenkins. We build our configuration with Jenkins. So it's a little bit of an interesting relationship. We're building the next version of Jenkins using the previous version of Jenkins. That's a great thing that we've adopted at the team level, because there is no excuse for not being automated. We're a platform team. You can still deploy CI/CD techniques and tools to build platforms right there in a rigid infrastructure, even if it's _____ for example. So we have a large installation and it's highly flexible, multi- regions, multi-AZs, supporting approaching 5,000 or 6,000 registered developers on our platform alone, and that's increasing every day.

Sacha: Wow. So you're using cloud. You're doing continuous delivery. We’re missing the third axis, containers. You are using containers.

Brock: We are using containers at Capital One, and that's increasing the popularity day-over-day. A lot of the use cases are coming from the developers who want a higher level autonomy, so that they can define their own infrastructure, define their own runtimes, and have a little flexibility in how they create packages in their applications. At the platform level, we're looking to adopt CloudBees Jenkins Enterprise. They give us a much higher density of compute in terms of how we leverage that infrastructure. I mentioned we have a large fleet of infrastructure in my previous comment. That's not as heavily utilized as we would like. So going to containers from a platform perspective gives us a denser utilization of that compute. From an application perspective, it gives them higher control and autonomy of their runtime in real-time environments, and, ultimately, that level of abstraction gives us the ability to more rapidly scale in response to demand for our resources because our build agents are thinner.

Sacha: In terms of control, central control, where did you put the balance between more central control versus giving some flexibility to the teams? How do you handle that?

Brock: We like to come at it from the perspective of the developer. We always try and defend, speak on behalf of the developer. The developer experience is paramount. We're never going to attract and retain the best talent if you give them a poor experience or a poor work environment. So, on the other side of that coin – is– I'm liking the music. But on the other side of that coin is the need to be a well-controlled and managed financial institution in a highly regulated environment. So what we look to do there is always understand the developer's perspective, so we can empathize with what they're looking for, what their challenges are, where their points of friction are, and then work backwards into the regulatory environment with our risk and client stakeholders, to inform them about points of friction and how we can work together in a creative way to solve this problem on behalf of the developer, and ultimately that of our customer, because the developers are delivering on behalf of our customer. I always like to re-emphasize that. But it's a challenge sometimes because we're caught between two tectonic plates. This demand from the developers to just be better and more flexible in their job, in their day-to-day, but also, we have to be airtight from a security perspective. Absolutely. There is no negotiating that. I'm right at the point, the line of scrimmage, between those two competing forces. So I have the luxury of working with a great team of developers as well as auditors and stakeholders to come up with creative solutions that ultimately service both of those needs.

Sacha: Yeah, because especially a lot of that is based on trust. When you negotiate, quote/unquote, with a regulatory team to get more flexibility on some processes, it's probably because you assure them that you're going to do a number of things for them that would prevent any issues. So as you add more tools that you need to validate and so on, you can't fail. Otherwise, you lose your credibility with the regulatory teams.

Brock: Yeah. I like to say perception is an average of experiences, and averages, in my opinion, are easier to be lowered than raised. So we work hard to keep our developers happy. Because we're able to succeed at that, we're able to more confidently deliver services that are impactful to them. Then, in return, they have more confidence in our ability to do that. Therefore, we have a feedback loop that works to benefit developers and the organization.

Sacha: Let's switch gears a bit and talk about culture. All of what you were describing happens in what's essentially an extremely short timeframe. It looks like a few years, but in banking terms especially it's like a snap. So it must have been quite a shock to a number of individuals. How did that happen? How much conservatism did you have to fight? Did some people have to go because they couldn't cut it? Can you tell me more about that?

Brock: There's nothing easy about moving a large organization from one infrastructure to another. You multiply that difficulty when you talk about a financial institution and you talk about the cloud. So the culture, again, started at the top. We were 100 percent bought in at the executive level, senior levels. That made it a lot easier to rapidly traverse that change curve. So I like to lead with that. But beyond that, I think the easiest way we've really been able to drive change is transforming our staff, so looking for people that are true technologists, that are really evangelizing different ways of leveraging technology to achieve outcomes. We find those folks in all corners of the enterprise, and when we work together as a cross-functional team, whether it's a card team or a bank team, or an auto finance team or a commercial team or a share tech team, we come together and we bring the best of all these different worlds and start to combine them into workable solutions, and we come together on it very quickly. Once we're able to align on these solutions, these patterns, these tools, then those change agents take that information back into their respective areas of the enterprise and it just catches fire from there. That's how we've been operating at Capital One over the last few years, and it's just really amazing to watch and be part of.

Sacha: That's really great to hear. I was interviewing before somebody from ABN AMRO, a European bank. He was telling me how his objective is to get to continuous not just delivery, but deployment. They have this as an objective. You know, you have to set objectives and that's one of their objectives. Do you think that's something that's feasible? Maybe there are some differences between European and American banks or regulatory constraints. What's your perception on that?

Brock: Nailing the deploy piece is much more difficult than nailing the integration piece, the continuous integration piece. I think that it's absolutely achievable, and I think we're achieving that at Capital One through a variety of different things. We're an API first company, cloud first, but mostly API first. By forcing everybody – not forcing, but really impressing upon everybody the value that this provides the enterprise, we've been able to much more rapidly integrate complementary services, to achieve higher levels of control and be more well managed, as we move toward and accomplish difficult things like continuous delivery. So we don't necessarily need to have all the rigidity of yesterday's enterprise when it comes to change management, for example. We can break down some of those barriers and still be automated and well-controlled by leveraging APIs, by coming together with these cross-functional teams, by working with our regulators, by working with our risk and cyber security stakeholders, to ultimately come up with a rock solid solution that our developers can leverage and be successful with. So continuous delivery absolutely is something that is achievable in the banking industry and something we're doing here at Capital One.

Sacha: I'm amazed because I feel like in just a few years we've moved from a place where banks were saying no to cloud, were saying no to agile, and now we're talking about cloud first. We're talking about continuous deployment and empowering developers. It's pretty amazing. In just a few years, that's a real drastic change in that market.

Brock: It absolutely is. I've been in the tech industry long enough to have seen a few different changes, and this one by far is the most rapid and amazing in terms of all the different dimensions of challenges that can be presented to any organization that embarked on this path. Yeah, it has been fun. It's been high-paced, but there's no other organization that I've been a part of like Capital One, where something of this magnitude would have been possible. So yeah, it's something that – you know, give us the next challenge. What's up next? Something else will be just right around the corner and Capital One will tackle it just like everything else.

Sacha: Yeah. If we have European listeners or Asian listeners, it might be good. Do you know a few metrics around Capital One? I know it's a huge bank, but I know also that maybe it's less known than some other banks for people in Europe. Do you know, for example, how many employees you have at Capital One?

Brock: We have approaching 10,000 technology associates across a variety of different locations in North America and outside of North America. We'll have upwards of 50,000-some-odd Jenkins jobs. So that gives you an idea of the scale that we are actually delivering against for our customers on any given day. To give you an idea of some of our other metrics we can round out for you, we have hundreds of thousands of GitHub repos.We have dozens of thousands of _____ organizations. So when you talk about all that source code, that is just nothing but business opportunity. So the faster we can ship that, the faster we can turn that into a work load product, the faster we can test it, roll it out, combine it, rework it, get it out the door, that just gives you an idea of how that can actually come to life and work for our customers. So the next KPIs we'll be looking at very closely. What is the throughput? What are the number of commits? What are the number of successful builds? What are the number of deploys? I think that will be really telling to our ability to achieve this technology we have set out for ourselves.

Sacha: What's your preferred metric? What's the thing you look at? What's your magic number, essentially?

Brock: For me, because I'm engineering, I like to look at number of incidents per month. I'm a big quality guy. I like doing things right. I like doing things that are sustainable. I love the fact that we've seen a reduction in overall incidents, in not only the DevOps model, which brings smaller teams to bear for the development and support of an actual piece of software or application, but I think that it provides a much more intimate relationship between the application and the people that actually support and care for it. Also, there's that feedback loop that everybody talks about, which is if you build something that's not going to work, you're going to be getting the call in the middle of the night. So all those things come together to help us build a more quality product for our customers and ultimately reduce the amount of downtime. So that's one metric that's near and dear to my heart, and I think a lot of folks in engineering would probably identify with that.

Sacha: All right. That's pretty cool. Look, it feels like you live in a perfect world. So you don't have any more challenges to solve, right? It's all done.

Brock: There will be challenges to solve. There are always challenges to solve, some of which are identified, some of which are yet to reveal themselves. But I think the number one challenge I see coming up in our near future is the ability to provide a cohesive set of services or an ecosystem for developers that transcend any specific region or any specific provider in a very predictable and scalable way. You talk about doing it within a single provider, within a single region. It's difficult. Once you conquer that goal, you look up and there's a whole 'nother mountain in front of you. That is my mountain right now, is to figure out how to do that, all the while making sure that we're providing continuous as a predictable service to our developers.

Sacha: All right. Maybe as some closing thoughts, where do you think DevOps is going? What do you think is going to be the next big thing?

Brock: I think there's a lot of opportunity in the continuous compliance space. I think that shifting left compliance controls, as opposed to a detective capability, is something that's going to be a high area of focus for us. It just starts to fill up some of those little air holes that exist in your control network. So I'm looking forward to that. I think that there's a lot of opportunity for additional automated testing for the tools that we roll out. Quality engineering I think has been, not de-emphasized, but something that is maybe deprioritized in terms of teams as they're going forward and making these investments into automation. So I think that will continue to see growth and maturation in that area of the pipeline. And then, as I mentioned, moving to more agnostic tooling, so that we can really be seamless as we interchange and move across the different providers that we do business with.

Sacha: All right. Thanks a lot, Brock.

Brock: Anytime, my pleasure.

Sacha: This is DevOps Radio. Thanks a lot, everybody. Bye-bye.