Episode 73: Live from CloudBees Connect - The Reality of Delivering Modern Software Part 1

Brian Dawson: So today, I as the host of the DevOps Radio podcast, which if you haven't heard it before, self-promoting plug, please go download it. If you have, tell a friend. We've decided to conduct DevOps Radio live here at CloudBees Connect and we have a panel of real-world practitioners and leaders that are going to participate with us. 

First off we have Daniel Ritchie, Distinguished Engineer at Broadridge. Daniel, nice to talk to you again. I'm excited to talk to you again. How are you doing?

Daniel Ritchie: Fantastic.

Brian Dawson: Daniel and I always have really interesting conversations on DevOps Radio, really insightful, so thank you for joining us. Next we have Jennifer Hansen, Director of Product Management Delivery Experience at Capital One. Jennifer?

Jennifer Hansen: Hi, Brian. Hello, everyone. I am excited to be over here today talking about this panel topic. I'm the product lead for creating a unified, cohesive delivery experience at Capital One, so this is very near and dear to my heart.

Brian Dawson: Awesome. Thank you for joining us and thank you for that intro. And then we now have somebody from my local hometown area, Redwood City. We have Adam Robertson, Head of DevOps at Pinger. Hello, Adam.

Adam Robertson: Hey, good morning. Hi, everybody. I'm Adam. I'm the Head of DevOps at Pinger here in Silicon Valley and excited to see you guys again.

Brian Dawson: Thank you for joining. Love the surfboards in the back. And you guys are already familiar with Bob Kelley who just delivered our customer keynote. He again is the director of delivery engineering at IHG. So we're going to go ahead and jump in and discuss some key topics in relation to speed versus quality and compliance and DevOps in the enterprise.

So first, we want to cover the topic of defining DevOps in the enterprise, right? What does DevOps mean in the enterprise versus small to medium business or versus that happy path that we always hear about at conferences, right? There goes your mainline trunk development, get everybody on tool, what have you. What are the unique challenges and benefits? So Jennifer, I'd like to start with you.

You know, I realize that Capital One operates at a massive scale, and in fact, Capital One is outpacing some competitors that are two to three times as large, so it seems like you've, you know, got a hold on how to modernize your software to development and delivery. Can you share with me or share with us how you define enterprise DevOps?

Jennifer Hansen: Sure. So you called out pretty accurately, we are a digital leader in the financial services industry today and on a mission to change banking for good, but at the heart of it we are aiming to empower our engineers to build great products for our customers. So for me and for all of us at Capital One, DevOps at its heart is about people and culture. So we've been building a culture of continued learning and development. We've been reimagining our operating model.

We want to have our designers, engineers, product managers, business partners – they all work together. We're trying to continue to create the shared experience across the board, shared responsibility, shared accountability, because ultimately we are all in this together trying to create a better customer experience. Now our Agile and DevOps transformation has been a significant multiyear journey. We have been focused on delivering high-quality working software in a consistent, safe, reliable and efficient manner, so that's a lot. [Laughs]

Brian Dawson: Right. Easy.

Jennifer Hansen: Yeah. So to really get to this we've transformed how we work across the software delivery lifecycle. I was listening to Bob and all the challenges and the great accomplishments that he has been driving internally with IHG, and in a very similar manner we shifted. We shifted from Waterfall to Agile and DevOps, we started to align on RESTful APIs, we moved to a microservices architecture, containerization in the cloud. We've been actively contributing to and using open source software and then we have a cloud-first mindset.

So yes, ultimately our goal is about speed and automation while remaining secured and applying all those good best practices of what it means to be well managed on the cloud. I think like a few key takeaways as I reflect on the journey, it starts with having the right cultural and mindset shifts, right? Agile practices at scale, you build your own shared accountability models with our engineers and across the company. It's about maturing our engineering practices and understanding that we're on this journey.

It's not going to happen overnight but we need to continue to mature and elevate. We need to promote reuse, and we've been encouraging innovation across the board. Open source first inner sourcing. Bob called out his contribution model. We're big on that. We have a lot of hackathons. Simplification, automation, and yes, we do have some standardization across our delivery platform. We need to get to common patterns, core toolsets, and think about automation of our control gates. So these have been some of the things that we have been working on actively for the last seven-plus years.

Brian Dawson: Wow.

Jennifer Hansen: It is definitely a journey. We're not done, but I'm excited to be part of this journey at Capital One.

Brian Dawson: Awesome. Thank you, Jennifer, for the detailed response and those insights. So what I heard is it's super easy and you guys will be done tomorrow, right? No, okay. I appreciate that. One of the things you brought up there is shared ownership and empowerment, and I look forward to having you chime in when we dig in on that topic a bit later. Now of the organizations on the call, Adam has a different perspective, right?

He's dealing with a slightly younger, small, more digital-native organization. Adam, at Pinger how is your experience different from what Jennifer shared and what some of the other panelists may experience?

Adam Robertson: Yeah. As you would expect it has a lot to do with scale. You know, we have less than 100 committers and six to eight engineering teams all local and we have, you know, Romanian counterparts, but for the most part it's a small team. You can establish a little closer relationships with the engineers. Most of what Jennifer had mentioned is actually – it holds true just at a different scale, right? Some of these things they transcend from enterprise to smaller companies and you want to keep that in mind, because when you're building some of the foundational elements you want to have in mind the greater picture, which is you want to get the enterprise to scale and you want to be able to build the systems to scale up in that way.

A lot of the times you're not building for, you know, 10,000 engineers, but at the same token you want to build like you're supporting 10,000 engineers so you can support that scale as you grow as a company.

Brian Dawson: Thank you, thank you. So what we're going to go ahead and do is we're going to shift from enterprise DevOps to something that's, you know, not necessarily unique to enterprises but critical, and that is getting the balance between security, compliance and quality with speed. So we're going to move to Daniel to start out on this one. Daniel, Jennifer and Adam had different perspectives and points of view on enterprise DevOps. Now I think you'd agree that this balance we're talking about is a key part of it. Can you share with me how you guys have approached balancing security, compliance and quality with speed at Broadridge, especially at the size of your company?

Daniel Ritchie: Absolutely. You know, when you're a financial services industry and you're dealing with other people's money you have to make sure that you're not making mistakes by leaving doors open or opportunities for bad guys to come in and cause trouble. So we've definitely have to err on the side of caution, but there are many opportunities to build in things which will help speed up the pipeline, and so the idea is security can be a built in aspect of what you're doing. You can have that be automated or at least have those checkpoints be automated so that we can have a high level of confidence about what we're doing.

Brian Dawson: Awesome, awesome. I always like to say that in a number of cases speed can increase security, right? At minimum, having the ability to respond and react fact versus having a slow, high mean lead time product cycle. So thank you for that.

Daniel Ritchie: Absolutely.

Brian Dawson: Adam, we're going to circle back to you real quick. How do you balance these elements? You called out that you do have some of these same problems, not at the same scale. How do you balance them and do you err towards one versus the other? Speed versus security or – I guess you won't say you do. [Laughs] But how do you balance it?

Adam Robertson: Yeah. It's interesting because, you know, as a smaller company, and we're not in the finance or health industry so we're not subject to, like, HIPAA or any other kind of compliance, we still have to honor compliance for things like GDPR and CCPA because we do have customer data, but it's less important when you're not a financial company or a health company, but they're still important. You still want to do right by your customer and honor their privacy, and security is important no matter what company you're in and what scale. So these are important things. I think one way you can – one way that I found of balancing it is making sure that, you know, they all have a seat at the table and that you build a system that can be agile enough to where you can add these elements as you need them if you need to emphasize them.

So as we grow, for instance, if you didn't have to deal with GDPR and all the sudden you do, you go international with a product, you need a system that is able to – like it's modular. You can plug in these individual security components or extra steps along the way, along your pipeline, in a very seamless and painless – as painless as possible manner.

Brian Dawson: Right.

Adam Robertson: If you can kind of see these things coming and you build your system to expand – you have plugins, you've got modules you can plug into these systems – you can scale a whole lot easier and then it's just another day, another thing. It's not a large, scary thing that you're not used to doing, just the team has practice with it and you know how to scale.

Brian Dawson: And what I love and what I take from that was kind of, one, is a progressive rollout, two, is future-proofing, right? Build out an architecture where, you know, you can add them in as you need. Another thing that you brought up circled back to something Bob said with his – how they implemented security and governance standards with a committee using a voice of the customer approach. So maybe, Bob, we'll bring you back in for a moment. Do you have any comment to add about the need to involve stakeholders when defining security gates and compliances gates?

Robert Kelley: Oh, absolutely. We're lucky enough to have a very capable security team that provides us with certain guidelines. What we're trying to do to help teams out is to look for ways to provide a shift left model to the security testing piece, moving some of that as we can into the CI flow so that teams or developers get feedback on certain aspects of their code as they're actually developing it and as they attempt to merge it into the branch they're working in. So we work with them there and in the delivery flow doing similar things to help identify for them, you know, what they can be working on.

Brian Dawson: Okay, great, great. Thank you, thank you. So give us some time for questions and answers, Jennifer I'm going to help you segue us into the next topic, which is something that you touched upon earlier, and that is this concept of shared ownership and empowering the practitioner, right? There's a lot of talk. I'm hearing it more and more from this panel here, from your peers, something I really believe in and that's having happy and empowered practitioners and developers.

It means you have productive and innovative practitioners and developers. And, you know, what I'd like to learn from the panel, right, is do you believe in that, how does that tie into culture and how is that necessary to enable scale? So Jennifer, to direct a question specifically to you, how have you balanced standardization with autonomy and empowerment of your developments, and I like to say, your practitioners across the spectrum?

Jennifer Hansen: As I think ab out that question, Brian, we are in the financial services industry. We want it all in terms of we want to empower our customers and our engineers but we need to delight our auditors, right?

Brian Dawson: Right. [Laughs]

Jennifer Hansen: That is a critical component of this. We've done a lot of empathy interviews, design thinking sessions, usability studies, focus groups to really get to the heart of what our engineers want, and we've got this tagline. It's #makeshiphappen.

Brian Dawson: [Laughs]

Jennifer Hansen: That's what our engineers want. They want their code to get out of the door really quickly, and we want it to be well managed. So as we thought about this and we said, "Well how can we empower our engineers and improve the experience?" So plenty of journey maps later we realized process simplification and automation, reuse of highlighted. The whole you build, you own is that accountability model. But what we came up with is we set up a software engineering clean room, so we borrowed from manufacturing, if you like.

And within this clean room or product teams are now able to continuously deploy high-quality software, right? What we figured out is what are those core principles that Bob was mentioning and what do we want to hold our teams true to? How do we certify the delivery pipelines? Now I have to give a shout out to Dr. Topo Pal.

Brian Dawson: Oh, yeah.

Jennifer Hansen: He's a distinguished fellow and a peer of mine at Capital One, and he's been instrumental in helping us set up the clean room. And it's not a once and done thing; it is about maturity, it's about ensuring that we can get to those high-quality compliance secure releases. I'm happy to say that we are here in this journey and we have hundreds of teams that are releasing today via the clean room. So it's insightful because it's not just about we track incidents, we've seen reduction in our incident trends, we've been tracking the number of releases, but I think the value it brings is you get real-time delivery, insights and metrics, right?

Brian Dawson: Right.

Jennifer Hansen: And that's what you need. You need to understand where the gaps are, where the problems are and how to mature the practices. So sometimes you think you want something and you have to tradeoff, but what we've tried to do is approach it with we want to promote open source, we want to be that kind of a culture, and at the same time we need to react quickly to any open source vulnerabilities. So there isn't a tradeoff. It's this and that. We want that faster commit to product deploy but we don't want to have – we want to ensure every commit is reviewed.

I heard some great conversations about the types of development, the PR checks; we want our commits tested and verified and we want that lower incident rate. So the clean room has really become that capability and that notion of helping us get there.

Brian Dawson: Well, I love the idea so much to dig in. First, delight auditors. That's the first time I've heard that. We want to satisfy and empower developers and delight auditors, and you're bringing them along on the road – well, on the journey. Thank you for sharing that with us. Oh, and the other thing I'd like to call out that I love from other conversations I've had with you that kind of showed up here, and it's a product management approach to figuring out how to build out this service, right? You spoke about journey maps and design thinking.

For those here that aren't familiar with the design thinking approach to doing this that lives in modern product management, give it a look. Now Daniel, I happen to know and I'm gonna tell on us again because I always do, I happened to learn over a bottle of maybe too much red wine that you are passionate about this topic. You've shared the analogy of letting developers run with scissors if they want to, so how does that relate to this concept of shared ownership at Broadridge?

Daniel Ritchie: You know, I might say running with blunted scissors.

Brian Dawson: [Laughs] Okay.

Daniel Ritchie: There's something, I don't know, really important about this topic because it touches on some of the joys of human existence really, right? I mean the people that are involved in all of these different processes have chose their career, right? We are autonomous beings and we got excited about learning about these things and practicing these things because we got something out of it. And as an enterprise you have to have standards. You have to have frameworks that give people an understanding of what you're trying to accomplish as an organization.

You have to have security guardrails and guidelines in place. You have to have all of these different aspects come together, but if you take the experience away from the individual that is driving them to be creative and productive you're going to, you know, really soften the ability of that individual to be productive in a meaningful way. So the key is to find that sweet spot between maintaining that structure that a large organization requires but also giving as much freedom and autonomy as is possible to the individual so that they can exercise their creative license to do something that's really going to benefit the organization.

Robert Kelley: Very –

Brian Dawson: Awesome. Did you have a comment, Bob?

Robert Kelley: I was just agreeing. [Laughs]

Brian Dawson: Emphatically.

Robert Kelley: Right.

Brian Dawson: Well, so let's go ahead and actually shift to you for a moment, Bob, as we unfortunately start to move towards the end of our panel here. You know, you're engaging disparate teams and you're bringing them together, and you did talk a bit about how you encouraged collaboration and adoption by at the same time letting them choose tools. Can I ask – if you were to share, you know, one key principle in balancing sort of empowerment, adoption and collaboration, what would that be?

Robert Kelley: One key principle.

Brian Dawson: Yep. I'm putting you on the spot. We didn't plan for this one.

Robert Kelley: In the collaboration piece I'm probably going to focus more on the DevOps's teams collaboration with a development team and kind of take the rest of the organization out for a moment, and that's where we've adjusted our thinking similar to the way Jennifer described the way they work at Capital One. And in terms of bringing to the table a high level abstraction of the goal that we're trying to hit and then partnering with the teams and the developers to achieve that goal in a way that satisfies the corporate need and ability to contribute, it's driven in part by our opening up our code base to their contribution and it's opened up in terms of the way that we really want them to be involved in the pipeline construction and to tell us what they need rather than for us to describe or define what they should be doing.

And so we do have a series of quality gates that are not exactly the same but semi standardized across the board, but we don't walk in with a lockdown set of rules and the way that you're going to achieve that goal or when. We work that out with each team and we let the team be very largely responsible for their success because they have been responsible for that success in the past and our company has grown based on that effort and so we need to recognize it.

Brian Dawson: So you allow them to run with blunted scissors as well.

Robert Kelley: As much as –

Brian Dawson: But then you –

[Laughter]

– offer to help patch them up, bigger, better, stronger. Well, thanks for that answer. Shifting to Adam, I'm really curious, right? We've gotten answers from actually organizations that are orders of magnitude larger, right? So I have to assume that because you've been able to take more of a greenfield approach in building these practices and teams that it's gotta be really easy, right? All of your practitioners are empowered and happy, no problem, right?

Adam Robertson: That is not the case.

Brian Dawson: [Laughter]

Adam Robertson: It's got its own set of challenges as well because a lot of – you know, when you have smaller teams you have a lot of people that have been a part of building the existing systems that are already there and they have a pride of ownership and a lot of comfort level – like a high level of comfort with these systems as well. So when you're walking in as, you know, someone who's an agent of change making suggestions it's a very careful approach. You can't, you know, go in just saying you're going to change everything that they've built because there's some pride with what they've done and there's some, like I said, level of comfort. So, you know, you have to spend a lot of time listening to why things are built the way they are and why they're operating and why they're comfortable there, because there's reasons for it.

Like I think someone mentioned earlier, there's a lot of smart people that design systems that have worked with them for a long time and so your job is kind of to understand what the ecosystem is that you're getting into and start to make a change and actually get everyone on board with that. I think an analogy that I've heard that I really like is, you know, the team that wins the Iditarod is the one with the Huskies that take you across the finish line. You're not dragging your dogs, right?

Brian Dawson: Yeah.

Adam Robertson: So you want your team to be excited about what you're doing and they can see the benefits, and the approach of doing that can be harder in smaller companies because a lot of times people that you're trying to convince to change are the ones that built those systems in the first place. And generally what I found is that it takes time. It takes longer than you think and you have to build some trust. Daniel pointed out that, you know, these are people who take a lot of – they did this for a reason. They didn't just do it because they were told to, right?

They have a vested interested in the systems that they built, and for you to go in and ask them to make sometimes really hard changes – you know, like I think Bob's slide of saying who wants change? Yes. Like, you? Okay, let's do it. No. Wait. [Laughs] This is hard. And that's very much the case no matter what I think of any scale.

I think that's just the way it is. So you show the benefits, you show how this will work, you show how empowering them and building these self-service tools it can really benefit them and they start to see this and you see the benefits all around. It's a win for management, it's a win for the engineering team, it's a win for you to basically let them run with blunted and eventually extraordinarily sharp scissors that they're running, right?

Brian Dawson: [Laughs]

Adam Robertson: But it's a slow process and you just have to be really mindful of the personalities and just the fact that you're asking people to make a large change to their day-to-day and that day-to-day is something that they've been used to and that they had a vested part in building and it's a careful thing no matter what scale, I think.

Brian Dawson: Awesome, awesome. Yeah, I think there's important points there. First, I do have to joke that Bob had a slide where resilience was sort of in the center of the slide and I think, you know, for your shared services in DevOps teams there has to be a certain amount of resilience, and for the people – you know, the development teams there has to be a certain amount of resilience because it's hard for all of us to change. It's not easy. And what I love about a common theme across all of your responses that I want to highlight is, you know, at the end of the day we talk DevOps, CI, CD, all of these practices, but it's people that deliver software, right?

And you can't unlock really any of the potential of what we're talking about until you tend to and address people, right? So I appreciate you bringing that up. So now what we're going to do is we have a minute to bring in one question. We'll see if maybe we can get two answers to it based on time, and this question was actually asked multiple times by multiple people, and that's – you know, to paraphrase it, how did you gain management or C-level support for your DevOps initiatives? And Jennifer, I'd like to start with you to see if you have some thoughts on this.

Jennifer Hansen: So this is actually an easy one for me to answer because our commitment and journey has been bottom-up and top-down, so we have had tremendous support for the Agile DevOps transformation, and, you know, how you work is a key thing. You have to really understand that. And then working back with the teams, understanding the challenges, figuring out how to operate at scale is what makes you successful. So I can't say we would've got here without that commitment right from our CEO and the rest of the leadership and teams enjoying how Agile is, right? The whole waterfall and handoff wasn't quite working as effectively.

Brian Dawson: Thank you, thank you. Now I would love to dig in. An apology to the rest of the panelists. They sent me a remote taser and they're tasing me letting me know I'm at time now, so I just want to take a moment to thank you guys so much, not only for the time you put together in preparing for this panel, the time you spent on the panel, but the experience that you've acquired doing this in your current roles and the fact that you're sharing it with a larger group. I do have to say, by the way, before I let you go I do have to say you all look phenomenal; perfect framing, perfect pictures, so thank you, thank you so much.

Jennifer Hansen: You have to thank Jude for that.

[Laughter]

Brian Dawson: Thank you so much for your time, and if you get a chance, panelists, there's been a lot of questions for you, please take a look at Zoom. Maybe we can get you guys over in Slack or chat to participate with our participants, and I look forward to our next conversation.

Jennifer Hansen: Thank you, Brian.

Adam Robertson: Thanks, Brian.

Robert Kelley: Thank you, Brian.

Daniel Ritchie: Thank you.