Episode 74: Live from CloudBees Connect- The Reality of Delivering Modern Software Part 2

Brian Dawson: So today, I as the host of the DevOps Radio podcast, which if you haven't heard it before, self-promoting plug, please go download it. If you have, tell a friend. We've decided to conduct DevOps Radio live here at CloudBees Connect and we have a panel of real-world practitioners and leaders that are going to participate with us. Unfortunately, Aswini, the lead CI automation engineer at AIB is not able to join us. We may get her on later. 

But we do have Sanmat Jhanjhari, lead DevOps engineer at Nationwide Building Society. Hello, Sanmat. How are you doing today?

Sanmat Jhanjhari: Hi. Yeah, not too bad, thank you. How about you?

Brian Dawson: Doing good, doing good. Thank you for joining us. Heck a background, heck of a background. It reminds me of the cogs we had on our earlier screen and people working together. Next we have Jimmy's colleagues, Aoife Fitzmaurice, architect for Enterprise Cloud Computing at Fidelity, and it looks like Jimmy's been giving you a lot of plugs with Cloud First and Cloud Native. How are you doing, Aoife?

Aoife Fitzmaurice: [Laughs] Doing good, doing good. Jimmy is a hard act to follow.

Brian Dawson: [Laughs] Great. Well, I'm sure you'll kick his butt, don't worry.

Aoife Fitzmaurice: [Laughs]

Brian Dawson: And then of course you guys have already met Jimmy McNamara, software engineer and product manager at Fidelity, a colleague of Aoife's, and I'm just joking, Jimmy. Thank you. It was a great job in the keynote. So at this point why don't we go ahead and remove the slide and jump right into our panel topics? So, you know, in planning for this we identified a list of high-level topics that we wanted to cover as a team, and as soon as I can locate the screen where I have those topics [laughs] we will jump in. 

I seem to have lost it in one of my 15 different sets of tabs I have going on. Now here we go. So I'd first like to start with the topic of defining DevOps in the enterprise. I know we go to conferences like this. We always talk about kind of the happy path to DevOps, what the ideal is, but I'm wondering, and I'd like to discuss, are there differences in the challenges and the benefits of DevOps in the enterprise? 

And I'd like to start by going to Sanmat. Sanmat, Nationwide Building Society operates at an enormous scale, so I particularly want to start with asking you what's your definition of enterprise DevOps?

Sanmat Jhanjhari: From the Nationwide perspective what we say is you build it, you run it, so – and we proved it last year. We rolled out Atlassian suite, and literally in eight to ten weeks, right from purchasing the software through to delivery to the first team onboarding. We did it in eight to ten weeks, and now the team still builds it and continue to evolve it and run it and it's at the moment about 5,000 users plus in less than a year. We also believe in building process-dependent environment, not the person-dependent, and one of the biggest impediment in implementing enterprise DevOps we think is thinking and working in silos. Yes, it gives accountable freedom but it comes up with the increased overall organization cost and it may also lead to the inconsistent engineering practices.

So we, from the DevOps enablement engineering team, what we do is we enable and run the shared services and implement DevOps practices for every scores or the target team. The way we adopt or the way we work with the team is we engage with them, we understand their flow of work. Their list stream analysis is a key thing also and understand what the current pain points are, how they want to see the target. Then we work with them to achieve their target, defining the MVP. We also shadow them and they shadow us to make sure they are confident to build and run their own system, which we help them to build it at the beginning. It also gives them a confidence that they are building their own capability within the scores; they can build and run it and also mobilize them very quickly. 

Brian Dawson: Awesome. Thank you, thank you. And I'd then like to Aoife, take it over to you and understand how is your experience at Fidelity, in particularly with the cloud like or different from what Sanmat just described?

Aoife Fitzmaurice: Yeah, it's kind of interesting, right? And I don't know about you, Sanmat, but it's more kind of a culture shift versus anything else really. It's a huge kind of change in mindset in terms of like traditional kind of setups where you had, you know, siloed approaches to your developer, your QA, your testing, and then into your kind of operations kind of space. So, you know, it's kind of interesting with Fidelity because you have different teams that you're operating against and there's different maturities that exist, so some teams are very mature and they are doing the DevOps practice, so you build it, you own it, all the same things that you're talking about, Sanmat.

But then you've got other teams that are just not there, so they may be more traditional, they're dealing with institutional applications, they have a lot of audit, they have a lot of compliance, they have a lot of restrictions in that space, so then how do they kind of transform? They want to transform but then how do they transform with those limitations? So it's kind of interesting to be able to operate against both of those types of teams, you know, and then help them in that journey to the DevOps framework, and Jimmy is actually – he's quite good because in our kind of offices – obviously with COVID and everything nobody's in the offices anymore, but Jimmy is throwing up all of these kind of posters around for our kind of teams and it's like you build it, you own it, you architect, you own this and all of these types of kind of, you know, mottos are kind of pasted around the place. So from our perspective we're truly a DevOps team.

Brian Dawson: Awesome. Thank you, Aoife. So, you know, just to emphasize that you guys really had to start with especially in terms of your complexity – Aoife said your hybrid portfolio with building the culture or unifying a culture across that portfolio to move forward.

Aoife Fitzmaurice: One-hundred percent. It doesn't – you can't progress without that culture. You need to have that drive to actually – it can't be centrally and artificially kind of owned. You know, it needs to be at the team level itself and they need to have the control to be able to make those changes. Obviously there's other kind of fundamental things that they need to work within but, you know, within those kind of constraints, allowing them to be able to kind of make that transformation and applying that culture into their teams is key.

Brian Dawson: Okay. Excellent. Thank you, thank you. So I'm going to go ahead and move forward in the interest of time to our next topic, and this is about balancing security, compliance and quality with speed, so especially in financial institutions, right? How important is it that you maintain or implement governance measures while you build this culture and this movement to modernize? Aoife, I'm going to go back to you. How do you balance these elements? What's most important? 

Aoife Fitzmaurice: I know. We should just leave, shouldn't we? We should join a gaming kind of industry or something like that, you know, get rid of all these restrictions and all this in compliance and SOC and all of these types of things and, you know, get onto more exciting stuff, but unfortunately for us, you know, banking institutions, you can't get away from it. It's your bread and butter. It's not a choice. It's fundamental to how you kind of operate and how you do your CI/CD.

So regardless of, you know, how far you are in our journey you must kind of incorporate these principles in terms of security, audit compliance, quality, all of these kind of things. So I think Jimmy kind of referred to it a couple of times in terms of like some of our models in Enterprise Cloud Computing and it's passed to the cloud in a safe, secure manner. That's all it is. It's not a question of, you know, if there will be a breach. We know potentially there will be a breach down the line.

It's going to be a matter of when, and when that breach does happen have we done everything within our control to ensure that that is not our fault, that we've secured all of our systems, that we've ensured that the quality of the code is good, that we're doing good SDLC against the actual applications themselves. So there's kind of a couple of layers probably within that, but I'd be interested to hear you, Sanmat, and kind of what you're kind of doing in your space, but from our perspective there's probably kind of two things. You know, part of it is the ownership of the developers themselves to feel like they must be in control and understand the reasons for having good quality code and to going through those security checks to ensure that that code itself that they're developing and pushing out is of a standard, and then from a central perspective, you know, there are things around compliance of tagging and so on and so forth that we can monitor and make sure that some of the practices that we have set out, so things like you must redeploy your application every X number of days, your base images, et cetera, can only be X number of days old, things like that. 

You know, practical things that you kind of have laid out in terms of your policies, that we do enforcement against that. So we have things like you can't do your promotion to release if your images are, you know, X number of days old; you therefore can't deploy that image, you know, and if you have deployed an image and it's X number of days old, you know, we have controls in place then that basically alert out to say that your deployments are too old, you need to redeploy your applications. So there's a lot of kind of the back and forth between the two things, you know, how do you enable fast deployments but still you have to do it in a safe, secure way. 

Sanmat Jhanjhari: Yeah. That's a really good point. I think the shifting left is always – you must have heard – everyone must have heard about shift left. Everything should be a developer end. So the way I think we – from the Nationwide perspective, I think any financial industry is all these elements are equally important, and you wouldn't be able to release the code into production if the code isn't secured. At the same time if in securing the code it takes you so long that it's too late to deliver in the market, that you're out of market so you can't get the benefit of it, and I must admit here I can name the CloudBees Jenkins helping a lot here in Nationwide, is we are creating templates and for the team those are never CNCI. So the templates give them the confidence that within the pipeline they have everything baked in; the security, compliance, approvals, quality checks.

So that gives a developer confidence that if I'm a releasing a code into production I'm safe, I'm secure, and I'm a full degree of confidence, so there will not be any delays in getting the approval because you know the financial organization CAB approval is tricky. So if you have all of these practices, which are baked in with the templates, you don't need to worry about it, and that's really a great feature from the CloudBees Jenkins. At the same time is we – again, you mentioned quite right that we have a degree of variance in our teams, like we are 100 percent Cloud Native application when we have mainframe and Unisys and legacy application. And when we work with those teams we understand that they are still starting their journey in the DevOps area so we give them some CI/CD to start with the basics, to get the automation in place first then go into build it and run it mode. So I think everything is quite important within the space of the entire software development lab cycle.

Aoife Fitzmaurice: Yeah.

Brian Dawson: Awesome answer, awesome answer. Did you have a comment, Aoife or Jimmy? 

Aoife Fitzmaurice: No, it's – the correlations are huge, right? You know, previously there would've been a lot of kind of let the developer kind of do what they want to do, you know, let them have the freedom of that choice, but I think we're kind of coming around, you know, just in terms of how things have been applied culturally and it's more kind of like the developers just want to be able to push out the code, so give us those standard templates and pipelines that we can leverage so that we can just pump this stuff out. I don't want to have to worry about having to do, you know, SonarQube or Veracode or whatever else that's there; just plug in all the bits and pieces that I need to do and just let me get at it.

Jimmy McNamara: Yeah, I think if you create – there's always going to be a natural tension between the various areas in terms of what _____, then you've got your security and governance also having a natural tension. As long as you create platforms for open discussions and everyone understands, you know, what's going on, what risks are available or what risks are there then you can put the controls in place so that you can enable that fluid movement of features from left to right. Everybody wants to help support the business, right?

Brian Dawson: Right.

Jimmy McNamara: It's all in everyone's interest. So it's about creating good forums and good communications and dialogue between the folks to enable that.

Brian Dawson: Yeah, it's a very interesting statement that you make there, Jimmy. It came up a bit in our discussion in the keynote and discussions I've had before, is generally if you hire the right people they do their work with good intentions and they'll be fully on board with standardization and governance, and in fact will help implement that in a smart and intelligent way provided that they're given context and they're read in and there's a bidirectional discussion. Now this actually – you guys set me up well. I'm going to hire you guys – hire y'all on DevOps Radio because, you know, this shifts us to shared ownership and empowering the practitioner. 

You know, people talk about happy developers or productive developers. Productive developers stay, and I'd extend that to say practitioners, whether DevOps engineers, operations engineers or development teams, and you talked a lot about culture, Jimmy. How does shared ownership and empowerment enable faster adoption of these practices at scale? And I think specifically for you Jimmy how have you balanced standardization and autonomy to provide for empowerment of developers?

Jimmy McNamara: Yeah. I mean it's like a gearing. That's like the analogy, right? So you've got common needs, right, and that then makes sense to take those needs and make them available centrally, because as long as they're common across the organization then that means that the folks who are looking to develop software don't need to redo stuff, right? They can just take it up, do it once, right? It's a gearing effect, right, so that you're enabling at the center.

And if you're making features and capabilities available that are commonly needed across the organization and as long as you provide that service to a really, really high level then that's value. So when people and developers see value then that's not a problem, especially when you have really smart developers because they realize that that's actually going to help them, right? 

Brian Dawson: Right.

Jimmy McNamara: So when you're helping people that's always good. So I think that's – it's a balance and it's not trying to be overly prescriptive or constraining; it's about having a good dialogue, having a good communication and say, "Hey, you guys, you're doing this and actually there's lots of different businesses in our company doing this." You know, that makes sense to kind of centrally harvest that back, right? And, you know, part of this central group is to (a) identify that, (b) help facilitate communication across their various business units so that they understand that and then it becomes, well, you know, actually, yeah, you guys take that one because that makes sense, because that then means that we can all focus on what we need to do in our various business areas.

Brian Dawson: Right.

Jimmy McNamara: And actually, when you harvest back to the center then you can get lots of different point of views and actually harvest back the best practice as well, right, for that whatever you're dealing with so that, you know, you're getting – in some ways you're almost like a central consultancy. You know, you got visibility across all areas, all the BUs, and then you harvest back what makes sense and you harvest back to practice with so that then the rest can focus on generating business.

Brian Dawson: I love that term. I hadn't really heard it – harvest back to the center. I think that is a really interesting answer to how do you empower people but then scale or leverage that empowerment to scale? Sanmat, how are you encouraging collaboration and adoption and, you know, at the same time empowering developers and teams to use the tools they want?

Sanmat Jhanjhari: Yeah, the tools I think is a hot topic in almost every organization I believe, and _____ the DevOps tools. If you're _____. Nationwide believes in empowering people and creating an environment that you have the accountable freedom, and that's one of the key pillars of Nationwide leadership framework. So the way I look at it is like developers have – you know, they constantly look for the new library specific in this open source world and experimenting new tools, technologies. So then why shouldn't it be the same approach for DevOps tools? 

It is important to understand why teams are choosing for their own toolsets, why they want to do that. There could be a gap in the enterprise tooling capabilities out there or there may be a better way which the team may have devised to deliver something or there may be an out-of-date tooling strategy that the organization may have, so I think nowadays it's quite important to reassess the tooling strategy every 9 to 12 months, and the way we are dealing in Nationwide is to have the clear exist strategy from the tool and developing the adaptive pipelines so that I can plug and play any tools, like if any tool you can security scan. So you can turn one tool off, turn on another one; it's just a standpoint change. 

So creating that flexibility in the pipeline helps to quickly migrate from one tool to the other tool because it's quite important nowadays, and when we're moving into the cloud world as well things are getting into push models rather than pull models. Everything just you can't release. You don't have a choice. You have to uplift your technology or uplift your code in order to keep up with those dynamic changes happening. We also have the _____ tooling governance board just to – the idea is not to command and control; it's to come up and talk about it so that the rest of the community and Nationwide understand that there is a new capability coming out there. 

Why don't we try a lot and why don't we uplift our current pipeline to cater to the latest development needs and also to assess whether the particular capabilities – fit for use, fit for purpose – is we don't want to – no one wants overcrowded tools because it's going to cost a lot for an organization paying for the same capability paying to two different vendors, as an example. So – and also we said, "Teams, please remember DevOps is a culture, not the tools."

Brian Dawson: Right, right, right. Unless it's a CloudBees tool, of course. No, just joking. So I'm kind of scared. I'm going to go over to Aoife. I'm worried that Jimmy, Aoife, Sanmat, you guys are going to take my radio job with the way that you guys are handling these answers, but I'll risk it. Aoife, do you have any comment on this? Do you have your own perspective on how you see practitioners being empowered at Fidelity?

Aoife Fitzmaurice: Yeah, it's kind of an interesting dichotomy that we kind of have to deal with. So I mean within our company it's we're many companies in a single company, so that's – you're kind of dealing with all of these different business units, as we call them, and they have an equal voice. You know, so how do you kind of get that consistency across those business units to then say, okay, our role is to provide that central stack for you, you are our – you know, we are your provider, you are our customer, so we want to do the best thing for these business units themselves, so trying to get the business units to work together to come up with that common stack that meets the needs of the developers overall. Because after all, if you kind of take it above the developer – the needs are common across the kind of stack itself, so you don't want to have so much diversity that you're just carrying a lot of weight as you kind of evolve these tools as well, which is key because it's not necessarily that even in the CloudBees world, right?

You know, five, six years ago it was all configuration based and it went into templates and now it's pipeline as code, so you want to be able to evolve within the stack and also kind of make sure that you plug any holes across the stack and make sure then that that's viable across all of the business units that are there. So, you know, it's kind of interesting. We had like a situation, you know, over the last year or so where it was more of a dictation. You know, some choices were kind of made that essentially were pushed onto the business units and, you know, we spent a year of like, you know, back and forth _____ for this. So then, you know, we learned from that. You know, we don't always get it right and we just kind of took a step back and said, "Okay, let's make this a consensus, let's work together towards the next thing.

If this choice was not correct let's get the right choice. Let's work together to make that choice." And as soon as we kind of changed that conversation it becomes less about stop restraining me, allow me to be an individual, allow me to do my own thing, and it was more around well I don't want to make that choice for just myself; you know, I want you to own it and I want the rest of the kind of business units to use it, so how can we do that? So all the things you talked about, Sanmat, about _____ scale, all of that stuff we do ourselves as well. It's great stuff. It's a complex area. So huge, you know?

Brian Dawson: That's phenomenal, a phenomenal answer. I hear a little bit of what's often said, the adage, "Move slow so you can move fast," and I think sometimes it's important to take the time to build the dialogue as you said, Jimmy, to try to establish consensus that results in people taking on shared ownership and feeling empowered at the same time, and that may take time to do. I think it's not easy, right? But if you invest up front you're going to reap returns faster, longer, better later. So now's a great time actually, Aoife, your question – I mean your answer leads me to a question coming from chat that I'll first direct to you and there may be – Sanmat, you may want to followup – and this is how do you draw this borderline between DevOps and developers' responsibilities or is there even a line to be drawn. And Aoife, why don't we start with you and then we can go to Sanmat.

Aoife Fitzmaurice: Yeah. There kind of isn't really a line to be drawn. It's so blurry. So I mean if I kind of draw on what we provide, we have the platforms themselves, we provide some of the services on top of these platforms so you can integrate natively with, you know, things like that your quality scanning tool, your security scanning tools and things like that. And then on top of that, you know, we kind of work with the business units essentially to define what templates and standards they want to push against their business unit themselves. 

So across the business units that will vary, but we want to enable that out of the box. So things like some of the feature functionalities that you can do in terms of CloudBees, for example, with all the templates and creating pipelines from those templates, you want to hook those in because those are essentially supported. And then, you know, if you kind of get beyond that layer then it's almost like, okay, if you are such a snowflake, if you are so special then, you know, allow them to kind of do that – allow them to go off on their own so they can support their own kind of journey in that space and just let them go for it with that, and those are usually kind of like lab-type work, innovation, things like that, and then they come back to the central kind of forum to kind of actually productionalize this and bring this forward; and therefore, they want to conform to their business unit standards and then to start to leverage the enterprise tools themselves to be able to do that.

So it's never a clear line. It's always quite blurry, but it's always for the common good of just getting, you know, your features out as fast as possible, and if there's any kind of gaps or any slowdowns within that process then we work to kind of reduce that down into making the right choices to kind of make sure that you can go at speed. 

Sanmat Jhanjhari: Absolutely. Yeah, I think we feel the same. I think the way we look at it is if you draw – even thinking about drawing a line then that means you're creating silos.

Aoife Fitzmaurice: Yeah.

Sanmat Jhanjhari: Which you don't want. So as I explained earlier, the way we do it is we engage with developers, we engage with the team; what are the objectives, how do we want to design your pipeline? Yes, there are guardrails. If you are joining a party you need to follow the rules, right? It's as good as playing a game, right? So there are certain guardrails which we obviously define to make sure service is protected.

And in the interest of the developer, obviously in the interest of delivering the code securely, safely and at the speed and the production line. What we also create is – specifically talking about the pipeline template, just as an example, so we create MBP, we create a baseline product and then we encourage developers to collaborate so that we create an open source community within Nationwide itself so that can be considered – templates have been defined but can be consumed by the other scores, because normally what we've seen in the past is that everyone is doing their own bit. Even if I have to do a versioning of a library everyone has their own code _____ enterprise the versioning concept is the same with everyone writing the same code. So what we do is we create those templates, global libraries, and help developers to contribute there, collaborate there so that every other team can also take the benefit about it. So that's kind of our approach to encourage them to work with us to help them help them basically. 

Brian Dawson: Awesome. Thank you, thank you. So we'll move into our last question. We have a few, so I'm going to munch a couple together for you, Jimmy, and at the root of this is getting management's support or overcoming C-suite resistance. So can you explain if you've ever run into resistance in getting management to support your DevOps initiatives? And if so, how would you convince them to support this?

Jimmy McNamara: Yeah, I think – I mean we've been lucky. You know, our leadership has been very, very supportive through cloud, through DevOps, but, you know, like anything else, right, if you demonstrate value to anybody and present that value and that value for us it's about using our own stuff and showing that that's improved what we've done and then using data to show the outcomes, so if you have any executives sitting in any organization and you're presenting to them something that they can potentially leverage themselves to improve their own organization and meet their own goals, right, most, if not all people, are going to be very supportive of that. So it's about articulating a vision of value and then ideally demonstrating that in a very tangible way with the associated data so that basically what you're saying is, you know, we think this is good, we think this is ideal here, and here is it done and actually you can use it in your organization and it's going to improve these areas. So I think it's about ensuring that it's a value-based discussion, back it up with data, and ideally show it done.

Brian Dawson: Awesome, awesome. You know, I would add to that a bit of opinion and lesson that also came from another panel we did, is – it sounds like I heard both a bottoms-up and top-down approach. So it sounds like, look, if you got to just get moving, get moving, prove some of the benefits, and then work with management to get support. So everybody, I think we're at about time. We gotta move to getting some of our attendees some prizes and then getting them into some very valuable workshops. Before we wrap though I want to go around and see if anybody has any final words. We'll go ahead and start with our right. Jimmy, do you have any final words for our audience?

Jimmy McNamara: Yeah. I mean for me I think it's an extremely exciting time in IT. I think it's a very creative age. There's a huge amount of new feature functions available, a huge amount of creativity in terms of creating new, and it's just – you know, it's a really exciting time and it's great to be involved in an area where we can kind of facilitate the creativity and leverage the type of new technologies and new platforms the industry is pushing out. So, you know, it's a great industry to work in at the moment.

Brian Dawson: Awesome. Well, thank you very much for sharing your experiences. I think it makes it even better. Sanmat, over to you. Do you have any final words?

Sanmat Jhanjhari: Yeah. I think, again, in this rapidly-changing the technologies don't over rely on tools. DevOps is culture, not the tools. And also build it, run it. That will bring a lot of value and also the environment will be much more interesting to work in.

Brian Dawson: Awesome. Thank you. And Aoife, over to you. Any final words?

Aoife Fitzmaurice: Yeah. I just find these kind of connects kind of heartening because it really just confirms as you kind of talk to other people, other voices in the industry, that you're on the right path. You know, there's a lot of correlation between us, you know, the things that we're doing. Somewhat slightly different, but the main kind of building blocks are the same, which is great and it just means that, you know, our outcomes are just going to be that much more successful, which is huge.

Brian Dawson: Agreed 100 percent. I love these discussions. I love hearing, learning, sharing, and to that end I want to thank all three of you, and Aswini who didn't get to join us, for the time that you've taken to be here today, the time that you've invested preparing for today, but not only that, the time you've spent trying to move things forward, learning, and then sharing that experience here with the audience at CB Connect. Again, thank you very much. If you can and you have the opportunity please jump over into chat or over into Slack and we will be there answering questions and maybe we'll be lucky enough to be joined by our all-star audience. Thank you, Aoife. Thank you, Sanmat. Thank you, Jimmy. Over to you, Jude.

Aoife Fitzmaurice: Take care. 

Sanmat Jhanjhari: Thank you, Brian.