Key Takeaways from Continuous Discussions (#c9d9) Episode 66: Scaling Agile and DevOps in the Enterprise  

Written by: Electric Bee
1 min read
Stay connected

In a recent Continuous Discussions (#c9d9 ) video podcast, expert panelists discussed scaling Agile and DevOps in the enterprise . Our expert panel included: Gary Gruver , co-author of “Leading the Transformation, A Practical Approach to Large-Scale Agile Development,” and “Starting and Scaling DevOps in the Enterprise”; Mirco Hering , a passionate Agile and DevOps change agent; Rob Hirschfeld , CEO at RackN; Steve Mayner , Agile coach, mentor and thought leader; Todd Miller , delivery director for Celerity’s Enterprise Technology Solutions; and, our very own Anders Wallgren and Sam Fell . During the episode, the panelists discussed lessons learned with regards to leadership, teams and the pipeline and patterns that can be applied for scaling Agile and DevOps in the Enterprise.

Scaling Agile vs. Scaling DevOps

Talk about both the technology and processes when it comes to DevOps, says Hirschfeld : “One thing I would say is that people often get confused about with DevOps is DevOps is very much process-focused. But, it's okay to talk about the tech. I'm the type of person that says let's actually think about the tech. It's okay to be excited about Chef, Puppet, Ansible, CI/CD pipelines and all those things. I think it’s healthy to infuse DevOps tech with DevOps the process.”

DevOps is all about plumbing the last mile, explains Wallgren : “One of the things we've seen quite a bit with our customers and in talking to people is that (probably unintentionally) Agile became a local optimization for engineering and development. We got into the whole aspect of product ownership and Scrum and stories and all of that stuff. One of the things I talk about very often is how, practically speaking, DevOps has become a lot about plumbing the last mile. Plumbing the last mile from when you check in a piece of code to when it's available for the customer. Engineering teams have become pretty good at delivering functionality. They would have two week spreads to deliver functionality, but then it would take 90 days to put it into production. There was a lot of work in progress on the factory floor so to speak. We've got a little bit better at recognizing that as a problem. It's one of those lessons where you haven't really solved a problem until you solved it all the way through to the end user. Agile was all about that from the beginning, but I think practically speaking a lot of us looked at it as a way to solve engineering problems, and left the whole delivery to customer part as an exercise for the art people to solve.”

It’s not just about Agile vs. DevOps, scaling itself is a whole other ball game, per Hering : “I think scaling is entirely different to just doing Agile or DevOps. Someone told me once if you have four people in a room, you don't need a methodology or any kind of formalized way of working. I think as soon as you get to complexity - to scale, to distributions - as you're working across many different locations, that's when you really need to start formalizing. Where people struggle is either not having anything formalized, or being overly prescriptive and completely drowning out of the innovation part. That's why we are struggling with that because it's the tension between those two and that's very much the same for Agile and DevOps.”

Gruver discusses the challenges of releasing code when scaling up: “The challenge became when Agile scaled into the enterprise, releasing code in tightly coupled systems became more difficult. You either started with small teams that were able to do that or you went into organizations that had challenges and architecturally changed it towards small teams that can independently develop, qualify and deploy code. The reality is most large organizations don't have that architectural decoupling, so they have to coordinate the work across a large number of people and that is challenging. So, when Agile scaled into the enterprise a lot of people left that behind because it was hard. I think DevOps really started to gain momentum because Agile dropped that basic principle as it scaled in the enterprise. It became a lot more about technology with Jez Humble’s book around Continuous Delivery and infrastructure-as-code. Those were the types of things that were making it difficult for release code on a frequent basis and maintain quality.”

It isn’t enough just to say you are Agile, says Fell : “If you think about the Agile manifesto, I think number 12 is ‘Enable a Continuous Delivery of valuable updates to the end user.’ Even back then they knew that just being Agile in itself is not enough, you have to actually deliver that to somebody and taking those practices and pushing them downstream.”

What does it really mean to be Agile? Miller explains: “The core of being Agile is to be able to incrementally deliver. So, I'm going to go put my developer goggles on and see how many times I can tell you where I checked something in and everybody gets late, and my answer is, but it works on my machine. It might not work in an environment or it might work on half of the people's machine - I think that DevOps helps to cure that. Are you really Agile? Unless you're continuously delivering, I think that's a state where everybody wants to be in. One of the first questions I ask when I'm going into an organization that says they are Agile is, ‘What's your release cycle look like?’ A lot of times I hear, ‘Oh we are Agile, we release every three months.’"

Mayner talks about the Agile mindset: “When we began the Agile conversation we heard it said a lot that it really is much more about the mindset. The Lean Agile mindset, both at the practitioner level, at the leadership level, and throughout the organization. I would say as we've moved into the DevOps conversation it's really the same. I've seen a lot of conversations around technologies and practices, but to understand why this works differently, why this is more than just different tools and different labels for things, it's a much deeper, richer conversation.”

Lesson #1: Leadership

Hirschfeld on technical debt and DevOps: “There's an element that can cause external observers, especially in DevOps, to get frustrated which is what we call Yak shaving. It’s the idea that before you can do the thing you want to do, you got to do something and there's a prerequisite for that and a prerequisite for that and a prerequisite for that. So one of the challenges you get into is that you can be in a DevOps case, where your team feels like it's doing nothing for a couple of sprints because it has to do all this prep in automation or refactoring so that you can put it into a CI/CD pipeline. My advice for people is to go in armed up with understanding that there's a certain amount of technical debt paid to understand what technical debt means. Accept that your team might look like they're completely at a standstill because they're paying down technical debt.”

DevOps leaders need to give up the illusion of control, per Wallgren : “The leadership has to learn to get comfortable with, I was going to say giving up control, but it's really just the illusion of control. I had a great epiphany a few months ago when I drove for the first time on the Autobahn. I say drive, but actually I was driven. I was in a car that steers itself and the first few times I took my hands off the steering wheel I was like, ‘Oh crap.’ I was really nervous, foot over the brake, all that kind of stuff. But what you realize is the machine works perfectly, it's going be a lot better at paying attention to the car in front of me and braking when necessary than I am. So I'm not giving up control, I'm giving up the illusion of control because I'm human and I'm going get distracted and computers, as a rule, tend to not get that distracted. You also have to be willing to relax and learn to live with DevOps.”

Mayner explains that it’s not enough to want quality, you need to know how to get there: “As Deming said, it's not enough for leaders to know that they need higher quality, they have to know the way. They have to know what it is that we must do because who else has the ability to change the system? And it's not that we have bad people, we have great people, we've hired them. It's the system. We have to know where it is we are going and even more importantly to get to my OCM hat on, that's going to require people to change at every level. Down from the team level to the mid-management layer. The role of the leader is to create the safe space.”

Agile and DevOps can be challenging, but don’t let it stop you from reaching your goals, advises Hering : “I think leadership is a fantastic first lesson. I'm not sure whether you've seen any of the DevOps Enterprise Summit talks with Damon Edwards , but he goes with this curve where you're improving and then it goes a little bit back or gets hard and then you would transition into a new phase. A lot of people get stuck in that first bump, and that's where leadership grid is required. We’re okay, we've done Agile for this small digital application, we have done DevOps for our relatively trivial stack, but now we have the tricky, complicated application architectures that everyone refers to as the big monolith. That's not three months of quick re-architecting, that's a few years of loading the monolith. If you don't have the platform leadership then you will get stuck or worse, you will start looking for an achievable goal. People might say, ‘We've got you three months deployments, we're done here. We don't need to invest any further, this was hard enough.’”

Leadership style is dependent on the type of architecture you have, says Gruver : “I think leadership style depends on if you have a loosely coupled architecture, or not. If you do it's about empowering the team and removing roadblocks. If you don't, you have a tightly coupled system, and somebody needs to be coordinating the work across teams. How the teams work is a second order of fact; how the teams come together to deliver value is a first order of fact and that's a job that's traditionally given to executives and they need to play that role. Executives need to prioritize things that are slowing down flow through the organization. Microservices will always go faster, small teams will always go faster. But I think the DevOps principles have the potential to add the most value in large, tightly coupled organizations because it's about coordinating the work across teams.”

Treat your pipeline like a product, per Fell : “We have a concept here at CloudBees of treating your pipeline like a product. You should approach DevOps in an Agile fashion, you should approach Continuous Delivery in an Agile fashion, you should approach your pipeline in an Agile fashion. It’s your Minimum Viable Pipeline instead of Minimum Viable Product. These are all things that you can do to help your team start small, prove value, make sure you're going in the right direction, make sure that when you are doing your minimum viable that it's going to handle as much of the different use cases that you might run into.”

Don’t get caught up in speed, make sure you are doing things right, says Miller : “One thing I always like to be really cautious with about speed, is that it's faster time to market. It doesn't necessarily mean that you're going to be building software faster. It's the fact that Agile means that one part of our intention is to get software to production faster. It doesn't mean that if you undergo an Agile transformation from a leadership perspective, that you should be focusing all your time and attention on measuring how the organization is being able to produce software faster. Because you're always going to get the answer you want, right? There's this false impression out there that we're going to do, we're going to do DevOps, and we are going be developing five times as fast as what we're used to. No, we want be developing the right things.”

Lesson #2: Teams - and -

Lesson #3: The Pipeline

Running a bit over, we turned to the panel to share their lessons learned and proven patterns and tactics for scaling agile and DevOps- either from the team's perspective, or from the perspective of the pipeline - processes and tool-set.

Make sure your teams own not only code, but infrastructure as well, advises Hirschfeld : “If you're looking at DevOps, teams are going to own their code into production but they don't want to own the infrastructure. You really need to be looking at site reliability engineering, so that your DevOps teams can focus on their job owing code into production. Other teams that have done DevOps and collaborations but focusing on owning your infrastructure makes it that much more efficient.”

Leaders should help their teams solve problems, not just point them out, per Wallgren : “We have a phrase here at CloudBees where we say don't just be a problem pointer-outer. Grab a shovel and help dig the trench. It's not leadership to just poke at things and say that's wrong, that's broken. You have to come with a proposed solution or come with an opinion that you can help out. I think in any large scale organization, it’s that times a hundred. And if you're in a situation like open source where you only have a carrot, but not a stick. For better or worse, you better get used to carrots.”

Hering notes the importance of autonomy in teams: “You need to enable teams to be autonomous. The key to this is easy consumption. So it's not, we're going to create this DevOps framework for our organization and there is a formal 300 page document to fill in if you want to do a deployment. It needs to be on an API base around this loosely coupled system; the same is true for the DevOps tooling. To make these services really easily consumable, create a DevOps platform that is common language across the organization that people can leverage, that provides common language, and gives you the flexibility to build on top and there's a platform team that maintains that platform.”

Again, how you define team and how teams function depends greatly on architecture, explains Gruver : “When I think of team, it depends on how tightly coupled your architecture is and what you're trying to do, and how often you are trying to raise it. If that can be a team of five people, great, you're going to go fast, you're going to move quickly, this stuff is easy. If that's a group of 1,000 people, then I think the team is the executive leadership over that thousand people group. They need to map out how they are getting ideas to the customer and what the business’ biggest sources of inefficiency are. When I think of the team, I think of the team that has to coordinate the work to independently develop, qualify, and deploy code and manage it in production. Make that team as small as you can because it's hard to coordinate work. If you can get it semi-small but you can't get it any smaller, look for an architectural change that would maybe decouple two big parts of it.”

DevOps is a great way to give teams the ability to be creative, says Miller : “We have to keep in mind that it is the teams decoupling monolithic architectures. In my experience, I’ve had a lot of success in highlighting that there are a lot of dependencies across these teams to manage. Let them creatively think on the best ways to do that. DevOps is one of the most amazing ways to do this. You think about dependencies and integration issues, that’s really the idealistic solution. Regardless of leadership, the boots on the ground and creating teams that have all the skills necessary to get it into production is ideal. Let the smart people on the ground solve problems.”

Watch the full episode:

Want more Continuous Discussions (#c9d9)? We hold our #c9d9 podcast every other Tuesday at 10 a.m. PST. Each episode features expert panelists talking about DevOps, Continuous Delivery, Agile and more. Check out all past episodes and panelists here

All Blog Articles

Stay up to date

We'll never share your email address and you can opt out at any time, we promise.

Related Content

Maximizing Release Train Efficiency with Release Orchestration

Blog
All
6 min read

Automating our Software Release Process Internally at CloudBees

Blog
4 min read

How Jenkins® admins can provide scalable continuous integration (CI) for software delivery teams

Blog
9 min read

What Is Everything as Code?

Blog
8 min read

Advanced Deployment Strategies with Argo Rollouts

Blog
6 min read

YAML Declarative DSL Preview for CD/RO

Blog
6 min read
Product Capabilities
Resources
Why CloudBees
© 2023 CloudBees, Inc., CloudBees® and the Infinity logo® are registered trademarks of CloudBees, Inc. in the United States and may be registered in other countries. Other products or brand names may be trademarks or registered trademarks of CloudBees, Inc. or their respective holders.
Privacy PolicyTerms of ServiceCloudBees Vulnerability Reporting