Stefan Simenon - Learning and Unlearning DevOps

Sacha Labourey: This is DevOps Radio. I'm Sacha Labourey, CEO of CloudBees. I am with Stefan Simenon, who works for ABN AMRO in Amsterdam. Hello, Stefan.

Stefan Simenon: Hello, Sacha.

Sacha: So a bit of a noise around us. We're in the middle of Jenkins World. So I hope there is not too much noise there. Stefan, you're really the guy when it comes to DevOps at ABN AMRO, right?

Stefan: Yes.

Sacha: Tell us about your role there.

Stefan: I'm currently heading the tooling and the software development CV with the Center of Expertise team, which all kinds of specialists are in. Together with the CVs, we define all kinds of standards and guidelines in the bank. We work together with the Agile teams who adopt the standards. So we don't develop them alone. We develop them together with the Agile teams to get the buy-in.

Sacha: What we were discussing before essentially is a practice. You really started that effort. You started from scratch essentially, from a DevOps standpoint.

Stefan: Yes. We used to have a waterfall organization four or five years ago, and we have transformed into an Agile organization. We finished the reorganization in July of this year. During the transformation, we found out that doing Agile was not enough with the speed, because all the software development processes were slow. Defining and implementing a Hello World application, that would take five to six months, if you follow all the processes. At some point in time, we went to the management and we said, "These processes, competitors are moving faster. There are more fintechs that are popping up. We need to do continuous integration, continuous delivery in order to deliver software fast."

Sacha: It makes me think. We talk about the velocity, improving velocity, sophisticated metrics, but it's almost like we should define the Hello World metric, which is right now, if you deploy your Hello World from scratch, how much time do you need to get there. That's a pretty interesting metric.

Stefan: Yeah, if we could do it now, I think we could do it in one day.

Sacha: Wow.

Stefan: That's good progress to me.

Sacha: So what you're telling me here as well is you're using the cloud actually.

Stefan: Yeah. We have a hybrid cloud strategy. Most of the applications are hosted on our internal infrastructure, internal clouds. We have made AWS and the Azure platform available, and now applications are moving to clouds when they meet certain conditions.

Sacha: That's pretty amazing, because in the banking industry or financial services, I remember a few years back when I was talking about public cloud. Everybody would politely laugh at me, "You don't know what you're talking about. We're a bank." And here we go. We see ABN AMRO, who went from nothing to continuous delivery, usage of the cloud. It's a pretty amazing shift.

Stefan: Yeah.

Sacha: What does it mean in terms of the people, the culture and the resistance to change that you've observed?

Stefan: What you see is we also saw that when we started the Agile program or the CI/CD program, it's not only about tools and about technical things. It's mostly about changing your way forward. We have a lot of attention for the change management piece. We conducted all kinds of programs to make the people change, and the people were also changing themselves. It was the new facilities they got. We also had people who were used to the old way of working, and we did a reorganization. In the reorganization, the people who did not fit, they were about to leave the bank, so to say. So one of the most important things at the bank to assess the people was the mindset. Are they able to change their way of working, which is required for Agile DevOps.

Sacha: So unlearn and relearn.

Stefan: Yes.

Sacha: That's interesting. So what did you do? Did you have to let some people go?

Stefan: Yes. We let about 20 people go, and some of these people decided themselves to go. So before the reorganization, we allowed the people to indicate themselves whether they would like to go or not.

Sacha: So is it because you think they thought they could not unlearn and relearn, or do you think it's because they thought it was stupid?

Stefan: So, some people said, "Yeah, this Agile is not for me." Some people were already 55, 56 years old and it was time to enjoy their pension. And some people were just not able to change. After working 20 years in the waterfall environment, yes, it might be difficult for these kinds of people to change.

Sacha: Yeah. That's interesting. Can you tell us more about the infrastructure you have put in place, essentially because you have a centralized DevOps platform? How much flexibility do you give to the teams? How much control do you keep centrally? How does that work?

Stefan: We give the teams a lot of mandates. The teams have the mandate, to define what they need to do themselves. We also have an environment in which people can fail, as long as they learn from the failures. If you look to the toolset, we have a defined tooling framework in which they need to stay. So everybody is using Jenkins. Everybody is using XL Deploy. Everybody is using JIRA. But within those three frameworks, they have the freedom to use all kinds of open source tools which they are running. So by doing that, the people can be creative and these kinds of things.

Sacha: One of the common complaints of the centralized DevOps team is that customers, their own customers, teams within the organization keep bringing more needs for different tools, the tool of the day essentially, right, "I need this. I need that." How do you handle that?

Stefan: We are very open to needs they have, so to say. If we do a tool selection, we involve them as well. We do the selection of the tools together with them. Still, as I said, if they need specific tools in the CI process, open source tools, then they have the freedom to install the tools themselves.

Sacha: Okay. So you're very inclusive. That's very nice.

Stefan: [Inaudible].

Sacha: All right. You said something earlier today. You talked about how you want to move to continuous deployment. So we're not talking about continuous delivery here, but continuous deployment. I thought only the Amazons and Facebooks of the world were doing this. And here you go; a bank is going to do continuous deployment. What's that about?

Stefan: We are not yet doing continuous deployment. It's a dream. It's a dot on the horizon. But our firm belief is if you want to achieve something, you need to have ambitious goals, so to say. I said it two years ago in some internal meetings. People were laughing at me saying, "That is never going to happen." But you see it's getting closer and closer. We have now automated pipelines and automated CI/CD pipelines. Facilities are already in place for continuous deployment within the bank. We just now have the process and the faith the businesses give to IT. But if we are going to do this, if we start with applications of course, which have an impact on the production environment. But maybe who knows in the end, the mobile app, where I cannot do continuous deployment there in the end.

Sacha: So what you are telling us is that you think a bank, despite all of the regulatory issues, could totally move towards continuous deployment.

Stefan: Yes. That would be the case. That's definitely a business decision. If you automate things, you can build a lot of controls, a lot of proper gates to ensure the quality. If you do small changes, then the impact on the failure, the chance of a failure in production is less, if there is a failure, is also less.

Sacha: That's very impressive, Stefan. You are driving a revolution here, DevOps, cloud, continuous deployment. You must be a headache to your boss.

Stefan: No. I think –

Sacha: You keep bringing new ideas.

Stefan: I think my boss is very happy with me. If you come to Amsterdam, visit my bosses. They are very happy.

Sacha: Okay. I'm going to visit Amsterdam, Stefan. We need to talk. I need to see that. That's amazing. One thing that really I think is very interesting and maybe it has to do with the Dutch culture, but the way you manage – and we talked a bit about that before, but this cultural position, this discussion around, "Can you unlearn and relearn? Can you be fit for the job," and how you're being inclusive when you make tool selection and not try to do an us versus them situation. I'm sure you've built a framework on how you interact with your teams. Can you talk more about that cultural aspect?

Stefan: Yeah. So what we have done, we have reduced management areas. [Excessive background noise] We have the _____ _____. We have decreased the amount of people in HQ software delivery, so to say. So there's much more focus on building software, testing software before the software goes to production. We give people the freedom to define what they think they should be defining. We organize meetups, we organize hackathons, we organize all kinds of events which make work interesting for the development community. Because of these kinds of things, people are getting motivated to ______. It's also very important that we want to give the development community the best tools, the best work environment they need to do their work. So the better the tools they have, the more innovative work they can do, the more motivated they will be. So improving all of the distraction, that's good for us.

Sacha: Yeah. It's very clear that you're empowering your teams to do things. Tell us more about those hackathons. I can imagine a hackathon in a gaming company or some other company. But in a bank, how does that work?

Stefan: We have internal hackathons, data warehouse hackathons. But a few months ago there was a big hackathon organized also with teams outside of the bank. We went beyond banking – 40 teams, 40 or 50 teams programming for 48 or 72 hours. The bank management was there. And the team who made the best solutions got awarded with some interesting prize.

Sacha: And you invited external participants.

Stefan: Yeah. That was not me. That was senior management. So even the senior management that was in the bank, they're very committed to these kinds of initiatives.

Sacha: That's amazing. What was the goal behind inviting people? Was it recruitment?

Stefan: No. The goal in inviting all the people is to have the best solutions for the bank, to show external growth, that the bank is an interesting company to work for, to see where teams are able to deliver interesting solutions in a short timeframe, and also to learn, to learn from all kinds of things which are available to the teams.

Sacha: That's really cool. But those people were not paid. So why would people outside of the company say, "I'm going to spend four days."

Stefan: Because they like it.

Sacha: They just like it, passion.

Stefan: I also went there. Passion, be curious.

Sacha: That’s cool. Did you win?

Stefan: No, no. I did not program myself. I used to program in the past, but not anymore!

Sacha: That's really cool. It seems like it's perfect. So you don't have any challenges; it's all good?

Stefan: No. We have many challenges. We need to deal with compliance, regulations. We have continuous discussions with security departments, with the audit departments. At the audit departments, they all see now the benefits of CI/CD. But, at the same time, for certain things _____ _____. If I tell people to do continuous deployment _____ _____. What you see now in the continuous delivery process, heavily automated deployments now. But you still have lots of waterfall procedures in that area, so in change management process. We are discussing now with people of those departments how we can simplify the processes. How can we shift left? Because our view is if we guarantee quality in the beginning of the CI process, then you don't need this change management processes in there at all. They can be fully automated. That's a discussion of course which takes longer. You need to be patient. It could be long to implement DevOps, but at least more and more people are moving in the right direction.

Sacha: Yeah. That's interesting because you are talking about the type of audience, people in charge of compliance. Those groups of people tend to be more conservative, by design. Their job is to protect the company from a legal standpoint. So it must be a hard change for those people, because essentially you're pulling them outside of their comfort zone and asking them not to give you the safe answer, but the right answer for you to achieve your velocity.

Stefan: I can give an example. There was an audit requirement that a secure coding check needs to be done. But in the old way of working, the secure coding was checked after the deployment. So we had discussions. I said, "This rule is not any more applicable in the DevOps environment." We had discussions. Then we agreed that this kind of rule should be changed by philosophy that people, teams should have automated secure coding checks in a CI pipeline. So by doing this and by showing the results and by showing that we have secure coding, you see that these people get more confident.

Sacha: But I guess what this means is that for those people to have the correct motivation to do that, there must be some kind of top-down, not pressure, but objective to really move towards agility.

Stefan: Our senior management is fairly committed towards CI/CD and towards DevOps. Our senior management is continuously saying to the teams, "You should automate. You should change."

Sacha: When you say your senior management, how high do you think it goes? Do you think the CEO is in charge, the board?

Stefan: The CEO and the board.

Sacha: What do you think made them change? Do you think it's because now they are seeing competition in the market? Anything else?

Stefan: What made them change? Well, the senior management also saw that things were slow in the past. Senior management also did listen to us. We had lots of conversation with them. We would tell them about CI/CD. We'd tell them about DevOps. They got references and they'd see these kinds of things. They'd be on banking. They talked with engineers. So those kinds of things make the senior management aware that this is the world that we need to go.

Sacha: That's pretty amazing. So you've been in DevOps forever, for a long time.

Stefan: Yeah.

Sacha: You must have a pretty strong opinion on things. Where do you think DevOps is going? Where do you think it should go? What do you think are maybe the next innovations that could be useful with that space? Any opinions there?

Stefan: That's difficult to say. That's also nice about DevOps, because we only have to look on the horizon, but how we go there, we don't know. So that's also why we say we don't make long-term plans. Every quarter, we define what we're going to do next quarter. Not because you can't make long-term plans, but history shows that there's a big chance that they need to change. Simply, you don't know what's going to happen. You don't know what things you find on your way. It's a big learning journey.

Sacha: Yeah. It's amazing. Any closing thoughts?

Stefan: No. I am enjoying Jenkins World. I enjoy the atmosphere here. I've had good discussions with my partners, with my tooling suppliers, interesting sessions ongoing. I'm just enjoying the event.

Sacha: That's great. Thanks a lot, Stefan.

Stefan: You're welcome. Thank you.

Sacha: Thank you. This is DevOps Radio direct from Jenkins World 2017. Thank you.