Harald Gottlicher of Bosch: How to Undergo the DevOps Journey Without Making Enemies

Brian Dawson: Hello. This is Brian Dawson. I am here with Harald, and Harald, I'm gonna actually let you introduce yourself and pronounce your last name.

Harald Gottlicher: Oh, my last name. It's difficult. It's pronounced Gottlicher. I think it's hard for you.

Brian: Gottlicher, yes. I'm gonna mess it up, so forgive me. But anyway, welcome. Thank you for joining.

Harald: Thank you.

Read More >>

Brian: Harald's here with us at DevOps World | Jenkins World, in Nice. And Harald, you have pretty extensive experience with DevOps and DevOps transformations, right?

Harald: Yeah, I think so.

Brian: Okay. What is it – you've spent – six years ago you started implementing CI/CD in DevOps?

Harald: No, it's already eight years, I think.

Brian: Okay.

Harald: And the interesting thing is we don't do what you classically understand under the term DevOps. Because we have a wide variety of applications, like we have embedded things, firmware, we have desktop software, we have web applications. So a great variety of things and, yeah, I think still it's very important to roll out DevOps ideas and principles to all of these things.

Brian: Okay, okay. Now before we get into it, I want to hear...in a minute I'm gonna ask you a little bit more about specifically the process you went through and what problems you solved at Bosch, but before we do, can you tell me a bit about what your role is at Bosch today?

Harald: Oh, sure.

Brian: And what you focus on, what you do.

Harald: My role is software architect for all the continuous delivery stuff, so I am providing the technical solution, discussing the requirements with all of our projects. We call them our customers.

Brian: Okay.

Harald: And so I'm checking how their new requirements fit into our continuous delivery landscape.

Brian: Okay. Now I'm curious – now is that – it sounds like that's role that could either be in an enterprise architecture team, or could possibly be part of a DevOps team. Where does, where in the organization does your role as a software architect for your DevOps pipelines live?

Harald: Well, we have kind of build and automation team.

Brian: Okay.

Harald: Which provides all this as a service to all of the different projects, so we are something in between Dev and Ops, so you could say we are DevOps.

Brian: Right. Okay, right. You guys are the...you guys are bridge builders.

Harald: Yeah, kind of.

Brian: So I was told about a big win that you got in terms of DevOps I think roughly three years ago - you may correct me. I understood that there were – and I'm gonna get this wrong, so correct me – that there were component updates that when you guys had to update in individual potentially tens if not dozens or hundreds of components, your legacy process required that you updated all of those simultaneously?

Harald: Right, right.

Brian: And then you guys did something to improve that, right? Can you tell me more?

Harald: Yeah. Yeah, we could update parts of them, but we always had to handle everything in the build systems, which is roughly one million files and 30 gigabytes. And so what we did is we split up everything in smaller artifacts, and I think this is a very good practice that a lot of people should adopt, like you are doing in the web application environment.

Brian: The microservice.

Harald: You're doing – right, exactly, the microservices.

Brian: Right.

Harald: But people in monolithic applications and in embedded applications, they still don't think like the microservice thing.

Brian: Right.

Harald: But you can also split these applications in smaller parts in many cases and then you can only rebuild the small parts that really have changed.

Brian: Right.

Harald: So we split up our monolithic stuff in ten-thousandths of single artifacts, ten-thousandths of built jumps, which rebuild in a minute or so.

Brian: Right. Okay.

Harald: And the main – yeah, the main application assembly collects only the changed ones to build an update. Or if you want the full build you can just collect the readymade artifacts. It's also a lot faster than building everything.

Brian: Rebuilding all of the dependencies and all of the –Now, can I ask on that, was there a measurable benefit that came? What was the benefit of approaching that decomposition? You could move faster, it sounds like.

Harald: We can move faster, we can update faster.

Brian: Okay.

Harald: We are still in the process of improving all the processes to update up to our customers. Because there's a lot of quality gates and manual testing, so there's still a lot to do.

Brian: Right. Okay.

Harald: The benefit is a lot more internal currently.

Brian: Okay.

Harald: So we are able to rebuild the full product. Instead of days we can rebuild it in a few hours and we can build updates within a few minutes.

Brian: Okay.

Harald: So the goal is also to roll out these updates to customers immediately.

Brian: As fast – and so right now the internal benefit I would assume is you're getting increased productivity.

Harald: Right, right.

Brian: You could probably even internally add more value, innovate more because –

Harald: Yes, that's the development cycles.

Brian: Okay.

Harald: Faster feedback on bugs, such things.

Brian: Right. Okay. And that's setting the state for clear and present external benefit...

Harald: Right.

Brian:  ...to the customer and ultimately the business, right?

Harald: Right. We have shown what is possible.

Brian: Right.

Harald: The organization understood it and now we are about to changing also the organization and the processes and the testing, quality processes, to bring updates very fast or immediately to the customers also.

Brian: Okay.

Harald: But I think the precondition for this was that people understood what is possible. They were thinking in monolithic, long-term things and in large releases. And with continuous delivery we could show them what is possible and now we are rethinking this up to the customers.

Brian: Okay, and that – so it sounds – so I talk a lot about this. I think an interpretation of this is you gotta win. That win kind of allows your constituents to see the possible and then gets you – buys you more capital to improve even further. Is that fair to say?

Harald:  Yeah, right.

Brian: Okay.

Harald: That's how you could say it.

Brian: Okay, okay. So I'm curious, so during your session you talked about your experience transitioning from outdated tools. We kind of touched upon culture a little bit, right, but transitioning from outdated tools and rigid processes to a continuous delivery landscape. How important was updating your tooling? How important was tooling for Bosch to move from manual processes or less efficient processes to optimized automated processes?

Harald: I think tools are crucial in many cases. In our case, we had a very, very instable, outdated version control system with a lot of inconsistencies. And yeah, it was not even possible to do more than one build a day, one full build a day, and then –

Brian: Limited by the...

Harald: ...the server was dead.

Brian: Oh, wow, okay.

Harald: And it had different ways how to achieve the same thing, so different people used it differently. It was really horrible.

Brian: Okay. So you have – so your branching emerging models were all over the place, I assume?

Harald: Yeah. Right.

Brian: Okay. And so you identified that as one of the first tools you needed to update, your version control system?

Harald: Exactly. That was the core of our migration process to get rid of that tool. But all the projects, a lot of different departments and projects were on that tool, so it was very hard to get rid of that tool.

Brian: Yeah, and people end up – you know, people end up married to their version control systems even when it's a bad marriage sometimes, right?

Harald: Right, right.

Brian: It's hard to...

Harald: And that manufacturer even called it an ALM tool because they put some wacky functions on top of it and people adopted their release processes, adapted their release processes to the tool...

Brian: To fit their model.

Harald: So it was very difficult to get them away, and many people even didn't want to move away from the tool.

Brian: This sounds like something that may rhythm with cup. A unified – a tool that prescribed a unified process for development? What is the SCM that you guys were coming from? If you don't mind sharing, if you don't –

Harald: Yeah, I can share.

Brian: We can edit that.

Harald: It was called MKS.

Brian: Okay, yes.

Harald: It's....

Brian: Yes, okay.

Harald: And it was – in my opinion it was not really useable as an SCM.

Brian: Okay. And where did you guys migrate to?

Harald: At that point of time, I said we finished that migration four years ago but we started with the idea seven, eight years ago, and this is why we migrated that stuff to Subversion, because at that point of time it was relatively new.

Brian: Right. Yeah.

Harald: And for a traditional company like Bosch you had no services like Bitbucket and GitHub. They were slowly upcoming, and I think a large company didn't yet trust in Git. So that's why we had to move to Subversion first.

Brian: Okay, yeah.

Harald: But now we are moving by and by all the projects that are really development projects to Git, but we also have projects that are just like data storage or –yeah, like specification and such things; they do not really branch and every –

Brian: They don't need the distributed version control, right.

Harald: So for them Subversion is still good.

Brian: Right.

Harald: It works well for them.

Brian: There's an interesting moral there, and that is don't pick a tool just for the sake of picking a tool; pick a tool that is best fit for what your need is, right?

Harald: Right. And what fits your organization. I would have loved to take Git, but at that point of time I didn't dare to propose it because really the entire thing was so weird. Everybody said you can never migrate this, it's not possible, you will never do it. And I said, "Yes, I will do it," but I didn't want to add Git on top of it to make it even more difficult –

Brian: Right, there's already enough change going on, yeah.

Harald: – for management to decide for my project. So I said, okay, let's take a step back and let's take Subversion for now. We can – it's a clean base. You can easily migrate from Subversion later.

Brian: So that's an interesting – that takes me to the next point. So we talked about tools, right, and how important removing outdated tools are to your transformation. But now you've talked about sort of recognition of the state of the organization or the culture of the organization, and I would phrase what you just said as, “Let's not overload them with change. Let's figure out how much change they can accept.” I know that you talk about, you know, kind of in a cultural aspect how do inspire employees to embark on the DevOps journey or to get involved in the DevOps journey and avoid making enemies, right?

Harald: Right.

Brian: Can you share with organizations maybe how they can sort of successfully accommodate or achieve inspiration of employees?

Harald: Yeah. I cannot repeat my entire talk but… [Laughs]

Brian: Okay. Darn it. I didn't get to make it.

Harald: Yeah.

Brian: I was hoping I'd get a free version of it.

Harald: [Laughs] Yeah, I think the slides will be online.

Brian: Okay.

Harald: Still, what I have to say, I did a lot of mistakes, sure, because if you do such large migration for the first time, yeah, you will have a lot of new experiences and challenges. So what I would recommend is to make a lot of friends, to convince people in an early stage and to show them prototypes of how they could work in future and to listen a lot to them.

Brian: Right. Okay.

Harald: Yeah, if they say they have doubts how they can work or how their process fits into the new tool landscape, you should listen to them and you should adapt your prototypes to show them how they could maybe work. And maybe we didn't do this early enough.

Brian: Okay.

Harald: So this is very important, the social aspect of convincing all the people.

Brian: Absolutely. Okay, thank you. Great answer. And I, in my experience, I think that is extremely important too, and oftentimes as engineers we can overlook the importance of the social aspect of these things.

Harald: Yeah, we are enthusiastic about new tools.

Brian: Right. [Laughs]

Harald: And we think, well, this is the greatest and everyone has to use it because there's nothing better, but reality looks different.

Brian: Right.

Harald: There are good reasons why some people have other working process than we expect.

Brian: Right.

Harald: And they might have good reasons and you need to understand it before you criticize them, before you give them new tools. You need to understand how they are working and why and why it might make sense.

Brian: Right. Those – valuable, valuable. Wise words, valuable insights. So in terms of Bosch and in terms of what you are focusing on, what's your next step in terms of maturing and improving the application of DevOps? What's next for you?

Harald: So a lot of steps are next. What we are currently working on is to get everything in Docker...

Brian: Oh, interesting. Okay.

Harald: ...to be independent from any build agents, and also we have...

Brian: And this is Jenkins in Docker?

Harald: Yeah.

Brian: Dockerizing your CI/CD pipeline?

Harald: Yeah.

Brian: Okay.

Harald: Yeah, we have a lot of legacy tools and special tools that might run only under an older Linux version, whatever, and you can handle this really great if you put it in Docker.

Brian: Right. Okay.

Harald: On the other hand, we have a Franken-Jenkins, a Franken master.

Brian: Okay. [Laughs]

Harald:  Jenkin-stein.

Brian: Right.

Harald: We were talking about it in this conference. We also have one; we need to split that up in smaller masters. Yeah, so there's still a lot to do. The other area where there's a lot to do is test automation. That's a bigger challenge because in our case you have a lot of hardware-related tests, hardware and the loop tests with a lot of different hardware.

Brian: Right.

Harald: So there's this hardware to get it automated, but also organizational work. Our testing department is completely independent and separated, so we need to integrate this in a DevOps style. That's very important, I think.

Brian: And I assume – and I have to be careful – I love this stuff, I can go in, but I want to on testing, but I assume that based on your space that you're in there's also a retooling, a retraining that QA needs to do to get integrated, right? 'Cause your quality assurance is probably more focused around traditional full regression plan-based testing, right, or manual testing.

Harald: Yeah. We have a lot of manual testing because what we do is our main product is car diagnostics for workshops.

Brian: Okay.

Harald: So you necessarily need to connect stuff to a car and test manually

Brian: Right, manually.

Harald: We also have things like simulations where we can test automatically.

Brian: Okay.

Harald: So we have different areas of testing. I think we will never run –

Brian: Fully automated.

Harald: – totally without manual testing, but there's a lot that we could automate, and we should work on this, but also on the social and organizational aspect.

Brian: Okay.

Harald: Get the testing team into the DevOps process.

Brian: Right. Awesome. Thank you. So unfortunately, we're gonna come up against time, but what I do want to know is do you have any final thoughts to share with our listeners?

Harald: Yeah, final thoughts. I would say one thought is don't underestimate the social aspects. Take all the people with you on your journey of transformation, to understand them, to get to know them, to sit together at the campfire like I added in my talk.

Brian: Right. [Laughs] Yeah.

Harald: Yeah. And the other final thought is, yeah, you should really think about how you can adopt the DevOps and the microservice principles even if you don't have a web application.

Brian: Right.

Harald: Even if you have a monolithic application or embedded stuff, think about how to split it up in smaller pieces, how to rebuild and test only smaller pieces of your software. I think this is things that you can learn and adopt from DevOps principles even if they don't match for your case.

Brian: For your architecture, yeah.

Harald: There's still a lot to learn and to adopt.

Brian: Excellent. Agree fully, Harald. Thank you for those wise words and thank you for your time. Hopefully you'll get to enjoy the rest of DevOps World and Jenkins World. And for our listeners, I'd like them to be able to find your slides which will be posted. What was the name of your talk again?

Harald: The name was "Enemy Mine – Transition to Continuous Delivery without Enemies."

Brian: Alright. "Enemy Mine – Transition to Continuous Delivery Without Enemies." Check out the slides and learn more about what Harald shared with us today. And Harald, thank you. Appreciate it.

Harald: Thank you. It was nice talking.

Announcer: Like what you’ve heard today? Don’t miss out on our next episode. Subscribe to DevOps Radio on iTunes or visit our website at CloudBees.com. For more updates on DevOps Radio and industry buzz follow CloudBees on Twitter, Facebook, and LinkedIn.