Sam Oyen - Serves Poppin' Fresh DevOps

Andre Pino: You're listening to DevOps Radio, a podcast series that dives into what it takes to successfully develop, deliver and deploy software in today's ever-changing business environment. This show is sponsored by CloudBees, the enterprise Jenkins company and continuous delivery leader. Welcome to DevOps Radio. I'm Andre Pino, and today we're coming to you from Boston, joined by Sam Oyen, DevOps engineer at General Mills. Sam, thanks for joining us today.

Sam Oyen: Thanks for having me.

Andre: Sam, I know that you attended Jenkins World. I was wondering if you'd tell us a little bit about your experience there.

Sam: Yeah, I would love to. Jenkins World is a really interesting conference, at least compared to the other conferences I've gone to, because of the fact that it's part of the open source community. So probably my favorite part of Jenkins World this past year in 2016 was the fact that they had Expo booths, where you could go up and literally talk to all of the authors from the plugins, and pick their brains, tell them about stories that are going on in your company, and problems that you guys are having, and get help from them. And they literally had whiteboards plastered all around the room, so you could both do problems. And then also just the overall collaborative environment of talking to other companies, as well as those experts, that are going through a similar process as you of converting to CI and CD. So just a overall very collaborative community-driven experience. It's always a good time.

Andre: That's outstanding. Yeah, actually that was something new that we added this year. We always had an Ask the Experts booth, but we really expanded it this year. And we felt that providing those whiteboard areas to facilitate the discussion and collaboration was really important. So I'm really glad to get that feedback from you.

Sam: Yeah, I definitely loved it, and I try to give feedback every time I can of how much that was helpful for me and my other colleagues that attended. So I was a fan.

Andre: Awesome. So Sam, tell us a little bit about your background. How'd you first get into software development, and the role you have today.

Sam: I think my path was a little bit different. I wasn't actually very interested in computers or tinkering around with technology when I was growing up, in middle school or high school. I kinda fell into it. Like most freshmen in college, I didn't know what I wanted to do. I'd always been interested in math and science, so I was like, "You know, maybe some kind of engineer." But my best friend took an Intro to Programming course, and so I took it with her, 'cause I had no idea what I wanted to do. And I kinda liked the problem-solving aspect of it, and so I just stuck with the computer science route, and I majored in that at the University of Minnesota. Then I was lucky enough to get an internship at General Mills, after talking with them at the career fair. And so it's really funny, because the biggest reason that I liked computer science when I first got into it was the fact that I thought it was very logical and very black and white, you know? Like not as much of the gray area. Like this solves the problem, this doesn't; and that's why I liked it. Then when I got into industry, you find out how many areas of gray there is, and how complicated these solutions are. There's not one right answer. There's a lot of right answers, and there's pros and cons that you have to work through on which one to choose. But that's kind of why I love it now. So it's really interesting. I think the reason I went into it, if it actually would've been that black and white, I think my career would be a lot more boring than it is now. But yeah, so now I've kind of fallen in love with the messiness and the chaos that is programming and trying to wrangle all these developers into one software delivery platform.

Andre: So how long have you been with General Mills?

Sam: Just over two years, full-time. Before that, I did do an internship with them, and then they also worked with me so that I could work part-time while I was in my senior year of college, because I went to college in the Twin Cities. General Mills is in the Twin Cities. So it just worked out for me to be able to come into the office and start my journey here a little bit early.

Andre: That's awesome. So in your career at General Mills, have you always been in the DevOps and the automation space?

Sam: Not always. So my internship was kind of a typical you get a business case. I worked with the marketing department on a problem they were having, built them a custom website, and some databases behind that to help them solve their problem. But when I was working part-time in my senior year of college, I actually did get placed on the software delivery team. Which our big focus is the DevOps space and automating a bunch of that. So I worked on that team part-time. When I started full-time at General Mills, I did a little stint in smaller brand sites and international sites. So working on some of those; Nature Valley Australia, and working with the marketing departments in those areas to build some external-facing websites for them. Pretty soon after, I was probably on that team for around eight months, and an opening came up on the team that I had already worked part-time on, and I jumped all over it. So a little bit of other kind of dotnet website apps, and experience in that. But the automation and the software delivery platform is really what I loved, and so when I made that known, I kinda switched back into it from the external-facing websites.

Andre: So what is the scope of your role there? Is it corporate-wide, this DevOps platform that you're working with?

Sam: Yep. So it supports all of our dotnet and iOS and Android apps, building and deploying them for all of General Mills. So our application development environment has about 130 developers worldwide. We have a headquarters in Mumbai, and so probably about a fourth of our developers are over there; so some of them from my team are over there as well. And we support thousands of apps for our company, varying from external-facing websites, Pillsbury.com, internal business apps that the sales force uses on iPads when they go into stores, to internal websites that are used for tracking data or just making any of our business processes more efficient and more transparent. So we support thousands of apps. So some other areas within General Mills where developers are doing different type of development. But we support all of the dotnet development, and that is a majority of what we do here.

Andre: So how would you describe the current state of General Mills' adoption of DevOps practices and principles?

Sam: I think – you know, it's really funny. We have a pretty solid history in putting automation and efficiency first. I think that just drives from the business having very high expectations of what they need done, and developers rising to that challenge. So we have had a custom build-and-deploy system that's been around for almost a decade, that is robust, that's secure. It audits. It automates all of the build-and-deploy with one click of a button. And so starting from that point, it took us a little bit longer to start looking outside of what we had in-house to see what could actually make differential gains, as compared to the system that we had that was pretty dang good. So I think we kind of joined late in the game, just because our starting point was pretty high, and we had to find something that would be just as good as the system we had now, while helping us gain that continuous piece of it, that transparency into security metrics, unit tests and all of that. So I think we're making all the right moves, and we've made the biggest step of saying, "Our old system was great. We can do better." And there's a lot of other technologies out there that can help us with that, and help us iterate faster, and customize our build processes and deployment processes even faster than we thought possible. So we've made that move of looking outward to open source, which was a huge move for us. We definitely look at open source technology all the time now. We contribute back to it. And I would say on our road to DevOps, we're in that continuous integration place. So we have that automated build-and-deploys. It's the continuous piece of it, and the building up the metrics against those builds – that's the place we're in right now. And so we have a really good foundation using Jenkins right now. The pieces that we're still working on is more of those flexible deployment paths. How do we have a deployment path that is unique to your project? Maybe you don't need a Dev/QA and a Prod. Maybe you need five environments. Maybe you just need a Prod. So customizing those deployment paths, so that it's not just automated, but it's exactly what you need. And then also allowing developers to have more transparency into their apps when they're in production, and when they're live, so that they have transparency into that ops world, and they have the power to problem-solve without having to draw in that team.

Andre: So you're really seeing your platform as a platform that really serves developers, and looking at what they're unique project and specific needs are and trying to tailor the platform to meet those specific needs.

Sam: Exactly. And that is our hardest struggle, is opening it up to developers enough so that it's useful, it's customizable. They know what they need for their projects better than we do, and they should be enabled to do that. But it's hard to balance that with making sure the system is secure. We're still at the stage where our servers are pets, not cattle, in most areas. So we have one build server that a lot of our apps go through. Like we need to make sure that those build processes that are happening on there, we want developers to have control over that, and have builds that make sense for them. But we also need some sort of oversight. So how do you balance those two conflicting interests, is the...

Andre: Yeah, this is a topic that I've heard over and over again, as organizations, typically larger organizations like General Mills, try to create a centralized, shared service, but also meet the needs of the individual projects and teams. So it sounds like that's exactly where you are, in trying to centralize what makes sense to centralize and share best practices with, but to give the projects the opportunity to tailor and customize it to their specific needs.

Sam: Yeah. And I will say one thing that has worked extremely well with opening up our Jenkins platform to serve even more teams than we did with our old platform. So we're having some of our Hadoop deployments, elastic deployments, happening on Jenkins, which our old system didn't support. One of the key things that we did that made that possible was we have a Dev environment for Jenkins, and we basically give developers the keys to that. And we say, "If you need a type of job that's different than the kind of template we have in place, you need your own custom solution, let's enable them to build that. But then right now we still have a process where, okay, before we move that job onto Prod, and it's in that production environment, we're advising them on, "No, you should probably encrypt this piece of it," or "This is a better way to do it," because we do have more of that Jenkins environment knowledge to help them refine their jobs, and make sure we approve of them. But they get to do most of the customization, and to be honest, most of the work, which is nice, as a build automation developer, to say, "You know, I'll let you do this piece of it." And then I can work on looking into newer technologies, or other problems that we see. Maybe we don't have right now, but if we keep down this path, this is gonna be something we run into. And proactively tackling those problems, instead of customizing build jobs for developers that could probably do it better themselves.

Andre: Right. So I understand that you're also using the CloudBees Jenkins Platform. What does that bring to the table in helping you solve some of those challenges?

Sam: I would say a few things. So one of the greatest pieces is just that partnership that you have with CloudBees, so you have a support avenue for if you just have a question, or if something's not working correctly, you have somewhere to go with experts on the line. So that's really helpful. And just kinda staying in tune with them as well. Even more than just for support, but also having your ear to the ground on "Well, what are you guys working on next?" And then sharing just overall ideas that you have, or struggles that you have as a company, so your voice is heard. The other thing is there are some plugins that make organization of as many jobs as we have a little bit easier. And security – so those were two of the biggest pieces why we decided to go to CloudBees. And then also, they have this idea of an operations center, which kinda controls your masters, which is part of the reason that we are able to have a Dev master and a Prod master, and have those managed by the same OC, so that security and all of that is the same, and we know it's set up in the same way. And then you can also see metrics across masters, and – yeah, there's a decent amount of advantages to going with the CloudBees platform, and actually a lot that I don't think we've even tapped into yet. So there's some Kibana analytics that you can really get a lot of insight into what's going on on your masters, and that's something we haven't even really tapped into as much as we probably should be.

Andre: So can you give us a sense for the kind of scale of your platform? How many jobs are you running, and how many users you have on it?

Sam: So we have it open to all of our 130 developers. As far as the number of apps, we're about 300 apps on Jenkins, but that turns out to be around like 400, 500 jobs, because at various times, you're building feature branches as well as the masters and such. And we were able to do that with actually another feature that CloudBees provides, which is the job templates. So for I would say about like 95 percent of our jobs, we were able to fit the needs of that Jenkins job into one template that they could parameterize. We have some automation around – we have a build-and-deploy website that keeps track of all of our apps. If you want to create a Jenkins job, you go to that build-and-deploy website. You click “create Jenkins job.” It builds it off of that template, and it's kind of a one-click process to getting a Jenkins job that builds, runs unit tests, JS tests, code metrics and deploys to a Dev server.

Andre: It's kind of like Jenkins job as a service, huh?

Sam: Yeah. Yeah. Exactly. So that was one of the key features that made that scalability possible, because if we had to onboard every developer on how to set up a Jenkins job, and them having to go through the UI, choosing which plugins to use, that would've been a much longer process. And painful, all the way up. And so I think us starting there, and moving back towards okay, we have this baseline. There are still developers that want to customize further, and that's great, and that's at the point where it is how do I allow them to customize? Because templates, you give a developer a template. They can't do anything, basically. They can hand in some parameters, but they're pretty much stuck with what you gave them. So we're looking into some other plugins. Job DSL is the one that we're looking at right now, to see how do we give developers more of that control, and still have that inheritance that you get from defining a template once, and then any time you update it, all of the Jenkins jobs gets your update. Say you switch your track marks over. They don't have to go in and change that in each job. You change it once in the template, and that's inherited. So there's the idea of inheritance, too, in Job DSL.

Andre: That helps a lot. So it seems to me that you're in a kind of a unique position as the DevOps engineer responsible for this platform, because on one side, you're serving developers, and on the other side, you're also serving the operations folks, who are responsible for the production systems and other systems. You sorta sit in the middle with an automation platform that spans both of those worlds. How do you manage that, and how do you create that culture of collaboration between those two teams?

Sam: Yeah. So I think that culture piece is probably the hardest problem that most companies have to overcome. Because it's one thing to introduce a tool that can do something. It's another thing to encourage the culture that actually incites people to want to use the tool, and use it in a way that they're getting benefit out of it. I'm lucky that General Mills has always been a very collaborative culture. I probably spend half my day in conversations with other developers on how to do things. It's very rare that I'm working on a problem and I don't get an opinion from someone else on my team, or on another team, about how to best tackle an issue. The other thing that is probably a huge advantage for us is that the infrastructure team that manages our websites is actually on our team now. So we went ahead and merged the web hosting team and our software delivery team. So we actually sit side by side. We have Scrums together. We do task breakdown together. And there's a lot of knowledge share going on in that area. There's obviously other infrastructure teams that we don't sit next to, and we don't share Scrum with. But when their pieces of infrastructure start to collide with our pieces of automation, we do bring them on, temporarily. So what we'll do is if they want an integration point with part of our automation into their systems, we'll get a feature in our backlog for that. We'll consult with them on the requirements. And then we'll bring them into our team for like a week or two at least to start off, and get their insight and their knowledge on the topic areas that they know best. So there's a lot of collaboration going on. It's a really exciting time here, to be honest.

Andre: That's great. What kind of apps are the most critical for you?

Sam: I would say the one space that application developers do developing that we do not support build-and-deploying is our SAP apps. So most of our financial information, HR, the most critical pieces of our core business, are in SAP. And we don't support that, which makes me happy, because we get to play around and kinda be a little more forward-thinking, and push the limits a little bit more, than if we lived in that world. As far as the apps that we do support, I would say our BettyCrocker.com, our Pillsbury.com, those get a ridiculous amount of traffic. It's crazy. There's a whole season we call key baking season that is – I mean the amount of resources that go into those websites, and the amount of people. And I can't even remember the numbers anymore, but it's somewhere on the lines of like the amount of people that go to Wikipedia or something. Like it is crazy, the amount of traffic that those sites get. So the fact that –

Andre: That's pretty impressive.

Sam: Yeah. And that's actually – it's kind of interesting, the origin story of Jenkins here. It had a lot to do with those sites, because we have an engineer that was working on those sites, and said, "You know what, these are really critical. We need more unit test metrics. We need more JavaScript metrics. We need more integration with code analysis on every single build that we do." And the way that Jenkins started out here was actually an open source instance of Jenkins for those core sites. So they had a big role to play in us actually adopting that eventually. So that was kind of our lemonade stand of Jenkins. It was homegrown, authored by one engineer that was really passionate about it. And then because all of our teams here do talk and know what the other teams are doing, we eventually said, "You know, this is something that would be beneficial for the wider application development to adopt." And that's kind of how our start in Jenkins came about.

Andre: Really interesting. Sam, in my role here at CloudBees, I get the opportunity to travel around the world going to a lot of DevOps and developer conferences. And I gotta tell you, I don't see a lot of women in this area. What's it been like for you as a woman entering into the IT domain, and especially – especially – this DevOps area?

Sam: That's an interesting question, and it's one I get asked a lot. And I think it's really hard to answer, because in my experience at least, I haven't felt like I had a different road to go down, or a different job to do, or a different experience in the workplace because I was a woman. I will say in school, there was obviously – I would be in a lecture hall of 500 people. And there would be me and my best friend, Olivia, who ventured down this comp sci road together, as probably maybe two of five women in that class. And so it's something you noticed, but I think I was just so focused on my schoolwork, I didn't really care. And I do think it helped that I had another woman that I had known since middle school going through that major together, and kind of being allies there. I will say one of my biggest struggles with going to college and working in this industry, especially programming, was – and this might not be a gender-specific attribute, but I think I do notice it a lot in women – is the fact that I was a very big perfectionist. So I didn't actually get any great debugging skills until I was forced to go to industry and deal with those bigger apps that maybe someone else wrote, or I was just adding to. Because in school, I would be so meticulous about writing my program, and take so long to do it, and make sure it was correct, that by the time I ran it, I really didn't need to debug it. And I missed out on learning one of the most important skills of programming – yeah. And it's not something I realized until I got into industry, 'cause I made it through school. I did fine on all my assignments. But that was the biggest thing I learned, was that it was a really hard – it was really hard for me as a core part of my personality, that I was a perfectionist. And to kind of push that down, and just allow yourself to have more fun. And enjoy the problem-solving process, and enjoy the bugs that you run into, and the times that your program breaks, as a chance to investigate why that happened, instead of taking that as failure and being scared of that. So I think that was one of the biggest things I had to overcome. Which may or may not be more of a female attribute, but no. General Mills does a really good job at hiring a ridiculous amount of women for the industry that we're in. And still on my team, I work with mostly guys, but I mean I have a great team.

Andre: That's great to hear. So I know that you're also involved in a program at General Mills called Discovering IT. Can you tell us what that's all about?

Sam: Oh yeah. So what we do is we go out to high schools and we do presentations, either to career investigation classes or sometimes high schools or middle schools have actual programming classes, which is great. So we'll go out to these various classes – sometimes it's math and science courses – and just give them a presentation on what is IT? How would you know if you're interested in something like that? Do you like problem-solving? Do you like math and science? What does an actual developer, or any of the other various roles within IT, what do they do on a day-to-day basis? What are they thinking about? And try to give them a good picture of what this industry looks like, and then give them some starting points of well, if this sounds interesting to you, where can you start? There's a ton of online free sources – Code.org – especially for that middle school crowd. It's ridiculous the amount of kids that we go to, and they've already been on Code.org, and have been doing Scratch, and are just masters at it – better than me. And so I think there's a lot of groups out there pushing it, and a lot of teachers that are making that a priority, to have programming classes. But I think us going out there and further introducing it to them as a really real possibility of an industry that is not impossible. It is not – you know, one of the biggest things we look to do is not intimidate them. Because this is an industry that can be very intimidating. Like, "Oh, do I have to be a whiz at math, and have gone through all the levels of calc by the end of high school?" So we try to make it more approachable, and share those resources with them, and share why we like doing our jobs. So the people that would be successful in IT, and this is an industry they would enjoy, it's something they consider even before they go to college. And maybe take steps towards getting more involved, even at the middle school and high school level.

Andre: Yeah. I think we all know and recognize that the future is software, and that the earlier we can introduce development practices to the youth the better. And so in conclusion, Sam, today, what advice would you give for a young woman that's thinking about a career in software, or in IT, that might look around and see, "Wow, this is all men," be intimidated by that. What advice would you give them?

Sam: I would say a few things. Fail early and often. Make yourself write something that doesn't work, and figure out why it doesn't work. And don't view that as a failure. And also, just ask questions. I think a lot of times, asking the question is hard, because you might think it's a stupid question, or you're not sure if you're even wording the question right. You don't even know if you have that much base knowledge to even form a good question on the topic area. So what I do a lot is – it was really hard for me to start learning how to say like, "I don't know." Like someone would ask me, "Well, what's this?" And say, "I don't know, but I'll figure it out. And if someone asked me a question, "Sam, how would you do this?" you can say, "I don't know. How would you do it?" Not only is that taking the intimidation factor off of you, and putting you in a position where you're not thinking about how scary that question is, or if you don't know how to answer it. You're also getting knowledge from those people around you. And the really interesting part about that is that you figure out that not everyone else is as smart as you think they are. Because we all don't know it all. And so the more and more you ask other people questions, the more and more you figure out we're all humans that don't know everything, but we can kinda talk and figure it out together.

Andre: Sam, I think that's great advice. Thank you so much for joining us today, and it's been great speaking with you.

Sam: Yeah, thanks for having me.

Andre: Okay, great. DevOps Radio, signing off. Like what you've heard today? Don't miss out on our next episode. Subscribe to DevOps Radio on iTunes, or visit our website at CloudBees.com. For more updates on DevOps Radio and industry buzz, follow CloudBees on Twitter, Facebook and LinkedIn.