Julien Vehent of Mozilla Firefox: Implementing Security in DevOps

Andre Pino: In today's episode, I'm joined by Julien Vehent, the leader of security architecture for Mozilla's Cloud Services division. Welcome Julien.

Julien Vehent: Hi, pleasure to be here.

Andre: Julien, can you start by giving our audience a brief overview of your role at Mozilla?

Julien: Sure. So as many of you may know Firefox is a client-side application that you run on your computer. What you may not know is that when you run Firefox it talks to a number of services hosted online and that may be to receive updates to the application or to publish telemetry data or to synchronize your bookmarks between various installations of Firefox, et cetera, et cetera. And all of these online services are operated by a division of Mozilla called Cloud Services. And what my team does is helping Cloud Services managed the security of their operation. So we pretty much make sure that everything that is run on the backend side of the Firefox infrastructure is run securely, that the database protects the user data at a very high level because a lot of the data is you know, end user data that they trust us with, their browser history or shared passwords, et cetera and we make sure that we can block attacks, that we can prevent attacks, that we can ship new services at a high level of security from Day One, and we can help operations teams and developers build and run their services securely.

Andre: Sounds great. So Julien I understand that you've recently published a book; congratulations. It's called Securing DevOps. Can you tell us a bit about it, what people should expect from the book, and what the biggest takeaways are from it?

Julien: Right. So I wrote that book over the last couple of years to sort of summarize all the work we had done on the Cloud Services side of Firefox into a technical manual that people could use to implement security inside of their DevOps organization. So I once again noticed talking to folks at Mozilla but also in many other organizations that they knew they had to do security, they agreed it was important, but they weren't quite sure how. And particularly when we look at DevOps organizations that take risks, you know those who ship services faster and wouldn't want to automate a lot of their delivery pipeline and be able to release very, very quickly then the techniques that we use to increase the securities of their systems are fairly different from the traditional way of security in a data center. And I had to solve that problem with my team at Mozilla in Cloud Services and I saw it would be good both for myself, but also for people out there who have to solve the same problem to organize all of that knowledge into a book and that's what Security DevOps is. So it's a very technical manual with a lot of concepts that are explained at a very high-level, at a strategic-level of how to implement security into DevOps. You could call it DevSecOps or SecDevOps, whichever you prefer, and then we dive down into the actual implementation of these concepts and how we implement security testing for example in Jenkins or in a continuous integration, continuous delivery pipeline, how we will implement the GPS security, how we would control for example the configuration of an...account, but also how later on we would protect those services online by doing fraud detection and incident response, et cetera, et cetera, all the way to risk management and risk assessment techniques that are needed to kind of mature the security of the organization. So it's everything from Day One, you ship your website and it's online and you're probably going to get hacked at some point to three years down the road you have a whole system in place and you want to improve it and you need to do risk management for that.

Andre: Great and you know you mentioned DevSecOps. That's just one of the most biggest topics within DevOps today. In fact, at DevOps World | Jenkins World this year it's got its own track; it's just a very important topic. And, you know, within DevOps we always talk about continuous, continuous integration, continuous delivery, continuous testing. Is continuous security something that can be achieved, do you think?

Julien: Yeah, I think continuous security can be achieved. It's both a strategic achievement for many organizations that need to approach security in a different way and not the traditional way of doing testing at the very end of the pipeline, but also starting to do testing earlier in continuous security and continuous integration, but also it ... could change. A lot of teams currently do not integrate well with development and operations teams and culture in DevOps is all about that communication aspect, the aspect of having the developers and the operators working together very, very closely, sharing tools, sharing timelines, the devs may write some changes to the automation of the infrastructure, the ops might write some changes to the application to make it work better in the infrastructure, et cetera. That is a close collaboration and the DevSecOps aspect that I think will allow an organization to implement continuous security is having security engineers join the developers and the operators in those concerted efforts of building products and services and actually contribute to the products rather than being at the end of pipeline and trying to fix things after it's done after the fact.

Andre: And so when an organization wants to get serious about security and their continuous delivery efforts, what is your suggestion on how to approach security? I know as you've mentioned culture is a big aspect of it, but across the technical, the process, and the culture aspects, what would be your recommendation for organizations?

Julien: Well it all depends where you are. Some organizations stop their security efforts after a breach so they got compromised, maybe their data leaked, their database got breached and there's a very traumatic event that forces everyone in the organization to rethink how they do things. And for better or worse this is how most organizations get into security. They got breached, they will call somebody for help, maybe a consulting company and they will start maybe hiring a consultant, maybe they will start hiring people internally or really organize their resources to focus more on security, or sometimes you know they are ahead of the curve and they know that whatever they do is risky and they want to protect their customers from Day One and they want to start doing security. In the first case you have the resources. The leadership is already onboard because you got hacked basically so it's fairly easy to say, "We're going to take whatever got hacked and we're going to start fixing that." So you have a starting point, you have an entry point, and you can take that very, very small scope of your infrastructure, focus on it, and then when you're done fixing that, maybe write some tests to make sure it doesn't regress and you put these tests inside of your CI/CD pipeline and run them continuously every day, or at every deployment, and move on what is the next thing that we think may be vulnerable and you can expand from there. In the second case, if you're really starting from scratch and you're lucky enough that you haven't gotten hacked yet then you need an entry point; you need something that's easy enough to get you in, is not going to take too long, and is not too controversial that you won't have to fight the organization to do it. A good idea may be to take the OS top 10 and look at vulnerabilities in there and if you're concerned about SQL injections for example you could run a security scanner to test your web applications for SQL injection vulnerabilities and run that inside of your pipeline and you start from there and you expand from there on. Maybe you run a different type of scanner, maybe then you look at your firewall programmatically, et cetera. The point is I think to be successful, and I think that's a mistake that many security teams make, you have to start small and solve one small problem and automate it and write your regression tests, make those tests pass, and then move onto the next thing. If you start to take, for example if you run a vulnerability scanner across all of your applications Day One and you try to take that entire chunk of data and solve it as your first project, you're going to drown in data. It's too much and a lot of organizations get stuck in that stage where they have too much security data and it's not all organized, they don't know which one to address first, and they end up not making the progress they should be making.

Andre: So, for organizations that want to take a proactive approach and not wait until they have a breach, you know, are there organizational changes they should make to bring security to the prominent position it really needs to be in the software development and DevOps transformation that they're embarking on?

Julien: I think so. Well, ideally you don't have to change the organization. Maybe you have to pick a couple of people in the organization that are more interested in security and enable them. I wrote the book for those people too. Maybe you have somebody in the operations team, or somebody in the development team that's a little more interested in security and wants to start looking at these types of problems. And what I would do in those organizations, is to you know they can pick up a copy of the book, or any other resources online that explain security and tells them to take 30 percent of your time this quarter and go look at the security of a web application aside from everything else that you're doing and tell us what you think and work with the other developers who work on that project. Work with the other people from the operations team or QA and try to get a little bit of an assessment of where we are and what is potentially vulnerable. And the important part of this is that a security effort works better if it's done by people who are trusted by their peers rather than somebody coming from the outside, like an external consultant, that will come in and nobody knows the consultant and will give very shy answers, maybe not the full story. And if it's done that way with somebody from the inside then there isn't this resistance to the security efforts that a lot of consultants have to fight. And I think it's also better to build trust between security teams and operations teams and development teams so that everybody shares the same success for the organization; it's not just an us against them situation. I've seen a lot of startups and small companies grow their operations. It often comes from the operations team, grow a couple of junior engineers from their ops team into security engineers by essentially asking them to apply a security mindset to what they do and when that succeeds and it becomes a more important part of their job they actually form a security team that is dedicated to security. So it's entirely possible to grow your organization into a security-aware organization.

Andre: I like that term, “security-aware.” I think it's really important. One of the biggest security pushes recently has been secure browsing with HTTPS. How important is this to the software development community and to consumers?

Julien: I think it's so important that it is becoming the default and that ... HTTP is going to become the exception. And this actually, the fact that we're turning HTTPS into the default is a very strong change in the way security is being approached on the internet. For the longest time, security was a separate effort. We were trying to bring in security for people who wanted it after the fact. Nowadays, we learn over you know a few decades of experience on the internet, that it needs to be baked in in order to succeed. And the new versions of HTTP, for example HTTP/2 requires HTTPS so there's no way to do it the insecure way; you are going to be secure by default. And it is critically important for consumers because we've seen, and we continue to see, many intermediaries on the internet, ISPs or antivirus services that may appear to be free and provide free service but sometimes misbehave and we see them modifying the content that is sent by web applications to consumers and sometimes even injecting ads. So the lack of integrity checking, the lack of security on the transports tampers with the quality of the content that gets to end users and if it's just a, if you're just browsing some random site and the content you're viewing is not that important it may not be a problem but there are many, many cases where that is could potentially be a major problem if the information you're viewing is very sensitive. So for consumers HTTPS just allows them to trust what they see and that is very important. For developers, it is becoming more and more important to be HTTPS aware because the web browsers will just require it for everything that's new. So if you want to use the new fancy features that are going to ship in Firefox and Chrome and IE and Opera and Safari over the next few years, you basically need to provide your website on HTTPS. Otherwise, the browser will say you're on an HTTP channel and you cannot use these functionalities. So HTTP/2 is an example and there are a number of others that browsers will block. And that used to be very difficult because setting up on a site on HTTPS, like there's a whole chapter in Securing DevOps on HTTPS, used to be very difficult but improvements in cloud infrastructure and how we build and manage services have made this very, very simple nowadays. If you essentially host anything on AWS, and you for example create an elastic load balancer, you will right away get that on HTTPS, with just a couple of clicks it will be enabled for you. Most hosting providers, Heroku, GCP, Azure, et cetera, will make that very easy and if you're hosting it yourself nowadays we have Let's Encrypt, which again is very easy to use. So we have lowered the entry barrier and made it very easy for everyone to use HTTPS to the point where it's so easy we can make it a requirement, we can make it a default, and we don't have to offload that onto complex, sophisticated, security teams that have to do it for everybody else.

Andre: That's interesting, sounds like some big changes coming. So one of the other big movements today is the shift towards microservices architecture. And as organizations do transition towards a microservices architecture, are safe and secure services something that's possible? I think that's probably something you've spent a great deal of time on.

Julien: I have, and microservices are difficult. They are, I think, making the engineering aspect of services a lot easier at first, but then you have to, you can't do your microservices if you don't have a good deployment pipeline. Basically you need to be really good at DevOps; otherwise all of the services are going to put a lot of pressure on your operations team are going to be really, really hard to manage, and you're going to collapse at some point. I think there's a balance, like a few years ago everybody was all about microservices only in that it was the future and that's what we needed to do. Nowadays what I'm seeing, at least at Mozilla, is more of a balance between monolithic services and microservices. So we don't go fully microservices where providing a single capability will require deploying five different things, but we also don't concentrate all of the functions into one massive application that cannot be reuploaded without breaking all of our users and sessions and whatnot. So there is a small balance here. Maybe we'll find a middle ground, we'll slowly grow an application from like a single service when it provides a very simple functionality at first into a microservice environment when we want to add more functionalities. The challenge for security teams here is that if you are not fully DevOps, if you're not DevSecOps basically, if you don't have your automated testing in place, if you don't have a way to audit your infrastructure programmatically, then microservices will multiply the number of services you have to audit by an order of magnitude. And you will essentially collapse. You will probably be able to look at one or two services every time you do a round of audits, but you won't be able to maintain a good security posture on your organization. So when organizations move more to a microservices model and they improve their operations and they improve their development, security teams have an opportunity to look at those services and say, "Well, we can provide a template for all of those services and start all of them with a set of defaults," and we're going back to starting with a default secure environment and start them with a set of defaults in the pipeline. So for example you will have an Engine X configuration, right and all of the microservices will start with a template Engine X configuration and you shove all of your HTTP headers and your content security policy, your HSTS, all of your controls in there in that templated Engine X configuration so that all of the new services that are created will use that. In your pipeline when the services are being deployed you add a few security tests. At Mozilla we use the OWASP Zed Attack Proxy, ZAP and we use Jenkins as well and we will in Jenkins run ZAP to test that application and test that indeed the security headers are present so we know that when a new microservice is being deployed to a staging and production environment it is compliant with our security standards. So that allows us to not have to worry about the new stuff, not have to do a lot of manual testing, and know that things are being taken care of programmatically. That also frees up time to go look at more complex type of problems, for example auditing access controls on databases and sure, there is the aspect of firewall rules, but maybe you want to look at the grand permissions, which if you have a microservice environment hopefully they don't share databases. They will have their own _____ but sometimes they do so if you have two or three services talking to the same database which permissions do they have, how do these permissions change over time can you audit them? That's pretty hard to do. And we can go look at these problems that are historically hard to solve because we freed up time by implementing automated testing in CI/CD. And I think that's really the win in the DevSecOps approach is that we are freeing up time to not have to worry about the basic stuff and go look at the more interesting and more complex aspects of security.

Andre: Managing by exception, right?

Julien: Yes, exactly.

Andre: Julien, thank you so much for your time today and congratulations on your book Security DevOps. If folks would like to get a copy of that book how can they do that?

Julien: Well they can go on securing-devops.com and that will redirect them directly to the page. I have a paper copy in my hands right now and it looks great. I'm very excited to have it finally published and I'm hoping that folks will be able to order their copy very soon.

Andre: Julien, thanks so much for being with us today. It was a great conversation. Thank you.

Julien: My pleasure. Thank you.

Like what you’ve heard today? Don’t miss out on our next episode. Subscribe to DevOps Radio on iTunes or visit our website at CloudBees.com. For more updates on DevOps Radio and industry buzz follow CloudBees on Twitter, Facebook, and LinkedIn.