Key Takeaways from Continuous Discussions (#c9d9) Episode 45: DevOps and Security

One of our recent Continuous Discussion (#c9d9) podcast episodes focused on best practices around the tenets of DevOps and Security. Our expert panel included: Andreas Wittig , author of “Amazon Web Services in Action” and software engineer, speaker, teacher and consultant; Dave Bechberger , a senior architect and developer at Expero; J. Paul Reed , an internationally recognized speaker on DevOps, release engineering, and operations complexity; Martin Cron , principal engineer at WiserCare; Sukhbir Dhillon , founder of Addteq; and our very own Anders Wallgren and Sam Fell. During the episode, we discussed how DevOps can help secure your code, environments and processes, as well as how it can help improve visibility and compliance.

Securing Your Code

#Code quality hygiene is just as important to code security as release engineering hygiene - @martincron #c9d9 pic.twitter.com/2wAGAp0MCM

— CloudBees (@electriccloud) July 12, 2016

Security in your source code is something you must educate your teams about - @andreaswittig #c9d9 pic.twitter.com/79DM1ja2vK — CloudBees (@electriccloud) July 12, 2016

Release engineering hygiene is crucial to securing your code, and actually really easy, says Reed : “If you’ve got a library that has a security vulnerability, that’s a security problem. If you’ve got 58 libraries in your products with a security problem, that’s a release engineering problem. Training your team on StringBuffers and Java-whatever is important but also really hard. The really low hanging fruit turns out to be release engineering hygiene, and it comes from simply knowing what you are using.”  

Bake security into your code, says Bechberger : “I think the most important thing I’ve ever seen about securing your code is that you have to start building your code from the beginning, it isn’t something you can add afterwards…I was reading an article where someone found that they were using Reddit servers and about 6,000 Reddit servers were open to the Internet with no security. There has to be a tight integration between developers and operation, both of them have to work together to figure out how to build a product and then actually secure it going forward.”  

Dhillon advises if you are going to have libraries, have them come from one source: “We were working with one client: they have 9,000 applications, and they don’t even know what libraries they use.  If they are not coming from a common central location, where you author the security rules, you will always have these problems.”

  Working in the health industry means Cron has to take extra measures to ensure security; he does so by running unit tests: “I have unit tests that say ‘Try to talk to our servers on clear text’ and if they respond at all to clear text the we know we are doing something wrong. We’ve got the same set of tests running against all our servers so if anything talks clear text at all the tests fail, the pipeline stops.”

  When it comes to securing your code, securing your source code is crucial, per Wittig : “Security in your source code is something that you need to bring into your team and need to educate them about; you need to get into new techniques and iterate over the thing you already have security-wise in your source code. A big part of it in my opinion is educating the team to get everybody aware of the security issues that can arise when running source code.”  

Securing Your Environments

Shared passwords also impact governance in a #DevOps environment - you can lose sight of who made the most recent version @samueldfell #c9d9

— CloudBees (@electriccloud) July 12, 2016

Create a culture where people are encouraged to bring forward security concerns - @bechbd #c9d9 pic.twitter.com/4xP8bu6tSN — CloudBees (@electriccloud) July 12, 2016

Adopt a hacker mentality when securing your environment, advises Dhillon : “When you’re shipping your code you’re never thinking from the hacker mentality, you always think from a good cop side, whereas the hacker might be utilizing so many other things which you never thought of. When you are put in that position you will think like that and automatically improve your code quality.”   Securing environments is often a cultural problem, notes Reed : “The thing that is so interesting about security, especially from a DevOps culture perspective, is it’s a) how do you incorporate the knowledge that even the developers know and b) how do you organizationally deal with the problem so you don’t have passwords on sticky notes or self-sign certs that you rely on. This is a frustrating part about security because there is a lot of low hanging fruit and it’s not a technical, knowledge, or training problem.”

  The most important element for securing environments per Wittig ? Building security into a continuous delivery pipeline: “Have a secure infrastructure, a secure environment for running applications. That’s an infrastructure that is defined by code and that is part of your deployment pipeline. That means having pure reviews for security-relevant infrastructure parts like authorization tools, cloud services and then to deploy those things automatically to production to avoid human failure, to have a machine do all that work and also to have automated testing for the security configurations, testing infrastructure and also in combination with the application.”

  Use automation to secure your environments, says Cron : “The safest environment is the one you don’t let humans muck around in. If you have your pipeline with automated deployments, you’re not SSH-ing into the server and doing things manually; and, if no one can do things manually, or if doing things manually is such an exceptional case, it shows up in the log and you can see who was doing what, when.”

  Configuration management is key to securing environments, says Bechberger : “We have all seen tons of hard-coded configurations of connection strings, passwords or things where you have a configuration file that was just checked in, but build some sort of infrastructure around managing that. If I need to go to production, there is a path for me to get there, it’s not going through the back door and logging in with an admin password or an admin SSL.”  

Securing Your Process

If your processes are onerous then people will cheat and go around them - per @anders_wallgren | #c9d9 #security pic.twitter.com/Cle0bXP40k

— CloudBees (@electriccloud) July 12, 2016

"Teams don't spend enough time trying to hack their own code" @addteqsukhbir #c9d9 #security in #DevOps pic.twitter.com/CZOvA7MZSs — CloudBees (@electriccloud) July 12, 2016

Culture also plays a big role in securing your process. Cron explains: “Make the propaganda posters that say ‘This is a value we care about and is a part of our process.’ Have things like transparency and code reviews be absolutely integral to the process. we have total transparency – I trust everyone on the team, we don’t have to have different roles because everyone is literally watching everyone else.”

  Best practice for securing environments? Hack your own code like Dhillon   does at Addteq: “People never give a time for the developers to hack their own code. We run break-a-thon sessions and hack our own code, and whoever finds the most vulnerabilities gets rewarded and everybody is happy that they are doing something different.”

Want to secure your process? Do Agile, says Wittig : “ the idea is to iterate on your products so you improve them step-by-step, and sometimes I wonder why we are not using the same process to improve security. As iterations on our security are getting better step-by-step it means we have to start with the basics and then we have to get better and better. Probably our product owners are not happy with investing huge amounts of developer time to get to a security level that is extremely high, so we need to start small and then iterate and keep getting better and better.”  

Create a culture that fosters security, advises Bechberger : “I think the other part is another culture aspect, which is to make it known not only to the development team but also to management and your product owners why security is important and why it may take slightly longer to build up the secure way versus the insecure way. I have seen many times where people want to cut corners because security takes too long, but in the end it doesn’t really take too long, it’s far easier to start now than start later.”

  In order to secure your environment, have a little bit of trust up front, advises Reed : “When you talk about process, that involves people and people are really good at getting around process, especially brittle processes that doesn’t work. When you talk about securing process, what you’re basically talking about is a shell game – you’re basically saying ‘I don’t trust my developers.’ That’s a losing proposition – if you don’t trust your developers it’s already game over.”  

Improve Visibility and Compliance

Visibility & comliance: It's not a hard thing to explain and easy to show the value to people - @martincron #c9d9 #DevSecops

— CloudBees (@electriccloud) July 12, 2016

"Automation is auditing"- mantra from @anders_wallgren | #c9d9 pic.twitter.com/Wfs98oU7DV — CloudBees (@electriccloud) July 12, 2016

Advice on how to increase compliance, per Wittig : “Compliance rules are sometimes hard to communicate to the whole organization especially if your teams are changing a lot, so one thing that I like is to have compliance rules as code, and check your infrastructure and your application against them.”  

Visibility and compliance are your reasons for doing security right, says Reed : “If you do security right, visibility and compliance are the reasons you give to the business about why you do it. They are natural things that fall out of a good process.”  

Cron gave a traceability demo to their non-tech savvy CEO, and it clicked for him, so Cron suggests you can explain visibility and compliance to just about anyone: “Visibility and compliance are not hard things to explain and show value to others.”  

Watch the full episode here:

Want more Continuous Discussions?

We hold our #c9d9’s every other Tuesday at 10 a.m. PST, each episdoe featuring expert panelists talking about DevOps, Continuous Delivery, Agile and more. Next time on Continuous Discussions: Open Source and DevOps .