In the first two parts of this series, we've looked at: (1) Amazon Web Services (AWS) credential management and, (2) remote login/developer security. In today's post, we'll look at some best practices in credentials and password policies utilized by CloudBees in managing our Platform as a Service (PaaS) environment. These are important practices that are fairly easy to follow without causing undue burden on the development team.
Centralized Password Management
Aside from Amazon Web Services (AWS), we have quite a few external service providers we use to run the day-to-day business at CloudBees. For many of these providers, we have a single account that is used by CloudBees to provide services that are both internally and externally facing. From time to time, our developers may need to log into these services to update information, check settings or get reports.
As noted in Part 1 of my blog series , having a single set of credentials is an undesirable scenario. Each new developer has to be given those credentials and it's hard to keep track of exactly how far and wide they float around. In addition, if a developer leaves or some event happens that necessitates change, then redistributing the new credentials becomes painful.
As a small team it was less of a problem, but as the team grew a better way of handling this had to be developed. For the past year and a half, we have tackled this problem largely through the provider we use, OneLogin.
Each developer has a separate login to our OneLogin system. This can be username/password based, or as I prefer to do, sync with my Google Apps account and use my Google Apps login via OpenId. Once logged in, I am presented with a menu of services that have been predefined for me to be able to log into. A simple click, and I am taken to that service providers page and am logged in automatically. I don't have to remember any specific login credentials and account passwords don't have to be shared.
OneLogin has made it very easy for us to control access to all of our important systems (including logging into the AWS console). An obvious downside is that OneLogin is now a single point of failure for access to our critical infrastructure. However, we've never experienced a technical issue that has prevented us from using the service. Our admins still maintain the username/password list of various logins, just in case.
As an extra layer of security, our administrators use two-factor authentication via a cell phone app when signing into OneLogin.
One often overlooked area in security control is the email address used for service registration. For many of our services, we register an email@example.com email address that gets forwarded to the development team using that service. This is great for informational updates, but it also means that password reset information goes to that alias -- and then on to the email address of the team members whom we may be trying to abstract that information from.
This is one important detail we've combed in setting up our Amazon EC2 accounts; we want to ensure if a set of credentials did get stolen, particularly from a hacked email account, that an attacker would find it difficult to reset an account password in order to gain access to a system.
Keeping Credentials Private
We're big fans of GitHub, and we have a number of public and private repositories there. However, we also maintain a local private Git server for a handful of critical repositories that for internal security reasons we've decided are better kept closer to home. This includes certain repos which don't need to be widespread and are more critical for behind the scenes operation. We tend to keep repos that may have services credentials stored in them contained completely within our own infrastructure, as opposed to storing them in a third party location.
In addition, we never use plain text email to send important credentials to end users. When sharing AWS credentials with a new developer, or getting an SSH key set up, we always transact using GPG encrypted data. This added step adds only about 30 seconds to the process, but helps ensure if a developer's email is ever compromised that credentials are not part of the leaked data.
We've found that a lot of developers use email as a permanent archive of data. If you send them some kind of login information, it will stay in their inbox/archive forever for "future reference." By sending encrypted credentials, the credentials will continue to be available in the future, if needed, but will ensure that security breaches won't lead to data loss.
I hope that in this three part blog series you've learned how seriously the CloudBees team takes security, as well as can appreciate some of the best practices we use in maintaining security on a day-to-day basis. If you have suggestions or thoughts on security that weren't covered in these posts, please add a comment below.
Caleb Tennis, Elite Developer
Read Parts 1 and 2 in Caleb's Securing the Cloud blog series: