Apologies in advance, as this post is rather long and can get quite technical in part. To warm us up I will start with the real life story that inspired me to write the Credentials API in the first place.
Back before Jenkins was called Jenkins (but I’m going to call it Jenkins anyway) and before CloudBees was even a glimmer of a glint in Sacha’s eye, I managed a few Jenkins servers. Corporate IT was good at issuing mandates...
NOTE: The following emails, while inspired by reality, are fictional.
From: "IT Security" Subject: Machine accounts in corporate LDAP serverTo: "All Employees" CC: "LDAP Administrators" Only services provided by and managed by Corporate IT may have machine accounts in the corporate LDAP server
From: "IT Security" Subject: Password policyTo: "LDAP Administrators" Please find attached the new password policy. The main points are:* Passwords must have at lease one of each of the following classes of symbols:- an upper case letter- a lower case letter- a digit- one of the following characters: ?&$@#![]{},./\<>%^()* All account passwords will be rotated at least once every 90 days.* Where an account password is known to more than one person, i.e. machine accounts, that password will be rotated every 30 days* Accounts will be locked out after 5 consecutive login failures
From: “VP of Information Services” Subject: RE: Creation of machine account in LDAP for Developer CI serverTo: “Director of Engineering” Hi Jim,Look, because your engineers need to be able to manage the configuration of their CI server, doing things like installing new Java Development Kits, updating the application itself nearly every week to get new functionality that is being delivered and otherwise tweaking various bits of configuration, it is just not financially viable to have that machine managed by the outsourced IT provider. Our IT provider does not have any experience with this software and the quotes for having them manage it would have be laughed out of the room if I put them in front of the CFO.You’ll just have to keep the app servers in your engineering lab and have your engineers manage them. As such our IT Security team will not allow creation of a machine account in the LDAP server as that could potentially allow engineers to perform actions without traceability.- Bob
So, here am I an engineer left with a side responsibility of managing the three Jenkins servers that our development and QA teams are using to keep us honest and improve productivity as well as a host of other justifications I wrote down to convince the Director of Engineering to file a PO for the servers and even give me discretionary time to work on some open source plugins for Jenkins (in addition to the time I was allocated to work on closed source custom plugins integrating with our internal tools).
In this world, there is one nightmare that haunts me every 90 days… the password reset.
You see our Subversion server was linked to LDAP. Our defect tracking system was linked to LDAP. Our Jenkins Security Realm was configured to be LDAP. Our VMware ESX cluster was linked to LDAP. Everything was linked to LDAP.
In one sense this was brilliant. I only had to remember one password. But every 90 days that password had to change… and each Jenkins server had 15 copies of the password that had to be updated in a race before the old one was used to trigger five consecutive login failures.
It got to the point where the only solution I could come up with was to switch the user that Jenkins used from me to a co-worker on day 88, use tcpdump to check that my username was not being used by Jenkins when a set of jobs was triggered and then switch back on day 91… needless to say, even that was a pain… just a lot less of a pain than ringing up the automated IT help desk to navigate the menu system and request a password reset… with the fun of having to get the Director of Engineering to approve every 10th password reset on the same day for my account.
Fast forward a few years to when I am working for CloudBees and have to design a Credentials API. There was one guiding principle that I held dear when designing the API:
If the user cannot set it up that they only have one single record of their username/password that needs updating and have that record re-used everywhere that is appropriate, then the API has failed.
That does not mean that the user has to set it up that way. If a user chooses to repeat the same username/password in 50 different places on their Jenkins server then that is their choice, as long as it is something that they can choose.
So what does that mean, and how did that appear in the API?
Well the first thing that it means is that we do not want gach duine agus a mháthair (an Irish idiom literally translating to "everyone and their mother”) implementing their own version of a username password credential. We want one standard username/password credential implementation that has a username box and a password box.
Reading the above, it might come across that I am a brilliant shining beacon of wisdom who implemented this principle from the very first line of code that I wrote in the credentials API...
Ha!
Is mé an t-amadán (do your own Google translate)
Ahem! What was I thinking ? Well in my defence, I did get there eventually … (making many mistakes along the way… and thankfully making mistakes is how I learn… and boy have I made more than my fair share of mistakes!)
Perhaps I should explain myself. My first thought was that how you authenticate with SSH is an SSH specific thing. When configuring an SSH connection you want to be able to select the credentials that are valid for use with SSH. So I decided that all credentials for use with SSH connections should extend from SSHUser as the one thing they had in common was that they all had to have a username. Now because I wanted to keep the SSH specific stuff in an SSH specific plugin there was no way to have the standard username credential implement the SSHUser interface, we’d have a circular dependency amongst modules. So instead we ended up with two Username/Password credentials implementation. One for when you wanted to use the Username/Password for SSH connections and the other for when you wanted to use the same Username/Password for other purposes.
In affect we were using the type system to colour credential instances so that we can then filter out invalid credentials based on their colour.
I have learned that using the type system this way is a very big mistake.
To understand why it is a mistake, I need to introduce a concept, which I will call the “Oracles of Identity”. Perhaps the best way to explain this concept is by way of example.
When you create an account in GMail, somewhere in the back of Google’s servers there is an identity store that creates an entry for you and stores a hash of your password. When you go to GMail or Google+ or YouTube, ultimately all of those applications consult with Google’s Oracle of Identity to authenticate you.
When you create an account on GitHub, there is also an Oracle of Identity in amongst GitHub’s servers.
If you change your GMail password, that does not affect your GitHub password, just as if you change the password in your corporate LDAP server (which is your employer's Oracle of Identity by the way) then that does not affect your GitHub password… but that may well affect your GitHub Enterprise password (because enterprises - and users - love single sign on)
Oracle of Identity is easiest to understand for username/password type credentials, but it can also apply to:
- ssh keys (if you have a service that distributes your SSH authorized_keys file… and an NFS backed network home counts as one of them by the way)
- TLS client certificates
- OAuth tokens
- etc
Now I can hear people saying “OAuth Tokens, WTF! that should never” and I may very well agree with you, on the other hand...
Q: What is the most secure way to store credentials in an application?
A: Not storing any credentials at all is the most secure way.
Supposed I have been tasked to write some service, say maybe it does Black Duck style scans of code on demand. The service is not something that we want to run all the time, but when we choose to run it, it needs access to our source code. Now I could spend a lot of effort securing that service so that it has secure storage of the credentials it requires, add intrusion detection, credential abuse detection, monitoring, etc… or I could put it running in a reasonably locked down VM (or container) that I tear down after each use and pass in the credentials when I want it to do something.
Because the service doesn’t actually store the OAuth bearer token and because the service doesn’t exist when not in use, I can write a much simpler service.
It’s a pragmatic pattern… perhaps it would even get replicated for other services… and then a while later you have ten different services all using the same OAuth credential (because they all need the same/similar scopes, and the OAuth server will only issue one active Bearer token for the requesting hostname - i.e. the CI server - and treats new requests as a sign to invalidate the previously issued Bearer token)
Now my understanding of how that “should” be designed is that each of those ten services should have an OAuth authentication layer. Then Jenkins can authenticate against each service giving Jenkins a unique token for each service… each service then provides a UI to allow managing what the identity can do and manage the OAuth keys that the service has for each user, and then using that UI you then give the service the OAuth keys to source control… certainly there is a lot of design in that… and we can have a separate type for the OAuth credentials in Jenkins for each service, and then we can filter by type...
Or we can recognise that it is the users - and not us the developers - who should decide the colour of a credential. They are the ones who can know/find out the Oracle of Identity for that specific credential that they are giving to Jenkins. And in the Credentials API the users provide that colour by using Credentials Domains.
Each Credentials Domain is really defining an Oracle of Identity. The domain specifications define how to determine which Oracles of Identity might be valid against any specific service.
For example, when you select Git as your SCM, the credentials drop down get’s activated and returns all credentials within scope that are of a type that the Git SCM plugin knows how to use. When you start to type in the Git SCM URL the plugin uses that URL to build the domain requirements that will be used to refine the list of credentials available (the credentials in the Global Domain are always available). So by the time you get to git://git.example.com we have a list of domain requirements that says scheme=git, hostname=git.example.com. That immediately rules out the Google Oracle of Identity (because that domain has a domain specification that says scheme=https hostname=*.google.com) and the GitHub Oracle of Identity (because while that domain specification says scheme=ssh,git,https the hostname must match git.github.com) and we are left with the Acme Oracle of Identity which has just one credential, simplifying configuration for the user.
Now a lot of people will look at the DomainRequirements/DomainSpecification pairs and mistake them for DNS domain names. It doesn’t help that the only DomainRequirement/Specification implementations that I have provided to date are ones for URI Scheme, Hostname and port, in a large part because I haven’t had a need to implement additional non-URI related ones yet . Perhaps I should have called them a Credential Demesne instead… i.e. they define all the land in which the contained credentials are valid for use.
If a Jenkins Plugin developer needs to add colour to their credentials, they should do that by defining a DomainRequirements/DomainSpecification pair that expresses the colour. That lets the users paint their credentials by adding “Demesne”Specifications to their Credential “Demesne” so that when the plugin looks for credentials through the filter of the “Demesne”Requirement it only sees the painted credentials.
Of course, currently we do not make the Credential Domains/Credential Demesnes/Oracles of Identity easy for users to use:
- The in-place Add button only adds to the Global domain
- There is no UI to move credentials between domains
- (unrelated, but while I am at it) There is no in-place update button to let users correct a bad credential
- (another slightly unrelated) The in-place Add button only adds to the root credential store and does not let you select from the other potential credential stores, e.g. user and the job’s parent folder stores
But that doesn’t mean we should colour the Oracle of Identity using the type system.
So the question then comes for the plugin developer: When should I implement a new credential implementation then?
The answer: When you need to provide a different UI for the credential.
By way of example, let’s consider GitHub access tokens . A similar analysis applies for Google App Passwords . These are actually passwords. You can use them (assuming they are appropriately scoped with permissions) as if they were a password, e.g.
curl -u bob:passw0rd https://git.github.com/acme/...
curl -u bob:t0k3n https://git.github.com/acme/...
But we probably want to provide an alternative UI for users so that they don’t have to copy and paste the token manually. Instead we would prefer to provide some form of web flow to request the token and store it once the user has approved its issuing. So in that case we need a custom credential type. It will have the following characteristics:
- Implements StandardUsernamePasswordCredentials - if it can be used like a standard username/password (i.e. in the example above it works with curl -u)
- Stores the Oracle of Identity that issued the token - because we want to be able to validate/refresh/renew the token
- Perhaps stores the permissions that were requested be assigned to the token
Finally if the token itself can be used in a different way from a standard username/password pair, e.g., if we can do something like
curl -H "Authentication: Token t0k3n" https://git.github.com/acme/...
Then we may want to define an additional interface to colour those credentials that can be used in that way… alternatively, and in my view better, is to use the Authentication Tokens API to provide a means of converting credentials into a form that is valid for use with the “Token” authentication scheme as that avoids completely colouring via the type system and instead leaves colouring to the user or to plugin developers as capabilities exposed via the extensions lists… it is much nicer (in my opinion) to say
BearerToken token = AuthenticationTokens.convert(BearerToken.class, credential);
Than
BearerToken token = credential instanceof BearerTokenCredential ? ((BearerTokenCredential)credential).getBearerToken() : null;
and thus force other plugin developers to extend your BearerTokenCredential type just so that they can consolidate the credential usage to simplify life for the user.
In any case,
Plugin Developers: if you force the user to have the same credential information stored in two credential instances in the same credential store, then your design is making life worse for the user and forcing them to run the risk of having to update credentials in many many places every 90 days. I personally think forcing this on the user sucks, and if I were the user of such a system I would say that the design sucks and who ever designed it was stupid to design it that way… that does not mean that your design sucks, you may indeed have very good and valid reasons for wanting to design your use of the credentials API that way… but take great care because your users may think you suck, so make damn sure you have a very good and valid reason that cannot be addressed instead by using either/both DomainRequirements/Specifications and/or the Authentication Tokens API
Users: congratulations for reading this far. Hopefully you have a better understanding of what Credential Domains are for and now want to consolidate your credential instances down to save you from repeated calls for password resets every 90 days. Keep pestering us to make the credentials management in Jenkins better.
-Stephen Connolly
Elite Developer and Architect
CloudBees