Developers Eat Everything in a DevOps World

No doubt you have heard that Software is Eating the world and possibly even that Developers are the new Kingmakers. Will if all this is true, then there is a lot of responsibility that new rests with developers (with people practicing DevOps, this is also self evident). Even before the rise of DevOps, developers were effectively entrusted with a lot of responsibility for the success of the business. Millions of dollars, or in some cases even lives, could be at stake.

Surely you have heard that with great power comes great responsibility? The flip side of this “new kingmaker” capability is that now (a lot) more is expected of developers.

I have heard many people lament the amount of setup required (and knowledge needed) to build a service these days. Kubernetes leads the charge here, but it isn’t limited to that. I even spent (mis-spent?) a Friday afternoon trying to get something akin to a “Hello World” happening on AWS Lambda (well a little more, and I was limiting myself to tools that Amazon publishes vs. third party tools, but still…). You can read more about that here. It was more work than I had hoped, but the end result I was very happy with (warning contains a video with some swearing in it).

This made me reflect as to why things “seem harder” for web app development, and why I needed to keep more things in my head. At first it seemed something was wrong, but whilst talking with some colleagues (Pete Muir and Laura Tacho) they gently informed me that this is just development now (especially for the cloud). There are just more things that as a developer you may come across.

This does not mean that there is more busy work to do - it is just that in the past this stuff was hidden when apps were “thrown over the wall” to Ops to deal with. This work always had to be done. Now things have “shifted left” and developers have to know or be deeply aware about it.

I went through various things I have had to think about over the years that (in my experience) many developers do not tend to think about (but may need to) and cooked up this list:

  • DNS
  • TLS/SSL termination (and perhaps be nerdy enough to cringe when people say “SSL”)
  • Network topography
  • Database connection management (per environment)
  • API gateways, network edges
  • CDNs
  • Latency
  • Rollback implications
  • Traffic shaping (when rolling out new versions of the apps)
  • On call schedules
  • Managing disk usage and IOPS
  • Data schema upgrades and migrating data

I am sure you could think of many things that should be on there that developers now need to be aware of. I say “aware of” as no one expects someone to be an expert in all these areas. I once misallocated a bunch of time to attempt to build my own DNS server (I learned that DNS uses the word “bailiwick” which I am sure was retired from the English language centuries ago).

Different platforms put different expectations on what a developer needs to be aware of. If I plotted where responsibility for things like the items listed above lies:

  • Traditional deployment puts a lot of responsibility on Ops specialists, this is quite well known (and lots has been written about these roles in the past, and their ongoing role in DevOps practices).
  • Serverless actually takes a lot of traditional ops concerns away, but does ask more of the implementer (a developer) to consider things around quotas, access rules, API gateways and more.
  • Kubernetes possibly asks the most. Whilst there are a growing set of very well supported cloud providers that run it “natively” (Google, Microsoft, Amazon) and do most of the heavy lifting for you, developers should not be surprised when new (or perhaps long forgotten) concepts come up as part of their usage of the platform. For now it seems like Kubernetes could be the right level of abstraction that allows a developer to make sensible decisions around things like DNS, without actually needing to configure them at a low level.

For example: a common enough problem is how to include a connection string to a database. Many developers will find “novel” solutions, and many dev frameworks will help, but the core of the issue is how to specify you want to access “tcp://your-database” and have it use the right data, in the right environment. In a platform like Kubernetes there are ways included that do this with DNS (after all that work setting things up), so one of the payoffs is you don’t think about this, you can’t really mess it up. You get to benefit from that forever from that point on. Each deployment doesn’t have to reinvent how to do that. When you move projects, you don’t have to reinvent it (even if you move companies perhaps - if something like Kubernetes is becoming an operating system).

As I mentioned before, none of this is really new; it is just now far more visible to developers. This isn’t busy work for the sake of busy work and stress, there are benefits: faster delivery, making more use of the platform (this means less re-inventing things by developers, ie less bespoke code). You could think of something like Kubernetes as having for a one time cost as we establish more standard, and easier to use, building blocks of the cloud (vs. every deployment pipeline being bespoke).

I like to think of these new platforms as having an “OS” to build on, and install things into. For Kubernetes, that allows really powerful tools to be installed like Istio, Kubeflow, Kubeless, Falco, Knative and more almost like you install it on a “normal” operating system (with package managers, upgrading and all that good stuff you don’t have to worry about). It would be near impossible to achieve this kind of productivity with traditional deploy targets in a “flip it over the wall” world - but with a cloud OS we can!

Also a final shout out to Jenkins X for making me think about all this. It is doing some of the heavy lifting for app developers so there isn’t too much burden of knowledge (let it create the pipelines and helm charts and deployments for you).  

Additional resources