The early days of the cloud saw many exciting things - not all of it was good:
the ability to provision servers in a few minutes;
the ability to spend hundreds of dollars on instances due to stopping and starting instances and being billed an hour at a time;
the ability to terminate servers in the wrong account because you ran the script on the wrong server.
In the various development cycles and pivots within CloudBees, we progressed through various phases of cloud nativeness, system management and maintenance.
The IT operations team at CloudBees is responsible for the ongoing day-to-day management of all systems. However some engineering teams take on varying levels of responsibility for their systems (e.g. the CloudBees DevOptics team have some operations staff embedded in their team).
Day 0 Ops. "Wow Michael, this cloud thing is cool, what can we do with this?"
Day 1 Ops . "Nice features! Hey, we've got customers!"
Day 2 Ops. "Umm who has credentials to log into this failing server over here while Ivan is away?"
Source: DZone, Defining Day-2 Operations
Roll on seven more years of development, pivoting, scaling up of services, scaling down of services and the Day 2 Ops team is still keeping the lights on.
But now the engineering teams want to run services in Microsoft Azure, AWS, Google Cloud and perhaps a few line of business applications in Acquia Cloud.
Managing the system infrastructure went from something you could do in your spare time as an engineer, to a full-time job for a team of six with rolling timezones, holiday coverage and specialist skill sets to manage the multitude of systems, products and environments.
So you rolled out management infrastructure for one cloud (in our case Amazon) and all was well. You've got IAM and dozens of squad-level AWS accounts. YOU ARE ROCKING THIS CLOUD PARTY!
BUT NOW THEY WANT MORE CLOUDS.
ALL THE CLOUDS.
AND THEY WANTED IT YESTERDAY.
AND THEY WANT TO MANAGE IT THEMSELVES!
The 5 stages of operationalization
Denial . "This is a fad, we are only going to use Amazon."
Anger. "Why do they keep trying to use Google Cloud, when they have admin access in Amazon?!"
Bargaining . "Ok, you can run services in Google, but NO PRODUCTION."
Depression . "What do you mean the Google bill is up to $10,000 every 6 days? IT'S NOT EVEN PRODUCTION!"
Acceptance . "Sigh, ok, you win, lets fully operationalize these systems."
Come and listen to all these issues and more in my presentation, Hybrid Cloud Continuous Delivery at DevOps World | Jenkins World 2018 . We'll talk about the operations side of continuous delivery, but also covers the human aspects of working with a growing team of engineers who just want to get software out!
What did we do?
Where are we now?
How did we do it?
How do we balance InfoSec requirements against usability?
What did we do wrong?
What would we do if we did it again?
What really worked well?