How an online gaming company leveraged DevOps for a 4,000% increase in deployment efficiency
According to a recent report from Statista, in the last 7 years online gaming has grown from a $22B industry to over $50B annually, fueled primarily by net-based casino games like roulette, slots and poker. Behind the scenes of over 200 of the world’s most prominent online gaming sites is a company called NetEnt . NetEnt (NASDAQ OMX Net-B) is a pioneer in online gaming, entertaining the world for the past 20 years. The company’s innovative platform provides premium online and mobile gaming solutions to some of the world’s largest online casino operators in over 40 countries , handling more 110 million gaming transactions on a daily basis – roughly 16X more throughput than that of the New York Stock Exchange! Check out this short intro video to learn more about NetEnt:
Gaming is Big Business
Online and mobile gaming is booming. With millions of daily active users across the globe, on multiple devices (using mobile, laptops, tablets), and thousands of addictive gaming apps – gaming is big business. Customers are invested in these applications- in making it to the next level, solving the puzzle, beating their friends, and – in the case of Casino games – also winning some $$! Gaming apps have to continue to get better and better, load faster, be more entertaining, more secure, and – of course – be available 24/7/365 for players around the world, and for the businesses who rely on them.
Growing Pains
As NetEnt saw incredible growth, their IT operations experienced a few challenges. With hundreds of online gaming portals reliant on NetEnt’s technology, the backend of their systems became more complex and the Production footprint grew considerably. The NetEnt application is a typical Java app – with a clustered application server, database for transactions and reporting, and so on. Each customer running their own Casino brand also had their own test environment and supporting services. As the systems had to scale to service the growing number of customers white-labeling NetEnt’s technology, the development and IT operations that were supporting these customers became slower and more fragile. The Dev organization was creating strains on the Ops teams, who were having difficulties keeping up with the pace of updates produced. As the number of games and operators in NetEnt’s portfolio grew, they had to rethink their delivery pipeline to allow them to accelerate their releases as well as scale their operations to support the increased load and complexity.
Deployment Pains
In their delivery pipeline, deployments, in particular, became the Achilles Heel of the operation. A full-time employee was required to handle each and every deployment for every code change on every node in the growing Production footprint. This wasn’t just for major releases - but also for bug fixes, security patches and minor updates. The manual process took about 4 hours per deploy, and involved a 40-step checklist, often requiring downtime of the application. There was tons of risk with each deployment and downtime meant unsatisfied customers and lost revenue. These slow, brittle, manual, error-prone deployments meant new features that were developed were taking longer and longer to actually be released into the market, and that on-boarding of new customers and applications were greatly delayed as well. The process of seeing Dev work actually getting delivered into the hands of end-users became risky and unpredictable. For example, looking at the JIRA tickets the Ops team was spending their time on - only 12% was being spent on “revenue generating” activities- such as releasing new games or onboarding new customers. Majority of the time – 88% - was spent on deploying bug fixes and patches. 
How do we scale?
Yes. It was a pain. But, more crucially, it was a pain that can only be alleviated by throwing more people at the problem, The only way to really scale was by hiring more people, to do more manual processes, that became more and more complex. It came to be that over 17 full-time employees ended up doing deployments pretty much around the clock, just to keep the business running. The model, simply, was not sustainable for the long term. NetEnt was feeling the pain - and that was under ‘normal’ operations. But what if something happened?? And then – it did…
The “Big Bang” compelling event: NetEnt at a Crossroads
With no sign of slowing growth, and with a difficult and labor-intensive deployment process, NetEnt found themselves at a crossroads. The world of online gaming is highly regulated and being in compliance is incredibly important – both for keeping the lights on for generating new business. NetEnt found themselves needing to deploy a compliance fix across 70% of their customer base in order to meet new regulatory requirements. There was simply no way to cope with this volume of work with the current processes. A task force was created, with 10 engineers who were re-assigned to be dedicated to the compliance deployments. This team spent the better part of a year and a half (4 days a week, 4 hours a day) rolling out the crucial compliance fix across thousands of servers across all customers. Ultimately, the team accomplished the task. But at what cost? At what loss of productivity? To innovation? To employee satisfaction and retention? NetEnt realized they needed to make a change. From this Compliance ‘Big Bang’, came a new realization, unleashing a new era for NetEnt’s delivery practices.
An Ops-first approach to DevOps
At DevOps Enterprise Summit San Francisco 2017 , Aloisio Rocha, Agile product owner at NetEnt, described the organization’s challenging, but successful journey to DevOps and automation. 
“We wanted to solve the business needs first,” said Rocha. “We needed to mitigate risk – we might not have 18 months to solve the problem next time. We needed to enable compliance and be able to deploy quickly, whenever the business demands that we release. “
NetEnt realized that the next time a change of this scale needed to happen, they might not have the time or the resources to make it happen – and they understood the devastating impact this could have on their business. NetEnt announced “Project Time to Market (TTM) ” – taking an “Ops-first” approach to their DevOps transformation. This cross-functional team has set an aggressive goal for the company: cutting deployment time of new apps over the whole customer base to under 2 weeks , as well as accelerating and streamlining the onboarding of new customers. And they wanted to do it all within less than a year. Aggressive, for sure. But as you’ll see, they ended up doing even better than they expected! The main focus of the TTM initiative was around automation. The team, having documented their delivery process and value stream on a whiteboard, put their minds together to see what the most time-intensive actions were, and where they could use automation to accelerate the most crucial bottlenecks in their pipeline. Naturally, they chose to focus on deployments, to start with. Not only is this the most critical process on the Ops side (finally delivering new code to end users) and that has the greatest impact on the business, but – as we’ve seen from the State of the DevOps reports – Deployment success is the best indicator for DevOps success. “We also wanted to remove any technical impediments. So we took those deployment checklists, and automated them at every step- crossingthem off as we went,” said Rocha. “We took all our decision and approval gates and automated them as well. So, no more dependencies on other teams or manual steps in the process.” A self-service catalog of vetted pipelines and deployment processes allowed team members to easily reuse these automation pieces- ensuring they are the most recent process and configuration, that they are in compliance, and that they do not need ot re-invent the wheel to onboard or support new customers or variants of the applications.
The Solution:
After going through the value stream mapping excerise and white-boarding their deployment processes, NetEnt chose chose CloudBees Flow from CloudBees to model the deployment and release pipelines and to orchestrate all of the point-tools, scripts, configurations and environments involved in the software delivery process under one platform. This not only allows for automating the entire pipeline, but also increases visibility while maintaining compliance and control. NetEnt was able to put an end to “throw-over-the-wall” or manual handoffs between steps in the pipelines or between different operations or tools, and create a fully-automated process with shared visibility to all stakeholders. This helped not only with velocity and speed, but with compliance and auditability requirements as well.
The Result
The results have been staggering and enabled NetEnt to release better products faster.
- For their engineers, what used to take 2 to 4 hours to deploy now takes less than 15 minutes.
- from 1 deployment per month per engineer, the team now does more than 40 deployments/enginner.
- The 30 to 40 step checklists have been eliminated , replaced by a streamlined and automated series of tasks that happens nearly instantaneously.
- The team of 10 dedicated to a large-scale deployment for 18 months is gone – with that brain-power now re-deployed to more meaningful tasks than copy-pasting run books’ tasks. The same volume of work that previously took 1.5 years is now accomplished by a team of just two people, in less than 2 weeks .
- Overall, deployment efficiency has grown by 4000%. The increased efficiency and frequency of deployments has led to faster and more reliable releases, and to an overall better product experience and more satisfied customers.
“One of the benefits we have seen from using CloudBees Flow is the whole process is now automated and repeatable. We're scaling by automation , not by adding more people to our team,” said Rocha. “The whole process is visible to us. We have traceability and are compliant. We're actually spending our time on creating business advantage instead of sitting around copy-pasting all day. And we have achieved time-to-market for our platform.”
Ultimately, NetEnt has enjoyed incredible success because of their quality of products, innovative and creative vision, and dedication. Thanks to their foresight and commitment to staying on the cutting edge, NetEnt has now baked automation and DevOps into their development processes, positioning the company at an incredible competitive advantage as they continue to lead the future of online gaming.
NetEnt shows that when you bet on DevOps, Red (Dev) and Black (Ops) both win, and – of course – the house! :)
To learn more about the NetEnt's DevOps journey, check out the video of Aloisio's talk at the recent DevOps Enterprise Summit:
Accelerate your DevOps Transformation
Aloisio from NetEnt would also be a guest speaker on our upcoming webinar on March 1 , "3 critical keys to DevOps success " - featuring Forrester Research along with some of CloudBees's Customers. 
Forrester Research has identified the key steps to delivering software, faster. Come learn what the research has to say, and the concrete tactics used by businesses today – Intel, NetEnt and Somos – to adopt and scale DevOps throughout the organization. Join us on March 1, 2018, at 11am ET (8am PT) Register for the webinar