Silos Begone! The Road to DevOps at Autodesk and Lessons Learned Along the Way

Editor's Note: This blog post was a joint collaboration between CloudBees and Autodesk.

George Swan, Senior Director of Autodesk’s Build Platform, shared lessons learned from Autodesk’s move to DevOps using CI/CD pipelines over the last three years, at a keynote for the 2018 AllDayDevOps conference sponsored by SonaType.

Swan can now add “DevOps Transformation Leader” to his long list of accomplishments. DevOps.com named Swan as the Best DevOps Transformation Leader for 2018 for leading his engineering team and implementing “whole product” DevOps across Autodesk.

This DevOps transformation was driven by three critical business factors. First, Autodesk was moving from a perpetual maintenance model into a subscription licensing model where customers expect frequent and regular delivery of new content. Secondly, Autodesk went all-in on being a cloud platform company, which demanded that they adopt modern DevOps practices. Thirdly, remaining competitive in a fast-moving market meant adopting modern social-engineering practices, becoming much more collaborative in the way they work together and develop shared software, leveraging a shared reference development, build and deploy platform.

The environment at Autodesk will likely sound very familiar to most folks in a large enterprise. When Swan joined four years ago. Autodesk had around 200 products across desktop, web and mobile platforms. Product teams generally didn’t release frequently. Two weeks was considered fast, and one year being slow.

Development teams were highly siloed, as were the tools used to develop and deploy product.

“We were siloed and had every tool known to man,” said Swan. “Engineering prided themselves in their speed of execution, and executive management valued that oversharing/collaboration.”

If an existing component did 90 percent of what they needed, they would copy it and add their own 10% rather than inner-source the existing component. Much of the company grew through acquisitions, so this just exacerbated that problem.

Despite this, Autodesk were experts at getting complex product out the door, but it was clear that their tools, process, and culture around development and deployment were old-school and needed to be refactored.

Swan’s vision for the new world was one in which content developers can focus on what they do best – content creation, and let all the builds, tests, packaging, publishing, and deployment happen fully automatically, only to come back if tests failed in some way. Autodesk’s new aspirational goal is that the system allows each content developer to push two deployments a day, which works out at about 4,000 deploys per day.

The vision didn’t start with the idea of DevOps for the whole product, but now it does. Wouldn’t it be fantastic if a change to source code, product documentation, API documentation, or even a config file meant that the service or application was automatically rebuilt and deployed to all the endpoints needed? Wouldn’t it also be great if the product seen by the customer was always consistent across code and docs? Wouldn’t international customers love having localized content available at the same time as the English version? And lastly, wouldn’t it be even more fantastic if they could take an epic in the Jira backlog and execute on a deployment across the desktop, web services and mobile app impacted by that epic?

The journey started with executive buy-in and support. Luckily, the timing on seeking this was good - The business needs were very much aligned with how DevOps could help.

Swan said, “Had we tried this a few years earlier where exec focus was on localized fast-execution, I think it would not have progressed so well.” In the end, the execs saw the benefit and funded a team to implement CI/CD.

Before Autodesk could move to a modern CI/CD pipeline , it was important to update its development tools. Engineering migrated from Perforce to GitHub Enterprise and set up Artifactory to store all binaries. They also adopted Codacy, SonarQube, Whitesource, and Checkmarx to perform static analysis during CI. The deployment infrastructure for deploying desktop product still met the need, but they needed a whole new deployment infrastructure for cloud. So, Autodesk built out their own deployment platform for deploying containers to Amazon ECS. Project tracking was handled in Jira, and Slack was used for collaboration and communication.

Autodesk then created its CI/CD pipeline on top of this platform and orchestrated it using the pipeline plugin in CloudBees Core. Originally this pipeline was only for developing and deploying source code. However, the pipeline soon expanded to include what was needed to deploy a whole product.

The first extension was a Documentation pipeline based on using the Markdown language for documentation source. The pipeline took the Markdown, generated the HTML using Metalsmith, tested it, published it to Artifactory and then pushed the content out to various targets. So now the source for the documentation can sit alongside the software, either as its own repo inside of an organization or as a folder inside of a repo.

Since 75 percent of Autodesk’s revenue comes from non-English speaking customers, the next project was to enhance both the software and documentation pipeline to include Continuous Localization. In this way, any time a resource in software was changed (e.g. an error message or a UI component) it was automatically localized into the target languages. They did the same for all documentation content.

Lastly, Autodesk needed a better way to manage the generation and governance of open source, third party and internally-built binaries. They solved this by adding a component CI/CD pipeline. Components reside in their own GitHub repo and each has its own Jenkinsfile to build, test and package up the binaries before posting to Artifactory.

They learned several lessons along the way:

• Transformational leadership was extremely important, said Swan, noting that Autodesk could not have been successful without it.

• Rolling out reference solutions does not stop at implementation. Success must be defined as having a solution in place that engineers love *and* is used by over 70 percent of engineering teams. Swan said, “We never stopped selling, marketing, nagging, shaming and helping teams until we got there. Initial migrations got the full white-glove treatment. We tiered customers based on their need and our pipeline capability. To scale the migration, we needed to invest in self-serve on-boarding tools, documentation, and training.”

• Component governance was hard. Their siloed history meant teams had their own copies of shared binaries, and in some cases had modified the source locally. Finding owners for shared components, and unifying the source code and binaries back into a single GH repo and Artifactory folder continues to be a challenge.

• Handling documentation with Markdown is wonderful. The source is stored alongside the code, and all the DevOps best practices apply. Developers are more invested in documentation because they know what to do with a pull request (versus reviewing and editing in some alien documentation tool). Finally, this encourages teams to write docs at the same time as the source code.

• Localization can be continuous. It took a large investment in autotranslation, translation dictionary to store the content, and building out a compute farm to do the translations, but the technology exists for others to do the same.

• Automation is only the first step. If engineering does not adopt the processes and best practices recommended by the likes of DevOps Research and Assessment (DORA), then teams can’t fully take advantage of that automation.

• GitHub: 4,000 users, 16,000 GitHub repos, and 18 million Git commands a day – fetches, clones and commits.

• Artifactory: storing 1-1/2 million artifacts and downloading those at a rate of 1.8 million a day.

• CloudBees Core for orchestration: The Jenkins pipeline plugin was chosen because of the technical support offered, especially for the transition. Half of Autodesk’s 4,000 engineers are currently generating 54,000 builds per week.

• Codacy and SonarQube for functional defect detection.

• Whitesource and Checkmarx for security vulnerability detection and license compliance.

• Jira for bug tracking and for project management, processing 2-1/2 million issues a year.

• Confluence for the internal wiki: 400,000 pages.

• Slack: 8,000 users and 7,000 Slack channels.

Autodesk’s CI/CD pipeline is not one pipeline focused on integrating and deploying software. It consists of multiple sub-pipelines integrated together, as shown in the following diagram:

Autodesk has a:

• Component pipeline for generating and deploying the binaries used in product builds. Centralizing in this way allows Autodesk to control changes made to shared content, and manage which binary versions can be used in builds.

• Source code pipeline to develop and deploy content for desktop, cloud and mobile platforms.

• Documentation pipeline based on Markdown. Pipelines for generating and publishing API documentation does exist in the company, but are not yet integrated into the reference pipeline.

• Localization pipeline. Software resources or documentation can be automatically translated to any of the 12 languages that Autodesk supports.

What's the next challenge for Swan? Setting teams up to successfully deploy localized source and documentation 4,000 times a day in a fully automated, repeatable and SOC 2 compliant way. Exciting times ahead!

• Read how to unlock the power of DevOps

• Learn how to kickoff a DevOps journey

• Discover the top experts in DevOps