This is a follow on to our post about Android bottlenecks , where we talked about how CloudBees Accelerator can shorten Android builds by 300%. A few weeks ago we introduced v 9.0 of CloudBees Accelerator that included enhancements to the JobCache add-on, including the Shared JobCache feature. We’d like to take a few minutes to illustrate how this works, and how your development efforts can benefit from it. First, let’s review a bit about JobCache and how it works. JobCache is an optional feature of CloudBees Accelerator that intelligently analyzes past compilation tasks along with recent changes, and then smartly reuses object files that aren’t impacted by any of the recent changes. In other words, it shortens build times by figuring out what parts of the build don’t have to be recompiled. Caching is nothing new. ccache has been around for years but JobCache has some significant advantages:
- It’s up to 300 faster than ccache, especially on Android builds.
- It’s designed for reliability so that it can even be used for production builds – a well-known risk for ccache .
- The visualization tools make it easier to identify and correct build bottlenecks – a near-impossibility with ccache.
- It’s more collaborative by spreading the benefits of eliminating costly and redundant efforts of rebuilding unchanged modules across the whole team.
We recently attended Agile 2016 , DevOps Enterprise Summit, and AppSphere 2016 and surveyed nearly 1,000 attendees about their development and deployment challenges. We’ll discuss the results of those surveys in depth at another time, but one statistic that jumped out at us was that over 50 of respondents complained that they frequently experienced problems because of manual processes, workflow and dependency complexities, environmental differences, and deployment failures.
CloudBees in-booth survey of 1000 attendees from Agile 2016, DOES 2016 and AppSphere 2016 While we didn’t survey for the root causes of each of those types of issues, people told us one common source of those problems: developers hadn’t used the latest build components when they did their builds. They were relying on the contents of their own output trees and local caches without checking to see if newer versions were available. Ouch. That’s not helpful if your bonus is tied to timeliness or quality metrics. Not only does Shared JobCache eliminate the drag of rebuilding components that were unchanged, it eliminates the more costly problem of reusing the wrong or outdated components, too.
So how does Shared JobCache work?
Whether you are using JobCache locally or shared, the basics are that it maintains a slot for each combination of command line options, relevant environment variable assignments, and current working directory. A slot can be empty or can hold a previously-cached result. If the appropriate slot holds an up-to-date result, a cache “hit” occurs, and compilation is avoided. A cached result becomes obsolete if eMake detects that the contents of any of the files needed for that compile have changed in addition to checking all of the combinations mentioned above. Such file changes include any files that are read during compilation, which means all source files, precompiled headers, and compilation tools included in the eMake root. JobCache is able to detect file system changes by using the distributed file system called Electric File System that is part of CloudBees Accelerator. To get going with Shared JobCache someone from the Release Management Team designates the shared cache in a path name. Individual developers get a local instance of JobCache by default. When a “golden” build is run, overnight, for instance, the results are also copied to the Shared JobCache. When an individual developer does a build of their morning’s work, they reference the Shared JobCache path in their own make files. That build automatically draws unchanged objects from the shared cache, ensuring the entire team is working from the latest version of everything. If a developer changes one of those components, the MD5 hash will be different and emake recompiles that changed component during the build. 
Golden Output results are copied to the Shared JobCache so individual developers can automatically reuse unchanged components Users of IBM’s ClearCase will recognize that Shared JobCache feature provides the same functionality as the winkin feature in ClearCase.
Your Parents were right: Sharing makes everything better
It’s bad enough wasting the time to rebuild a component that didn’t need to be rebuilt. It’s even worse, and potentially more costly, to take the time to rebuild something because you or someone from the team didn’t use the latest version of a component. CloudBees Accelerator avoids both of those scenarios and can provide up to 300% speed improvements over open source tools like ccache . In an up-coming post, we will show how the visualization and build analytics tools within CloudBees Accelerator go beyond dependency management to actually predict and control build times based on the number of CPUs in the cluster.
Accelerate your Android Builds for free
Learn More about CloudBees Accelerator and Download CloudBees Accelerator Huddle Now!