Yocto Build Tutorial: BitBake Build Visualization with ElectricInsight

NOTE: Given recent updates with improved support for the Yocto Project and BitBake in CloudBees Accelerator 11.0, we are revisiting our original Yocto BitBake Build Visualization post from 2013 and created the new BitBake Yocto build tutorial.
As ecosystems such as the Yocto Project mature and become more popular for embedded device development, usage of the underlying BitBake build tool is increasing. BitBake is a powerful tool used to manage, build, and integrate complete operating system images through package and distribution management activities such as fetching of source code, configuration, cross-compilation, installation.
As with all software build systems, BitBake speed and performance are of utmost importance and a key enabler of productivity and quality. We have validated this multiple times through discussions with customers over the years. We also validated this with key contributors and users in the Yocto and BitBake communities, where there are ongoing discussions about what can be done to improve performance.

So a valid question is, can we understand how a BitBake build currently behaves and from that, determine if, and pinpoint where, there are opportunities for improvements? The answer is a resounding YES.
In this Yocto build tutorial we will explain our process of understanding Yocto BitBake build performance and provide some useful utilities that we hope the BitBake community will benefit from as the never-ending quest for additional build performance continues!

After following the Yocto Project Quick Start guidelines and some exploration of the BitBake build process and its artifacts, we notice a folder called buildstats under ${TOPDIR}/tmp. This folder has the below structure:

buildstats/
(e.g. core-image-sato-qemux86)/
/
/
do_compile do_configure
:
:
do_unpack

An examination of one of these do_ files (e.g. do_compile) reveals a lot of interesting data:
>cat do_compile Event: TaskStarted Started: 1367381613.31 xkbcomp-1.2.4-r8.0: do_compile: Elapsed time: 3.25 seconds CPU usage: 23.3 EndIOinProgress: 0 EndReadsComp: 0 : : StartTimeWrite: 1725266628 StartWTimeIO: 1725869092 StartWritesComp: 0 Status: PASSED Ended: 1367381616.57
For each task we can get start time, end time and a bunch of other potentially useful data points. Interesting!
The BitBake and Yocto Project communities have already created some profiling and analysis utilities using this data.
The BitBake utility called buildstats-plot.sh tool generates static graphs of aggregate time spent per task-type or time spent per package. The oe-build-perf-report lets you mine performance data - and justify a lunch break or game of ping pong while it fetches data. Interesting, yes - but not very insightful and helpful if you need to dive deeper into what’s actually going on with your build. Where are the build bottlenecks? How do we pinpoint opportunities for improvements?

ElectricInsight is a powerful tool for build optimization, troubleshooting, and analysis. Is there a way to leverage ElectricInsight and the buildstats data to visualize and understand BitBake build behavior and performance? With CloudBees Accelerator 11.0, we included a conversion utility called bb2anno that transforms BitBake buildstats data into an ElectricInsight-compatible annotation file. We can then use that data to uncover actionable takeaways such as when, and in what sequence, particular tasks are run, effects of BitBake concurrency, and overall task-by-time reporting.
Generating an ElectricInsight annotation file from BitBake build stats is done with the bb2anno utility, available as part of the CloudBees Accelerator 11.0 installation (in the /opt/ecloud/i686Linux/unsupported/ directory by default).
Following is the bb2anno command usage, where the -annoroot command is CloudBees Accelerator specific and should be excluded for native BitBake visualization:
bb2anno -id <BUILDNAME> -annoroot <pathto_anno_files> -buildstats <path_to_buildstats> -buildlogs <path_to_build_logs>
The following example shows how to enter the bb2anno command to generate an ElectricInsight annotation file for a native BitBake build:
~/bb2anno -id 99937022 -buildstats tmp/buildstats/20181212102014/ -buildlogs tmp/work > combined_99937022.anno

The best examples are from the real world, so we’re going to use a publicly-available Raspberry Pi Yocto BitBake build , as a reference to see what performance insights ElectricInsight can provide. In the example below we get an at-a-glance overview of the build details: total build duration, level of parallelization, and a timeline of how, when and where the individual jobs ran.

We can see that the build took 72m46s to complete, is using 16-way parallelization, and is composed of 3700 different jobs (i.e. BitBake tasks). Due to BitBake dependency serialization, we note there are significant sections of “whitespace” at about 12m and at about 52m. It is also easy to identify the longest tasks of the build, which suggest potential candidates for optimization.

ElectricInsight has a built-in report that can be used to visualize a heat map of where your build is spending its time, called the “Job Time By Type” report. Let’s take a look at what we get from our Yocto BitBake build:

As you can see, the do_compile and do_configure tasks combine for roughly 70% of the total runtime, where compile tasks average 49s and configure tasks average 42s.

It’s really simple: ElectricInsight is bundled as part of the free CloudBees Accelerator Trial Edition download.
In this blog we've shown how you can use ElectricInsight to facilitate performance analysis of BitBake builds, even if you don't use CloudBees Accelerator. We hope this work will be beneficial to Yocto users. If you want to try it yourself, you can get ElectricInsight and bb2anno via the CloudBees Accelerator trial download.
Keep an eye out for a future blog where we'll expand on this work by leveraging CloudBees Accelerator to dramatically reduce BitBake build durations and stay tuned for next Yocto build tutorial.