On the Proper Setup of a Virtual Build Infrastructure Part 2

In this post I will get into some additional findings from our deployment and also tie all our findings into the deployment of CloudBees Accelerator in a virtual environment.
You can find part one of this article here .

It is important to generously size the disks for the virtual machines. It is a hard balance as you don't want to waste all your precious disk space on the virtual machine server or SAN. Keep in mind, though, that OSes tend to grow over time, as service packs are installed, logs grow etc.
It is, of course, a good idea to keep the virtual machines in good working order, especially if they are long lived. With service packs being released continuously, it is important to have a few free gigabytes around to unpack and install that download.

When it comes to staying compliant, the current virtual infrastructure offers its own set of challenges. Licensing falls into three main categories:

1. Per user licensing

2. Per machine/core licensing

3. Volume/site licensing

The enforcement of these models varies among vendors, and can make virtual machine deployment very hard. This section mainly describes the challenges we have encountered, and, where possible, I will point out solutions.

1. Per user licensing can be tricky when you opt for a "generic" user account. If you have virtual machines that all just have an account called "build", and you have paid for a single license for that the account, you are probably not in compliance. Technically, this may work, but legally it is potentially problematic.

2. Per machine licensing is challenging in virtual environments, as many vendors key the license off the MAC , which will have to change for every deployed machine. It very much depends on how long lived the machines are. For machines that are permanently deployed, this is not an issue if you just ensure that you have a proper license for each of the machines. For machines that come and go frequently, you may want to check with the software vendor to see about a more centralized/floating license model for the tool.

3. For volume licensing, you generally end up with a license key for everyone, or a centralized license server. In either case, you should be problem free.

One resource that might help specifically with licensing questions around Microsoft Windows is Emma Explains Microsoft Licensing in Depth! . In general, it's a good idea to check with the respective software vendors to verify their policy.

Ok, so far this was kinda generic, so now you want to know how this all relates to CloudBees Accelerator. Here are some guidelines specific to CloudBees Accelerator when it comes to virtual environments:

1. Emake will happily run in a virtual machine, but because it is essentially a file server, it’s worth ensuring that slower virtual machine I/O performance doesn’t create a bottleneck. This is less of an issue if the build directory is on a mounted file system, but the emake machine is still the main bottleneck in this setup, so the more horsepower it has (cpu as well as disk I/O), the better for the overall build.

2. Running CloudBees Accelerator agents in a virtual machine is certainly not a problem, and should scale in accordance with the aforementioned constraints. Make sure that the disk I/O requirements are appropriately addressed in your virtual setup.

3. Running the CloudBees Accelerator cluster manager in a virtual machine should work fine. In this scenario, you don't want to upload large annotation files with all your builds (that is generally not recommended due to the high disk volume). You may also want to use a remote database for your cluster manager, as databases tend to have high demands on disk I/O as well.

4. When using antivirus software in your virtual machine, you will see a performance hit due to extra overhead of scanning files. The general recommendation is to at least exclude the CloudBees Accelerator agent's caching directories, as well as temporary directories, from scanning. From a security perspective, this is not an issue, as all the files stored in that cache will eventually be sent to the emake machine, which should have antivirus software installed. This means that files are scanned only once when they are written to disk on the build machine.

1. We opted to keep some physical backups around for corner cases in development/debugging and as a backup.

2. Virtual infrastructure works great for our testing and debugging, and has allowed us to stay on top of an ever-increasing number of machine configurations.

3. Disk space management remains a challenge. Our virtual infrastructure solution provides ways to build virtual machines off common templates, a boon for disk space saving. However, users can force a local copy of those images (intentionally, or unintentionally), which can quickly increase disk space usage.

4. Virus Scanner support for dynamically deployed machines is challenging. So far the solutions we have found do not do well with some of the more advanced capabilities of virtual machine deployments (fenced mode, etc.). We will probably have to wait and see what antivirus vendors come up with.

5. License management, in general, is a challenge.