Advanced Build Cloud resource partitioning and prioritization

Written by: Electric Bee

3 min read

Stay connected

With the recent advancement in cloud technology, more and more R&D organizations are looking to optimize and improve their internal development environments by deploying Private Development and Build Clouds – CloudBees technology is playing a key part for many in making this happen. As organizations grow their cloud deployments, flexibility in scheduling of resources and planning of infrastructure becomes a necessity. The rest of this blog post will discuss an advanced such use case that recently came up for one of our CloudBees Accelerator Build Cloud customers.

We recently organized our fifth annual CloudBees Summit . As we usually do at these events, we setup the “Dr Is In”-desk, an area that our Engineering team is staffing during the event to allow for direct interaction with our end-users, customers and partners. The discussions often become technical (as they should) – illustrated in the above diagram and in the below bullets is a request that we received about advanced CloudBees Accelerator resource partitioning and prioritization in their Build Cloud:
• Assume two different pools of resources, Prod and Dev.
• Developers will use resources from the Dev pool, but can also use idle resources from the Prod environment when available.
• Production builds will use resources from the Prod pool, and should always preemptively get back any resources from developers, on-demand when needed.
• Production builds should not use resources from the Dev pool because of incompatible and slow infrastructure.
• Finally, a higher priority class of builds should be able to override all the other normal production builds to get preemptive access to all needed resources from the Prod environment.
How do I configure this in my CloudBees Accelerator Build Cloud?

Solution

Below is an outline of the proposed solution using 3 CloudBees Accelerator agent resources in a small example. Please refer to the CloudBees Accelerator online documentation or http://ask.electric-cloud.com for complete technical details.

Setup and Configuration

1. Assume you have three CloudBees Accelerator agent resources in your Build Cloud, agent1, agent2 and agent3:

2. Configure the CloudBees Accelerator Cluster Manager to run in “priority pool” host manager mode, with 'shared' agent allocation, and 'priority' agent preemption.
3. Configure the different CloudBees Accelerator pools, using CloudBees Accelerator resources. Note the below “__” prefix that defines pools part of the Priority Pool scheme:
a. define ‘__pool_dev’ for the Dev pool of resources
b. define ‘__pool_prod’ for the Prod pool of resources
c. define a resource ‘prod’ that points to the same resource as in the ‘__pool_prod’ Prod pool

4. Define a build class ‘high’ that has a min-agent to be the same as max-agent e.g. 30, and ‘high’ priority:

Usage and results

• To run a developer build:
emake --emake-cm={cm} --emake-resource=dev: {additional options}
With no other active concurrent builds in the Build Cloud, this would yield the following CloudBees Accelerator cluster usage, note this build is using the agent1/agent2-resources despite those not being explicitly part of the Dev pool:

• Assume the developer build is now complete, run a production build:
emake --emake-cm={cm} --emake-resource=prod:prod {additional options}
Note there is no usage of the agent3-resources in this build:

• While this production build is running, let us now initiate one concurrent developer build on the Build Cloud. Note the developer build is not fairly sharing the production resources:

• Finally, initiate a high priority production build that needs 8 agent-resources from the CloudBees Accelerator Build Cloud.
emake --emake-cm={cm} --emake-resource=prod:prod --emake-class-high {additional options}
Note the preemptive scheduling of the CloudBees Accelerator resources to satisfy the high priority build, followed by the allocation across the Dev and Prod pools to satisfy the normal developer and production builds:

Stay up to date

We'll never share your email address and you can opt out at any time, we promise.

Loading form...
Your ad blocker may be blocking functionality on this page. Please disable for an improved experience.