How We Automate Our SDK Tests to Support Delivery to Millions of Mobile Users

Written by: Eyal Keren

In our last article, we delved deep into how CloudBees Feature works

and how we leverage method swizzling to make our customers’ mobile applications robust. Having explained how you can search for and fix problematic live code in production environments without AppStore approval, we’d now like to share the ins and outs of how we run our test and delivery operations. With our SDK installed on millions of devices, CloudBees Feature supports hundreds of mobile application developers across the globe. One of the challenges that we faced when planning our test operations and SDK delivery was preparing our backend servers to communicate with multiple versions of our SDK. For example, SDK version 0.1 of the CloudBees Feature configuration for the “Disable Invocation” hot patch looks different than version 0.6, which means that the backward compatibility must be maintained and tested. As a result, we came up with structured and automated testing operations. They enable us to deliver new features on a daily basis without breaking the communication and compatibility between our backend and a mobile endpoint’s SDK.  


How CloudBees Feature Works

To better understand our QA process, let’s look at how our hot patching process works. Let’s say a user configures a hot patch on CloudBees Feature’s site for ‘Application A’ (e.g. try/catch or specific argument). Our backend system then generates a configuration for the hot patch and stores it in our backend server. During the application’s initialization, our SDK (that is installed on the mobile device running ‘Application A’) fetches the hot patch’s configuration data. The SDK then parses the configuration data and alters the application’s runtime to match its newly desired behavior. Learn more about CloudBees Feature’s product and how CloudBees Feature works under the hood.  

Testing Multiple SDK Versions with a Few Clicks

Returning to our QA process, with a single click, we create a (dummy) mobile application to test the new SDK version release. Once initialized, the application runs all hot patches from the configuration data on CloudBees Feature’s backend. We then save the application and tag it according to its tested SDK version. Once the test successfully passes and the new version is out, we put the application aside to test future versions and validate backward compatibility. Today, we have multiple SDK versions that are being tested with new releases. With a few clicks, all of our dummy applications run their individual tests, ensuring every SDK version can communicate with the updated backend server. The test also validates that application behavior changes according to the available hot patch configurations.


One Scenario - Two Tests

In order to remove the dependency dashboard development has on SDK development, our development is separated into two teams. In order to develop in parallel on both fronts, while eliminating the element of dependency on a new release, our dashboard development team provides configuration data to the SDK development team. This way, both teams can develop and test with the same configuration data that the dashboard generates and the SDK needs to work with. Instead of of having one test that starts with dashboard (web) configuration and ends with application behavior (e.g. disabling a function), we create two tests for this scenario:

  1. The dashboard creates the configuration data ⇒ the SDK fetches the right configuration

  2. New configuration ⇒ the SDK impacts the behavior of the application

As mentioned above, these two separate tests share the same configuration data, allowing our teams to be synced and isolated, but have different acceptance tests.

Testing the Dashboard

It’s easier to simply show you how our dashboard tests work: [video width="1920" height="1080" mp4=""\][/video]


Testing the SDK with XCtest

When testing the SDK, we need to validate that it uses the configuration correctly in order to appropriately perform the hot patch on our customer’s mobile application. This step is fully automated using XCtest (Xcode software testing framework), which is equivalent to the JUnit or NUnit testing frameworks. Once we have the configuration data, we plug it into our test environment and write an XCTest to check that the desired application behavior is achieved. Here is an example of a test for the “Disable Invocation” hot patch:

- (void)testHotfixType_disable
   NSArray *configuration = [MMTestsHelper configurationForHotfixId:MMHotfixId_basic_disableInvocation];

   [MMTestsHelper runTestBlock:^(NSString *debugComment){
       MMTweakableClassA *object = [[MMTweakableClassA alloc] init];
       int a = 0;
       int *b = &a;

       object.block = ^{
           *b = 1;
       int result = [object executeBlockAndReturn7];// Call the original function
       XCTAssertEqual(0, a, @"a should remain 0 because block was not called");
   } withConfiguration:configuration];

  This test initiated the system with the disable function’s execution behavior and verified that the function was indeed not called.  

Final Notes

At CloudBees Feature, everyone is involved in the test process. It is vital that developers test their own features to ensure that they are fully working by the time they move to production. CloudBees Feature was created to make sure that your online mobile app is stable and that it is free of critical bugs that can harm your operations and business. We believe that QA needs to start at the definition stage and is an integral part of the development process. As a result, we implement this easy and robust application test and delivery to support millions of mobile user experiences. Learn more about CloudBees Feature set of features

Stay up to date

We'll never share your email address and you can opt out at any time, we promise.