Making Automated Tests Truly Automatic

_
A recent poll of software development professionals showed that the majority would rather be doing their taxes than dealing with their company's test infrastructure. The reason: automated software tests require tremendous amounts of manual time and energy to configure, run, and monitor. This can be a startling revelation to companies that have invested substantial engineering efforts into automated test frameworks specifically to reduce the human cost of continually running large regression suites.
_What's at the root of this disconnect?

We've had the opportunity recently to interview some of the world's largest development organizations and were astonished how similar the problems were across wildly different shops. Test suites take time to configure and launch, there's a perpetual arm-wrestling match with IT for resources, and the staff is still spending hours trawling through log files, painstakingly trying to tease out the real defects from the spurious failures and bad tests. Effective automation should obviously be about reducing manual effort, and yet the most successful test systems are backed by large infrastructure teams tasked solely with the care and feeding of the "automatic" system.
Three challenges crystallize as the answers to the questions above:

1. It's hard to get the environment, including both harness and product, set up on an appropriate resource

2. It's hard to invoke the test

3. It's hard to mine the results for concrete next actions.
   This triumvirate -- resource configuration, test execution, and results monitoring -- is the "last mile" of automation, and when ignored, the entire value of automated test pays a steep penalty. Fortunately, for each of these systemic problems, there are simple though not necessarily easy solutions we can apply to great effect.
   The resource selection problem is simply this: where do I run the test? At most shops, the answer itself quickly leads to a sysadmin puzzle: where or how can I easily log into 20 systems in one click, where do I have ssh keys configured, how will I keep my tests from stomping on my neighbor's, how do I fire up VMs from an ESX server for my config, and so on. The correct answer: stop asking this question; get your test harness to do it for you. There are three critical pieces you need to make that a reality and one big trap to avoid. First, tests need to support distributed, remote execution; second, the target host and the build need to be logically divorced and merged together only at test invocation, and third, as much possible, standardize on one host environment for high-volume, frequent 'smoke test' runs. The pitfall: if the test process is hardcoded to a host ("version 3.4 can only be tested on Windows 2003 hosts with VS 2008 + our addin") the inefficiencies and complexity start to creep in. Standardizing on a host configuration for all active branches can open extraordinary optimization opportunities. Need to run a new system test? Grab the next available node from the large homogenous grid or cloud, load product and go.
   The concept of carefully sub-setting the test suite to run a critical selection of smoke tests first is not new, but it's essential to efficient automatic testing. The natural objection from developers is that the project-wide smoke test possibly will not cover the code they most recently modified. Enter the second solution pattern: tests harnesses need to be highly parameterized, but their invocation needs to be easily aliased. This is a common pattern you use all the time: type "alias" at the prompt of grizzled Unix command-line hackers and you'll see shortcut entries encapsulating commonly used options to programs like "ls"; on Windows the Start Menu is nothing but a tree of shortcuts to programs and documents. Effective automated test systems pull the same stunt: a million possible options to pick the smoke tests, the server tests, the UI tests, and so forth -- but they are easily abstracted behind a single, user-specific click on the harness interface. Corollary gotcha: don't let your harness sources live with your product sources, it encourages an artificial integration that makes building such an interface much more difficult.
   Finally, the most widely cited culprit for consuming valuable man-hours is dealing with the large volumes of data produced by test systems. Automated harnesses are usually great at posting gigabytes of HTML tables with green and red icons to web pages and inboxes, but they often fall far short of making that data actionable . Actionable reporting means that for every data point presented to the user, you ask yourself the question: "If I show them this, what will they want to see next?" Too often the email says, "test run #231 failed", but leaves it to the user to identify the offending test; too often the web page says "test 144.... FAILED" without any indication if it's been failing since last March or if it's been green since 2006 but turned red this morning with the new guy's first checkin. Designing the test harness to make error collection -- and just as important, error reporting -- as intelligent as possible can have a dramatic impact on ease of use and productivity.
   So while the natural evolution of a test harness will lead it being tedious to configure, slow to run, and cryptic to interpret, there are straightforward strategies to remedy all three problems. And as shops that took the time to refactor the critical parts of resource selection, harness invocation, and results reporting have shown, an easy, fast, efficient test harness may still not top your list of favorite leisure activities, but it's a great deal more pleasant than filling in Form 1040.
   ElectricCommander provides efficient, effective unit and system test automation by dynamically provisioning and configuring the target environment, eliminating manual invocation of complex test suites, transforming volumes of test result data into actionable metrics.