Build is largely a solved problem. CI is prevalent in the industry, people understand how to speed up builds and frequently if using C/C++ turn to CloudBees Accelerator to get fast, reliable builds. Unfortunately, testing is still a problem.
The problem with test automation
Compared to build and deployment automation test automation in many companies is less mature.
Automated tests frequently rely on state from previous tests and therefore can’t be distributed and run in parallel easily.
I’m not going to talk about some fantastic automated way to distribute tests, but should such a product exist I’m sure it would be popular. Whether you have minimal automated tests or a complex suite of automated tests you need to plan to scale.
How can you do this? At the simplest level break your tests into logical groups that have no dependencies. That will allow you to distribute tests across systems using tools such as ElectricCommander so that the test groups can execute in parallel.
The diagram below illustrates splitting tests from sequential execution on one machine to parallel execution on two machines.
This will work well for a while but eventually if there is any level of complexity to the software being tested even that will reach a point where the groups of tests take so long to execute that it becomes impossible to run a full suite of tests frequently.
So what can you do now? Stop writing tests? Remove tests that you believe are no longer needed? Execute a smaller subset of tests with each build and then execute a full battery of tests at a longer interval, for example weekly?
The current solution is to ensure that as many of your tests as possible do not have dependencies on the state of previous tests that have been run. It sounds more challenging than it is and as usual there will be exceptions that cannot easily be isolated so that the tests can run independently but with this approach you stand a fighting chance of having a scalable automated testing solution.
We have frequently heard from people that this level of scalability isn’t necessary only for them only to hear later that the test automation initiative has really taken off and the problem described is now an issue.
Remember, it’s easier to do plan and do it right than to have to fix the problem later.