Why Fidelity of Environments Throughout Your Testing Process is Important

Product Release testing can be a complicated matrix of requirements, especially in today's complex software and hardware systems. Releases take untold number of hours to test to ensure the quality of those products before they reach the customers' hands. This article addresses the release of web-based product (SaaS), and the importance of ensuring environment fidelity throughout the QA process. When testing complex applications, it is important that during all phases of the testing process that the test environments are as similar to the actual production system as possible, but it becomes more important the further along the release process the Release travels. The Product Release may go through multiple phases of testing, including: Development Testing, QA Testing, Performance Testing, Soak testing, and a Staging (Pre-Production) environment before it is released to the actual Production Environment. As Aliza Earnshaw at Puppet Labs states: "To get there , though, you have to move code from the developer’s desktop environment into the testing environment — or environments — and then on into production. All these environments have to be configured the same way, or you run the risk of finding that code that worked in QA , for example, breaks in production. " Let's consider the reasoning behind homogeneous test environments:

During the Development phase of testing, the Engineering team will be testing multiple iterations of their changes on a daily basis. While it is important in this phase of testing for the environments to be similar with what will be found in the Production system, there are many aspects that will be harder to duplicate. For example, there will likely be only one server which may serve multiple purposes, such as the Application Server, Web Server, Database Server, and so on. There will likely be no load balancer, no redundant servers, and no fail over process. At this point in the testing, that is expected. But the servers themselves should be similar to what might be found in production. Testing an application on a Windows Server that will eventually be running on a Linux Server is not as effective as testing on a Linux Server with the same Operating System and other software packages installed. It is important that the same Operating System (including the patch level), and other software packages be the same versions as those found in the Production Environment. The potential for testing an application throughout all phases of the Release Process, only to find bugs when it hits the Production Environment because Production was using a different version of software, is an issue that no team should encounter. By having the Development Testing environment similar to the actual Production environment that the application will eventually be installed on, the team can eliminate potential problems related to software versions. If there are interactions with other software packages that are not available in the Development Testing environments, it is important to have simulators or emulators in place to mimic that interaction. If this is not done, then potential problems related to that functionality will not be found until later in the Release Process, when the cost of finding problems is much higher. The Development team's ability to test this functionality, especially if they are making changes that could affect it, is vital to the quality control of the application.

Once development of a new functionality has made progress, the application will be turned over to the Quality Assurance group. At this point in the process, there should be a more comprehensive set of environments available to this team, that are even more similar to the Production system than the Development Testing systems. This is where multiple Application servers, Load balancers, and other systems found in Production should be in place. Emulators and Simulators will be more important in this testing phase as well, as more comprehensive testing will occur at this time in the process. The environments that will be used here should be nearly identical to those used in production. All software installed should be the same versions as those in Production, and the hardware itself should be as similar as possible as well. This will help to reduce the possibility of any issues that are related to the environments themselves.

Both Performance Testing and Soak Testing need to be done in specific environments that again should mimic the Production environments as closely as possible. While Performance environments will generally be specific and well documented, to ensure that the performance metrics are useful, they should be similar to what exists in production, allowing the Performance metrics to be a more accurate representation of what customers might encounter. Soak Testing is to test typical production load, and so it should also be done on environments similar to what will be seen in production. The closer that the environments are to a production environment, the more accurate the testing results will be.

The Staging environment, used to test the release before installing in the production system, is best managed by the Operations team - the same team that will be managing the installation on the Production environments. By having the Operations team do the installation into the Stage environment, it allows them to have a first look at the installation process, and to have a practice session before the installation of the application in Production. The Stage environment may be the first chance that the QA team has to actually test some third party integrations (such as the banking or credit applications that do not allow testing in Development and QA environments). By having the Stage environment be as similar as possible to the Production environment (in both Software and Hardware), it allows the QA team to have the most accurate view into how the application will operate in Production before it is actually installed there. By allowing the Operations to do the installation in the Staging environment, they have a chance to use the automated installation before using it in the production system, allowing another set of eyes to check the accuracy and quality of the installation before being used in a critical production environment. By testing in this environment, QA has a last chance to find any issues before installing the application into Production. This is their best chance to find bugs before the application is released to the customer. Therefore it is vital that this environment be the closest to production as it can possibly be, to eliminate the possibility that issues will be found in production that are related to the differences in those environments.

Once the application is installed on the Production Environment, testing on this environment should be done to ensure that the software runs correctly. If the previous environments have been configured as similar as possible to the Production environment, very few issues should be found at this point in the testing process. Any issues will more likely be related to load or other factors that are strictly available within production (such as the links to 3rd party systems that were not available in the previous environments). This will help the QA test to narrow down the possible causes of any issues, and hopefully allow them to resolve any issues more quickly. By reducing the variables between the various phases of the testing environments, issues can be found more quickly, when they cost less to fix, then have them found once the applications reach the Production Release.

A vital factor to this process, which was not addressed in the previous sections, is ensuring the similarity of the installation during the various testing phases. Having a single installation process for all environments ensures that the installation process itself is tested throughout the process, and also further ensures that the install is done similarly on all environments. While there will be certain things that will differ on each of the test environments throughout the Release Process, those differences should be kept to a minimum. Differences in configuration information can be stored in a text file and read in during the installation process. This ensures that any differences can be tracked for the various environments (by storing them in a file they can be checked in to your Code control system, allowing for traceability as well). Storing these values in a file also allows for repeatability , there are no data entry problems if these values are modified by hand during the installation, the values are simply read from the configuration file. By using an automated installation process , it increases the repeatability of the installations, and ensures that the same process is used on all environments throughout the process. It further reduces the possibility of making errors during the install. The automated install can be used in all phases of the Release Process, testing it multiple times, ensuring that by the time the installation is done in Production, the automation has been thoroughly tested itself. The key is repeatability- with an automated process being much easier to repeat than an installation process that includes many steps where human interaction is necessary. Having an automation platform that allows you to easily install and configure your Release will allow the teams to concentrate on any issues that arise, rather than on any errors made during the installation itself.

In conclusion, it is important when testing any product that it be tested in environments that are similar to those that the customer will see. When you are testing a complex, integrated, web-based system this can be a challenge. For example, if your application has some type of banking links, or credit authorization, you will often not have the ability to test these in your internal QA environments against live banking systems. You may not have that opportunity to test these links until you reach your staging environment, and that testing may even be against a testing system based at the bank, or it may use emulators. However, it is still important that in all phases of the testing that the systems throughout are as similar as possible. This allows the Quality Assurance team to ensure that any environment related issues are kept to a minimum, and that the systems that are being tested on accurately reflect what the applications will be run on in Production.

Electric Flow Deploy allows companies to manage the deployments of various applications and all components associated with them in an automated fashion. It allows for increased repeatability, traceability, and reliability. It truly allows for Continuous Deployment in a way that ensures the various testing environments are consistent throughout the process. References: 1) Aliza Earnshaw, "Build a Toolbox for Continuous Delivery"; https://puppetlabs.com/blog/build-a-toolbox-for-continuous-delivery