Your Essential Test Environment Management Checklist
by Carlos Schults
Your Essential TEM Checklist
“Test Environment Management Checklist.” Yep, that sounds like a mouthful, but don’t let that discourage you. The idea here is quite simple—adopting a checklist to evaluate the soundness of your test environment management approach.
Even though the idea sounds simple enough, the execution won’t necessarily be as smooth. There are lots of potential questions. What should the items on this checklist be? Who’s responsible for authoring and maintaining it? Even the definition of “test environment management” might be fuzzy.
This post’s mission is to answer the questions above, and more. By the end of the post, we’ll have answered these questions:
- What is a test environment, and how do you manage it?
- How can you use checklists to verify the health of several processes?
- Which items should you include in your test environment checklist?
Without further ado, let’s start.
Test Environment Management Checklist:
Understanding It Step-By-Step
We’ve been talking about test environments as if everyone was familiar with that term. We understand that’s not the case, so before we get into the checklist itself, we’re going to devote a little bit of time explaining what a test environment is and the importance of managing it.
Test Environments and How to Manage Them
We’ll start by defining test environments. What are those? Why do we need them? What’s so special about managing them?
First things first—let’s get the “what” out of the way. A test environment is a dedicated space for the execution of test cases. A testing environment is neither just software nor just hardware. Instead, it’s a full-fledged system, made up of a combination of software, hardware, and network that supports the creation, execution, and maintenance of tests.
When it comes to setting up a stable test environment, there isn’t a one-size-fits-all solution. You have to configure it according to the needs of the system under test (i.e., the application you’re testing.)
Why Is Test Environment Management so Crucial?
Since we’re done with the definition of the test environment, let’s now turn our attention to the motivations behind it. Are test environments important? If so, why?
It all comes down to the struggles technology organizations have trying to stay afloat in this highly-competitive scenario we currently live in. Not only does everything change at an amazingly fast speed, but the pace of change itself is accelerating.
Software organizations struggle trying to balance the need for producing high-quality software with the need to deliver their projects as quickly as possible to beat their competitors to market. A third and crucial factor that comes into play is money. The budget for any project is almost always lower than it’d need to be to ensure excellent results. When companies fail to increase the time allotted for the project and/or its budget—which, as a rule, is what ends up happening—what suffers the most is the quality. Designing, deploying, and maintaining a sound testing strategy quickly falls to the last position in the list of priorities.
So, up until now, we’ve established that the lack of a great testing approach is one of the main reasons why companies deliver poor quality applications. But what is the role of test environment management in all of this?
Well, one could argue that, when it comes to setting up a successful testing strategy, the test environment is the most crucial piece of the puzzle. If you don’t have a reliable test environment, all of your other efforts will be in vain. It won’t matter a bit if your QA team is exceptional, and your test cases are comprehensive and well written.
A “Negative” Test Environment Management Checklist
In the previous section, we’ve explained that:
- The lack of a robust testing strategy is one of the leading causes of the issues companies have with poor quality, and
- That having a test environment management process up and running is essential if you want to have a proper testing strategy.
But we’ve ended the section before covering the specific problems that a poorly-managed test environment can cause, so that’s what we’re doing now. I’ve jokingly called this section a “negative” checklist for test environment management, and that’s what it is—a list of common problems software organizations struggle with when they don’t have a proper test environment management process in place.
Test vs. Production Environments
We start with one of the most pervasive problems when it comes to testing environments. Namely, the test environment is too different from the production environment. The test environment should be as close a perfect copy of the production environment as possible. Still, often, they differ in many various factors—versions of operating systems and other types of software, several configurations, security privileges, software updates or patches, and many more. When these differences are relevant enough, that dramatically increases the likelihood of defects in the final product.
Production Data Replication
Another common problem has to do with obtaining data to feed the test cases. Companies will often replicate data from production, apply some primitive masking, then use such data for testing. This approach is inadequate for several reasons. First, it’s slow and error-prone. Additionally, it might be hard to ensure the availability of test data using the production replication technique. Last, but not least, there’s the risk of exposing or compromising client data, which can bring catastrophic financial and legal consequences, not to mention the damage to the brand’s reputation.
Lack of Accountability in Testing
Lack of accountability is also a serious and prevalent issue that stems from a poorly managed testing environment. Many software organizations, due to budget reasons, can’t afford to have a dedicated QA team, or at least a QA department that is part of the development team. So they attribute the QA responsibilities to developers, which increases the risk of lack of accountability since they’d have to balance the testing/QA tasks with their already existing workload.
Yet another common issue is the lack of proper bug tracking tools (or the poorly configuration of the ones that exist.) This might lead to bugs being assigned to the wrong individuals or teams, which can cause a delay in the solving of these issues. Delays of any kind means time and money wasted.
Finally, we couldn’t fail to mention testing in production is a common problem that exists due to a poor or inexistent test environment.
Test Environment Management Checklist to the Rescue
After you’ve taken a look at the negative checklist, it’s finally time to check out the positive one. Without further ado, here are our steps to ensure you have a good test environment management process in place.
Ensure Testing Start as Early as Possible
This one is less of a “to-do” item and more of a general best practice that you should be aware of all the time. Starting the testing activities late in the project is one of the most common and harmful testing anti-patterns. This leads to many problems, such as less time to create and configure the required test environments. Additionally, it may lead to defects being found too late in the process, which dramatically increases the cost of fixing them.
Verify If It’s Possible to Reuse (At Least Partially) an Existing Test Environment
Make no mistake—creating a great testing environment isn’t free. But that doesn’t mean you shouldn’t try to save money and other resources where it’s feasible to do so. Before committing to any large investment check to see if it is possible to reuse some existing infrastructure, at least partially. For example, you might have an existing test environment for a canceled project, and many of its components are fit for use in your current project. There would be no reason not to take advantage of that existing infrastructure.
Check That You Have All Required Hardware and Software
The next point is just to verify whether you already have the required hardware and software. That might include physical servers, network equipment, operating system licenses, and much more.
Decide on the Test Data Management Strategy
One of the crucial decisions of a testing strategy is the test data management strategy. How are you going to obtain data for the test cases? Replicating for production? Using a full-fledged test data management tool?
The organization needs to make this decision as early in the process as possible since it’s a blocking issue— without testing data, there can be no tests
Decide How to Ensure the Maintenance of the Test Environment
Designing and implementing the test environment is the short and “easy” part. Then comes the hard and indefinitely long-lasting part—maintenance.
When people hear the word “maintenance,” many think it only means fixing issues when they happen. While that certainly is an important part of any maintenance project, that’s not all there is to it.
In the case of a test environment, you must decide, among other things:
- Who’s accountable for the maintenance of the environment.
- Whether to use on-premises or cloud solutions to host the environment. You shouldn’t forget to factor in important factors like costs and backup strategies.
- How to ensure reliability (availability, performance, etc.) of the environment.
- How to ensure physical integrity in the case of on-premises solutions, when applicable.
Proper TEM Is a Path for Software Quality
A sound testing approach is essential for any company that wants to remain competitive in this current scenario that changes surreally fast. Unfortunately, many companies fail to implement a solid strategy, and often that’s due to either the lack of a test environment or the existence of a poorly-managed one.
Today’s post was our attempt at remedying this problem by featuring a checklist you can use to ensure that you have a test environment that is able to help you and your organization on your quest for high-quality results.
This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.
19 MARCH, 2020 by Michiel Mulders SRE vs DevOps: Friends or Foes? Nowadays, there’s a lack of clarity about the difference between site reliability engineering (SRE) and development and operations (DevOps). There’s definitely an overlap between the roles, even though...
06 MARCH, 2020 by Arnab Roy Chowdhury Top 10 SRE Practices Do you know what the key to a successful website is? Well, you’re probably going to say that it’s quality coding. However, today, there’s one more aspect that we should consider. That’s reliability. There are...
20 FEBRUARY, 2020 by Arnab Row Chowdhury Technically, the world today has advanced to a level we never could’ve imagined a few years ago. What do you think made it possible? We now understand complexities. And how do you think that became possible? Literacy! Since...
14 FEBRUARY, 2020 by Michiel Mulders A site reliability engineer loves optimizing inefficient processes but also needs coding skills. He or she must have a deep understanding of the software to optimize processes. Therefore, we can say an SRE contributes directly to...
07 February, 2020 by Arnab Roy Chowdhury Do you remember what Uncle Ben said to young Peter Parker? “With great power comes great responsibility.” The same applies to companies. At present, businesses hold a huge amount of data—not only the data of a company but also...
17 JANUARY, 2020 by Sylvia Fronczak Site reliability engineering (SRE) uses techniques and approaches from software engineering to tackle reliability problems with a team’s operations and a site’s infrastructure. Knowing the history of SRE and understanding which...