Are You TEM Savvy?
by Erik Dietrich
Measuring TEM Capability in Your Enterprise
Once upon a time, testing your software was easy. Or, at least, relatively speaking, it was.
Your team would write some code, tag an unreleased version of the software, build it, and hand it over to QA to do their worst. If this was, say, 20 years ago, this process probably involved creating a CD for QA to use, since this was almost certainly some kind of desktop application. And, to make life easy, the developers, testers, and end-users were all using Windows NT.
You could thus manage the testing process in pretty straightforward fashion:
- Run the software.
- Ask yourself, “does it do the thing that it’s supposed to do?”
And, basically, that was it.
Of course, this is a bit of an oversimplification. But it does serve to illustrate how complex things have become over the years.
These days, when testing software, you need to worry about a dizzying array of things. What device, OS, and browser combination does the end user have? What happens when users swamp the app at scale? How does the system respond to the various services it uses degrading or going down? The list goes on and on.
Testing software in today’s enterprise has so much complexity, that it requires an entire environment management approach just to keep up. Let’s take a look today at how you can measure and evaluate your capabilities in this regard.
What Is Test Environment Management (TEM)?
Before we get too far, however, it’s worth some definition housekeeping. What, exactly, do we mean when we talk about test environment management (TEM)?
Well, a test environment is, quite simply, the setup you have for conducting testing and quality assurance of the software that you’re building. In the introductory example, the QA department would have used a simple Windows NT machine with the software installed as its test environment. In today’s world of cloud computing and heterogeneous environments, this probably takes the form of one or more servers, containers, or even more exotic mixes.
Test environment management, then, is the administration of the test environment. Administrative activities include populating the environment with application data, configuring application settings, installing any necessary companion software, and generally simulating all aspects of the environment related to testing. In short, TEM is your strategy for recreating production-like conditions for the sake of testing.
What Is Your TEM Capability?
TEM is a relatively complex subject, because it quite likely means having multiple environments in your back pocket.
You might have different environments for different team members or entire QA teams. Or, perhaps, you have different environments based on considerations such as varied production scenarios or different user contexts. And, even if you only have a single test environment, TEM is critical because staff will need to quickly configure that single environment for all required testing scenarios.
There’s no way around it. Test environment management is a complicated concern these days.
So when we talk about TEM capability, what do we mean?
Well, as you can, no doubt, infer, TEM capability is essentially how good you are at bringing test environments to bear as you need them. How well does your organization respond to testing needs as they arise? Does it anticipate and schedule them in advance? How mature are you, when it comes to TEM?
Increasingly, sophisticated TEM capability is becoming table stakes for enterprises to remain competitive.
How Can You Measure and Evaluate TEM Capability?
Since TEM capability is so important, organizations naturally look to understand how to self-assess and measure their own progress. In fact, Enov8 has, in the past, offered some helpful metrics along these lines. Here, we’ll build on those metrics to create a series of questions you can ask yourself to ascertain how TEM-savvy your organization is.
1. What Is the Time Required to Get a Suitable Environment Operational?
Let’s start with an easy one—easy to articulate and easy to measure. How long does it take to have a suitable test environment up and running?
Let’s say that a dev team commits a candidate feature or software increment, and the definition of done requires a few different flavors of testing: functional testing, shakeout testing, and stress testing. How long does it take to get an environment ready for that scenario?
If the environment is ready as soon as people are ready to start testing, you’re in great shape. If, on the other hand, this requires a good bit of labor or waiting for any reason, you’ve got work to do.
2. How Much (or Little) Tribal Knowledge Drives Your TEM?
Setup speed is an important consideration, but so is setup knowledge. Specifically, the knowledge of how to configure, select, and spin-up environments is crucial when it comes to your TEM savvy.
When it comes time to implement a particular testing scenario, what happens? Do you have a runbook that anyone can refer to for the purposes of environment setup? Or, better yet, do you have some kind of push-button automation?
Or do you have Bill? Bill is the configuration manager, and when you need an environment, you email Bill. Then, magic happens, and you have your test environment.
If TEM resides inside the head of folks in the organization, you’re incurring a lot of risk. If, on the other hand, it’s well documented or automated, your organization is doing well.
3. Is Your Test Data Realistic (Without Being Too Real)?
We haven’t yet talked about test data, but test data is an absolutely crucial component of a good TEM strategy. It impacts every kind of quality assurance your organization can possibly perform.
You want realistic data. The form and volume of it should be realistic, in order to properly demonstrate how the application will behave in production. Testing an e-commerce platform on 2 test customers, for instance, isn’t going to cut it.
You also want individual fields within your data to represent reality. You’ll have much more luck shaking out subtle problems when your sample last names have apostrophes, hyphens, spaces, and other exotic characters in them than you will if you have 250,000 Bill Smiths.
But, you can take this too far. You don’t want actual user data in your test environments. This can run afoul of PII regulatory concerns and serve as a needless vector for potential exploits.
So ask yourself whether or not your data is completely representative of reality without being reality.
4. How Much Configuration Reuse Do You Have?
Here’s a concern closely related to the idea of standardization versus tribal knowledge. When it comes to your configuration, how much reuse to you employ?
If, for instance, you have two environmental configurations that are largely identical, but with a bit of variance in the data and application configuration, what does that look like? Do you have two completely different scripts to generate the environments, with a lot of duplication? Or two different runbooks with duplication? Maybe you just have Bill executing two sets of largely repetitive tasks?
Or do you have a scheme in place where you execute the same script, varying only for the details at the end?
If you’re breaking the environment setup into conceptual components and assembling those components as part of your TEM, you’re going to incur much less overhead managing the environments. If not, you’re going to waste a lot of time and effort.
5. Are You Making Use of Infrastructure as Code?
The last TEM-savvy indicator that I’ll discuss is the use of infrastructure as code (IaC). This measure of capability dovetails heavily with the last one, in that IaC is a strong driver of component reuse.
IaC is a model where you describe hardware and application configuration using a declarative style of code, often something like YAML. Without going too far into the weeds, IaC is a powerful way to track configuration information, complete with capabilities such as version control and comparison.
If you’re using IaC to help with your TEM, you’re almost certainly scoring well on some of the questions I’ve already suggested for self-assessment. IaC forces your hand when it comes to scripting, reusing, and tracking configuration. And, with all of this in place, you’re doing really well.
If you haven’t started to make use of IaC yet, don’t despair, since this is an extremely sophisticated technique. But it definitely gives you an important milestone to strive for.
Like It or Not, TEM Is Part of Your Quality Assurance & DevOps Strategy
You now have 5 questions to ask yourself in assessing how TEM-savvy you are, and I encourage you to make use of them. There’s really no downside to improving your maturity here.
But if you’re wondering how important this really is, you should consider the cost of lagging behind in TEM capability. All of the time that your organization spends cobbling environments together by hand, duplicating effort and manually inputting half-baked data is time that it isn’t spending testing your software. You will pay the piper in the end.
And, as a result of this, you have to accept that TEM is part of your software’s quality assurance strategy and thus part of its quality (or lack thereof). If you want to produce high quality software, you can’t afford to remain indifferent to TEM.
This post was written by Erik Dietrich. Erik is a veteran of the software world and has occupied just about every position in it: developer, architect, manager, CIO, and, eventually, independent management and strategy consultant. This breadth of experience has allowed him to speak to all industry personas and to write several books and countless blog posts on dozens of sites.
We often get asked by people “What is TEM (Test Environment Management), well for those of you looking for a quick overview of Test Environment Management, here is Use Case we developed as a way…
19 MARCH, 2020 by Michiel Mulders SRE vs DevOps: Friends or Foes? Nowadays, there’s a lack of clarity about the difference between site reliability engineering (SRE) and development and operations (DevOps). There’s definitely an overlap between the roles, even though...
06 MARCH, 2020 by Arnab Roy Chowdhury Top 10 SRE Practices Do you know what the key to a successful website is? Well, you’re probably going to say that it’s quality coding. However, today, there’s one more aspect that we should consider. That’s reliability. There are...
20 FEBRUARY, 2020 by Arnab Row Chowdhury Technically, the world today has advanced to a level we never could’ve imagined a few years ago. What do you think made it possible? We now understand complexities. And how do you think that became possible? Literacy! Since...
14 FEBRUARY, 2020 by Michiel Mulders A site reliability engineer loves optimizing inefficient processes but also needs coding skills. He or she must have a deep understanding of the software to optimize processes. Therefore, we can say an SRE contributes directly to...
07 February, 2020 by Arnab Roy Chowdhury Do you remember what Uncle Ben said to young Peter Parker? “With great power comes great responsibility.” The same applies to companies. At present, businesses hold a huge amount of data—not only the data of a company but also...