DevOps and TEM Go Hand in Glove

SEPTEMBER, 2019 by Mark Henke
DevOps is overall a healthy practice for most development teams, but it doesn’t come for free. Enterprises are eager to adopt the practice but their tools often lag behind DevOps practices. This is a bit like walking out into the winter cold with bare hands and being overwhelmed by Mother Nature’s bitter force. Like hands and gloves, DevOps and test environment management (TEM) are a practical pair. In this post, we’ll summarize what DevOps and TEM are, and with what specific practices they complement each other.  

DevOps: The Hands

DevOps is about marrying operations and development to achieve full software ownership. Dev teams are taking responsibility for code after it deploys into all of their environments. This gives us great power and autonomy. However, in an enterprise, we often have many dependencies—other software systems we rely on. And managing all of these environments gets increasingly difficult. In the current landscape, managing these dependencies with DevOps is a bit like walking into cold, wintery weather with no gloves on. Without proper tools, our poor, operations-soaked hands will get frostbite. This frostbite comes in the form of time we must spend on production support and operational activities. And this time steals away our capacity for new development and numbs us to the joy of writing code.

TEM: The Gloves

Test environment management (TEM) is the set of gloves we can put on to protect us from the bitter costs of supporting our software. This post is a good primer on why TEM is so important for DevOps. Here, we are going to cover a few specific cases where it’s valuable.

The Chilling Wind of Black-Box Testing

Black-box testing is a way to test systems without deep knowledge of how they work. It’s described as a black box because your dependencies are opaque to you. You know what goes in and what should come out, but you don’t care about what happens in between. Types of black-box tests include contract testing, end-to-end system testing, and smoke testing. Unlike unit tests, these verifications can’t run in isolation. It can be hard to set up certain data the way we need it or ensure a clean slate every time we run the test suite. Data from one team can collide with another team’s data. The environments we’re testing can also be unstable or unreliable, changing on a cadence that constantly disrupts the teams that use it. These sorts of tests are like a chilling wind that blows through your environments, causing flaky, inconsistent failures and false positives.  Without TEM, the cost of these tests can outweigh their benefits. When we treat test environments as first-class citizens of our software suite, we can mitigate these problems. When we can manage test environments explicitly, we can cohesively change them in sync so that teams know what to expect. We can script these environments so that our consumers can customize them or manage them directly. And we can partition our environments properly so that teams won’t collide with each other.

The Snow Drift of Performance Testing

Another area that can pile up on us without TEM is performance testing. It’s prudent to ensure that we can meet our service-level objectives, but we don’t control the full stack. It’s traditionally difficult to control or even understand the total performance you can provide. Your system is just one of many, each trying to meet customers’ latency needs. Not only is this tricky, but performance testing puts a lot of stress on systems. You can unintentionally stop another team from running their own suite. Or they may be deploying, causing your metrics to act abnormally. Test environment management ensures performance tests are cohesive. You can manage the timing and cadence of these test runs together and across systems. Or, you can self-service your dependencies and spin up the environments yourself for testing. We’ll dive into this later. TEM is a snow shovel you can use to clear your way to smooth metrics on your performance.

The Icy Road of Tracing and Logging

When practicing DevOps, the ability to quickly track down and fix weird, subtle problems is paramount to strong product support. This is a hard task with a flood of immature products out there that don’t quite enable us to do this. There are two main tools we have to combat this: logging and tracing. Tracing helps us pinpoint problem spots and logging helps us diagnose why those spots cause problems. But in a complex, distributed system there are often hurdles to effectively using these tools. Loggers may be inconsistent across applications. Or they may be stored in disparate places. Imagine getting a critical problem at 2 AM and having to click through and download half a dozen log files from four different places! Or imagine trying to walk through a request across seven different systems of which you only have a few admin dashboards to guide your way. These types of problems result in multi-hour phone calls with a dozen different people, all trying to stitch together an elaborate puzzle. The unknowable types of incidents that plague us on production support are like an icy road. We travel down it trying to reach our home, but keep sliding out of control and often end up in a ditch. The first step is for us to have consistent, aggregated logging across all applications. The second is to implement some distributed monitoring tooling. But how do we ensure these tools are doing a good job? Enter, once more, TEM. TEM will be the salt on our road, letting us see across systems how we can get these tools into place. We can also use our environment management to consistently bolt these things on. Then each application team need not worry about how to bring these tools to life.

The Inner Lining of Self-Service

In all these different areas where TEM warms our hands, a common theme emerges. There’s an inner lining in our gloves. Treating test environments as first-class concepts lets us move away from team hand-offs into the warmth of self-service. Traditionally, teams were very restrictive of their code. Over time they’ve become less restrictive, but many teams still hold onto their tools, their data, and their deployments. To get anything done, you would often need to schedule things in advance with your downstream teams. On the other side, you would need to hand off completed work for someone else to actually bring it to customers. Each hand-off increases the lead time for a feature and becomes a crack for potential miscommunication. These miscommunications develop into defects. With ideas like TEM, we realize that our environments are often as important as our code! We thus learn to package and automate it as much as we automate our application. Doing this opens up the door for us to easily share this with others. We can publish our test environments. With strong tooling, we can maintain these explicitly. Then developers are able to self-service the test environments they need for performance testing, black-box testing, and the like. Now developers who depend on our team don’t need to wait for us. They don’t need to hand something off to ensure it works end to end. By managing and publishing test environments, we can make features faster than ever with the highest quality.

Braving the DevOps Arctic With TEM

While DevOps gives us the autonomy to own our software, it can be a cold landscape. Many of the old tools are no longer sufficient. We need to warm ourselves with gloves that let us treat our test environments as first-class citizens. We’ll find that handling complex testing and monitoring scenarios with our dependencies will warm our entry into the cold. Sometimes, though, even gloves don’t provide enough warmth. Sometimes we need a space heater for our TEM. That’s where a tool like Enov8 can come in.
Mark Henke This post was written by Mark Henke. Mark has spent over 10 years architecting systems that talk to other systems, doing DevOps before it was cool, and matching software to its business function. Every developer is a leader of something on their team, and he wants to help them see that.

Relevant Articles

Self-Healing Applications

02NOVEMBER, 2022 by Sylvia Froncza Original March 11 2019An IT and Test Environment Perspective Traditionally, test environments have been difficult to manage. For one, data exists in unpredictable or unknown states. Additionally, various applications and services...

Data Operations: Defined and Explained

01NOVEMBER, 2022 by Justin Reynolds.Businesses across the board are spinning their tires when it comes to data and analytics, with many of them failing to unlock maximum value from their investments. According to one study, 89% of companies face challenges around how...

Software Security Anti-Patterns

02NOVEMBER, 2022 by Eric Boersma *Original 22 October 2019If you're like a lot of developers, you might not think much about software security. Sure, you hash your users' passwords before they're stored in your database. You don't return sensitive information in error...

What makes a good Test Environment Manager?

14 OCTOBER 2022 by Daniel de OliveiraIn today’s application-based world, companies are releasing more applications than ever before. Software delivery life cycles are becoming more complicated. As a result, large companies require hundreds and even thousands of test...

Staging Server Success: The Essential Guide To Setup and Use

01NOVEMBER, 2022 by EricStaging Server Success: The Essential Guide To Setup and Use Release issues happen.  Maybe it's a new regression you didn't catch in QA. Sometimes it's a failed deploy. Or, it might even be an unexpected hardware conflict.  How do you catch...

What makes a good Test Data Manager?

19 NOVEMBER, 2020 by Michiel Mulders What Makes a Good Test Data Manager? Have you implemented test data management at your organization? It will surely benefit you if your organization processes critical or sensitive business data. The importance of test data is...