SRE vs Disaster Recovery – (Friends or Foe)

JUNE, 2020 by Eric Boersma

Site reliability engineering versus disaster recovery: Every company needs a disaster recovery plan. This is just a simple fact of life. Your company needs to know how to recover when something breaks or you can’t get access to something you need.

In larger, more advanced tech companies, disaster recovery sometimes bumps against environment management teams. This is the sort of thing that can lead to conflict between the teams. For instance, when the disaster recovery team bumps against the site reliability engineering team, which team takes responsibility for what?

In moments like these, it can feel like disaster recovery and site reliability engineering are foes. In some companies, they stay that way. But in my experience, the healthiest companies don’t think of disaster recovery or site reliability engineering as being foes at all. They find that they work best together, each working toward a goal of making the business better.

In this post, we’ll talk about how that happens and how you can make it happen at your company.


What Is Disaster Recovery?

Before we get started, it’s good to understand what we’re talking about. Disaster recovery (DR) is the process of planning and executing in scenarios where there’s been a “disaster.”

Originally, this term meant something significant, like a flood or a fire or an earthquake. Businesses would make plans about how to handle these major events. Then, when they did happen, they’d put those plans into action and follow them to get the business back to normal.

In today’s climates, “disaster” can mean something a little less significant. A disaster in today’s tech world can be as small as a critical server suffering a hardware failure. However, it’s still up to the business to have plans in place for events like that and put them into action when they happen.

What Is Site Reliability Engineering?

Site reliability engineering (SRE) can feel a little difficult to define. This is because, as a concept, it’s still pretty young. People are still figuring out the best ways to do it.

Basically, it’s a concept where companies build software to automate building and scaling their technical infrastructure. Instead of systems administrators building and managing individual servers by hand, site reliability engineering expects automation. Effective SRE teams define their environments in code, then use tools to automate their deployment and manage their state. Really effective teams build that automation directly into their release management process. They also use SRE principles to build out test environments, too.

How Can SRE and DR Work Together?

At first glance, it can seem like DR and SRE will always be at odds. When you really break it down, DR is concerned with what happens when things fall down. SRE is concerned with making sure that things keep standing up. Those don’t feel like two groups that will ever overlap, and at some companies, they don’t.

But if you reframe how you think of DR and SRE, it makes a little more sense. At a higher level, both teams are concerned with how to keep the company in an operational state. When you think about it that way, SRE and DR are actually close friends. They’re both working for the same goal, and they have a lot to offer one another.

Configuration Management: SRE Helping DR

At their core, SRE and DR are both focused on defining a valid state for the business, then taking steps until the business achieves that state. In SRE, this is known as configuration management.

An effective configuration management system goes a long way toward simplifying DR plans. What happens when that server crashes and can’t come back up? In a world dominated by system administrators, someone’s pager goes off. They have to drive to the data center and begin the arduous process of bringing a new server back up. That’s disaster recovery.

On a team with quality SRE practices, what happens when that server crashes? The automated server management tool provisions a new server on the cloud server provider. Because its entire configuration is well-managed, it already knows where all the other services on the network run. Nobody’s pager goes off. The entire system is down less than ten minutes.

This kind of process scales out, too. The best configuration management tools monitor the status of entire data centers. If Amazon or Microsoft or Google have a data center that goes offline, your configuration management software will automatically switch to the failover data center. There’s no “disaster” anymore. Things are down for a little while, but the business moves back to a working state as quickly as possible.

Outage Rehearsal: DR Helping SRE

One of the major flaws of most disaster recovery plans is that they’re never put into practice. Obviously, for the business, this seems like good news. Nobody wants to deal with a disaster! But it’s only good news as long as there’s never a disaster.

A DR plan that hasn’t been put into practice isn’t an effective plan. If you haven’t actually tested bringing the system back up, you can’t know it’ll work.

Obviously, that kind of outage is difficult to test. Companies are scared of bringing down critical services, even on purpose. That’s understandable!

However, at companies with quality SRE practices, bringing down critical systems carries far less risk. In the most mature environments, it’s possible to set up entire copies of your production environment. You don’t have to bring down servers that are serving customers.

The goal of bringing these services down is to allow your teams, both SRE and DR, to practice identifying and restoring those services. People don’t learn how to remediate outages by reading slide decks and watching videos. They learn by doing.

An effective SRE implementation means that it’s possible to turn off critical parts of the infrastructure easily. Then, it’s up to the team to identify the outage and figure out how to remediate the problem. This kind of real-world practice is invaluable in an actual emergency.

Savvy readers will notice that this sounds a lot like chaos engineering. They drive at similar aims. The goal is to build resilient, robust systems that work even when things are falling apart. Both work by breaking things on purpose, then figuring out how to fix them so you know better next time. The bonus here is that an effective DR team will make things a lot easier for the SRE team. They’ll be able to guide the effort to fix things through their pre-existing plans.

SRE and DR Don’t Have to Be at Odds

Like we’ve mentioned, SRE and DR teams wind up at odds in a lot of organizations. This can happen for many reasons, and sometimes those reasons can’t be overcome. That’s the unfortunate reality of working in business; no organization is perfect.

However, if you’re striving to make your organization as effective as possible, you should aim to help SRE and DR teams work together. As we’ve shown here, it’s entirely possible for these teams to serve the same purpose. In the best environments, they help each other work more effectively at their ultimate goals.

The key to the effort is to make sure that each team focuses on their ultimate goal: a healthy, effective business that’s operating efficiently. When that’s the case, everyone wins, and SRE and DR teams are friends to the end.

Eric Boersma

This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.

Relevant Articles

Test Environment Management – Modelling

30JANUARY, 2023 by Jane TemovTest Environment Management (TEM) is an essential process for ensuring the stability and consistency of the testing environment. It includes activities such as setting up the environment, monitoring and controlling the environment, and...

What is Deployment Planning?

15DECEMBER, 2022 by Jane TemovDeployment planning is the process of creating a plan for the successful deployment of a new software or system. It involves identifying the resources, tasks, and timeline needed to ensure that the deployment is successful. Deployment...

Why CICD & Test Environment Management Goes Hand-in-Hand

12DECEMBER, 2022 by Jane TemovWhy CICD & TEM Goes Hand-in-Hand Continuous Integration/Continuous Delivery (CICD) and Test Environment Management are two essential components of a successful software development process. CICD enables teams to deploy new code...

Enov8 Releases their Latest “Evaluation Edition”

08DECEMBER, 2022 by Enov8Enov8 is happy to announce the latest “evaluation”* edition is ready for consumption. *A complete Release & Environment Management product with a full license for 3 months. Our Release & Environment Management solution is designed to...

The Agile Release Train Explained

04DEC, 2022 by Jane TemovIf your organization is starting an agile transformation, you might be looking at it as an opportunity. Or perhaps you’re looking at it with some healthy skepticism.  Either is understandable—or even both at the same time. The opportunity...

Self-Healing Applications

02NOVEMBER, 2022 by Sylvia Froncza Original March 11 2019An IT and Test Environment Perspective Traditionally, test environments have been difficult to manage. For one, data exists in unpredictable or unknown states. Additionally, various applications and services...