erm-best-practices

Enterprise Release Management Best Practices

14

JUNE, 2021

by Kathrin Paschen

Managing releases at scale is daunting. It involves juggling dependencies, timelines, and requirements. The stakes can be pretty high, too. Not all failures are as spectacular as crashing a lunar lander or losing $440 million. Even so, investing in enterprise release management makes a lot of sense.

Enterprise release management orchestrates the release processes of several projects and enables you to launch or update large software systems. When it’s done well, it looks easy (though you can be sure people are working hard to make it so). When it goes bad, it feels like a nightmare.

This post provides practical tips for how to manage releases at enterprise scale, successfully.

What is Enterprise Release Management?

The aim of a software release is to achieve the following outcomes at the minimum:

  • The update/new feature should work as expected.
  • The release should not break the existing state of the product/application.

Release Management is the process that helps achieve the above outcomes. It helps you plan and execute the release lifecycle such that teams can efficiently deliver releases without affecting the success of the release. The success of a release can be guaged by the functionality status of the update and its effect on the existing state of the product.

As an organization grows, multiple departments and projects take birth. Various teams work on these projects. With the approach of continuous delivery coming from different projects, it becomes difficult for a release to be successful. Enterprise release management (ERM) is the practice of planning and managing releases such that software delivery from multiple projects is a success.

What Does Success Look Like?

We’ve all seen releases that went poorly, and we can recognize some of the red flags that a release management system is failing. But what are your criteria for a good release?

Usually, the answer to this is “a good release is one where we launch on time with no big problems.” But there’s often a trade-off between speed and quality. Also, maybe you also want to include “the new feature makes our customers happy” and “nobody had to work a 100-hour week” in your criteria.

There’s a difference between a release and a feature launch. A feature launch is when a new feature becomes available to customers, but a release just means you’ve set some new code live. You can (and probably should) release regularly and often, no matter the size of your company. Small, frequent releases are great for quick feedback and innovation, and they make large feature launches easier.

Different companies choose different trade-offs:

  • Startups often want to launch quickly to maintain momentum. Therefore, they may be willing to take risks when it comes to releasing code with a few known bugs.
  • Big companies, on the other hand, often require new functionality to integrate well into their existing offerings. They may have legacy systems. Also, they tend to be reluctant to expose customers to bugs or downtime.

The main difference between startups and larger companies is in the amount of coordination—releases at larger companies typically require more of it.

Enterprise Release Management is a useful approach to make sure releases are efficient while maintaining the integrity of the release

Importance of Enterprise Release Management

Enterprise Release Management is a useful approach to make sure releases are efficient while maintaining the integrity of the release. For a small organization with only a couple of release teams, ERM won’t be a major differentiator. But think about a large-scale organization with various departments working on several projects. And for every software release, there can be multiple teams involved such as development, testing, and operations. It’s important that these teams are in good coordination for the success of the release.

ERM is important to have various teams coordinate with each other and to organize releases to be a success. When there is coordination and the lifecycle of a release is planned well, the release process becomes more efficient. Although ERM has a major effect on big companies, small companies can also utilize it to improve their processes. Below, I describe best practices that help in different kinds of environments, from startups to big companies.

Best Practice 1: Plan and Communicate

When coordinating project release schedules toward a feature launch, you often need to plan backward from a target date and align the work so everything is ready on time. Even when you’re not working toward a feature launch, you’ll have project dependencies.

This means you need an overview of…

  • the projects that contribute to the release
  • the features that they need to deliver
  • how those depend on each other

For example, let’s say one project provides a feature that others need. In that case, the first project needs to deliver that feature early enough for integration and testing. Similarly, if two projects access the same database tables, then schema changes have to be coordinated between them.

A diagram of these dependencies is essential; software that helps you track them is even better.

The next step is to define a timeline (a release calendar is useful here) that says which project needs to deliver which feature by which time. Include time for testing and for fixing bugs, and leave some slack because you always need slack.

Almost as important as having a timeline is making sure every project team knows about the release calendar and has agreed to it. If a project team pushes back, be sure to listen and address their concerns.

Best Practice 2: Manage Dependencies

The more dependencies there are between the projects in your release, the higher the risk of delays and late-breaking bugs. Some dependencies are unavoidable; you track them and reserve time for integration testing.

Whenever possible, encourage projects to reduce their dependencies. Ideally, project teams create self-contained systems that communicate with each other through clear application programming interfaces. This decouples project teams and allows them to develop and test against mocks or fakes. It also reduces long lead times and blocked user stories—two of the red flags mentioned above.

I find it useful when each project team creates tested and versioned packages of their software that other projects can use. Why bother doing this?

  • It helps the other projects test against a versioned package. Most package repositories also allow you to label packages. For example, you might use the labels “Ready for testing” or “Use this only if you want to reproduce bug #123.”
  • It helps testers reproduce bugs. If bug reports contain package versions, then the testers can be reasonably sure that they’re testing the right code.
  • It helps you track progress. Most task or issue tracking systems allow you to record which software versions include a feature. You can then track whether a package with a feature exists, whether someone has tested it, how many bugs they found, and so on.

Now you can do a fair amount of integration testing early, avoiding the risks around “big bang” integration. That type of integration happens when people on individual projects write their software in isolation and then put everything together late in the process.

What’s next?

Best Practice 3: Pre-Production Environment is Important

You’ve probably heard or seen cases where everything went smooth in the pre-production environment but the application broke in the production environment. This can happen due to several reasons. But one of the ways to reduce this situation is to have a quality pre-production environment. Continuous delivery is a priority but having a stable release should be a higher priority. Thorough testing in pre-production helps identify issues in the release that might be an expensive failure in production. Therefore, you can address these issues faster and before they cause harm.

Best Practice 4: Remember That Tools and Automation Are Your Friends

Wherever possible, automate steps, such as building, testing, and deployment. This saves you time and reduces the opportunity for errors. Excellent tools exist.

What if a step is too hard to automate? Document it. People who aren’t on the project team have to be able to build, test, and package a new version. This is essential if you ever need to deploy a security update quickly, for example.

If at all possible, deploy regularly. A few projects can’t do this—for example, those whose software gets flashed onto ROMs. However, even if you can’t deploy to your end users regularly, deploy to a test environment. Benefits include:

  1. You exercise your deployment tools regularly. They become familiar, and you can fix issues with them.
  2. You find issues that don’t show up on developer machines. Maybe the code makes assumptions about directory structure? About installed libraries? Maybe the code assumes the database runs on the same host?
  3. You need to run performance tests at some point, and it makes sense to run them on a test environment similar to your production environment. Can the system achieve the necessary throughput? How does it behave under high load? Does it degrade gracefully, or does it go into cascading failure?

For big launches, I recommend hiding the new code behind a feature flag. This cuts down on branch maintenance. Feature flags also provide you with a built-in OFF switch in case you need to back out of the launch.

In addition to automating your releases, I recommend automating policy and compliance checks, even security tests. This is no replacement for audits, but it helps.

Creating standard operating procedures (SOP) saves you the hassle of starting everything from square one

Best Practice 5: Standard Operating Procedures and Priorities

One of the major difficulties in enterprise release management is handling several concurrent in-process projects and their releases. With multiple departments, projects, and releases, it’s very difficult for release management teams to strategize every release from scratch. Creating standard operating procedures (SOP) saves you the hassle of starting everything from square one. These SOPs can include standard provisional timelines, processes for managing dependencies, coordinating with different teams, and anything that is in use regularly. And because you’d have a raw strategy ready, all of you have to do is update it as per specific requirements.

Getting all updates developed and released at once is not practically possible. When you have requests from several teams, it’s always good to prioritize. Talk to the stakeholder to understand the effect of a release they’ve requested. This helps you plan the release process such that the important things are taken care of first and the teams are also not exhausted from overload.

Best Practice 6: Track Project Status

If you’ve ever been involved in an enterprise release, you will have attended status meetings. They tend to be dreadful. If you have a way to track at least some aspects of the various projects’ progress without making people sit in a meeting, you can free up a lot of time and avoid people getting grumpy. Instead, you can focus on important points:

  1. Which projects aren’t on track?
  2. What’s holding them back?
  3. Who can help and how?

Identifying risks early provides more time for mitigation. Moreover, you can now replace the large, time-consuming status meeting with smaller meetings focused on solving problems.

You need to have a way for projects to publish their status. How much of the work is complete? How are they doing on testing? There should be a way for projects to update their status, either manually or automatically. Also, there should be a way to see project status on a dashboard. Best practices 1, 2, and 3 provide you with tools that make this easier.

Best Practice 7: Observe and Improve

This topic overlaps, not surprisingly, with site reliability engineering best practices. You need to notice when something isn’t working well and then do something about it.
Deployments and infrastructure changes are a leading cause of IT system failures. It makes sense to observe your systems, both in your test environment and in production. Use monitoring tools.

Usually, when an issue shows up shortly after a release, people will blame the release. They’ll often be correct, but it’s good to make sure because the mitigation strategy depends on it. If you have good monitoring data and a record of which software changes went into which version, you stand a much better chance of pinpointing the cause of a regression.

What if you decide the release is to blame? You have three options.

  1. Roll back the release.
  2. Fix the issue in production.
  3. Leave things as they are.

Depending on your systems, not all of these options may be workable.
With systems that do frequent small releases, rolling back is often possible. Ideally, your projects are sufficiently independent of each other that you can roll back just some of them.

Sometimes, rollback isn’t an option. Maybe you’ve got database schema changes that can’t be undone. Maybe the release is a highly visible feature launch. The risk evaluation that goes into deciding between “fix in production” and “leave it as it is” is beyond the scope of this post. At that point, an enterprise release manager can only advise on how to test and roll out a fix quickly but responsibly.

Monitoring can also tell you whether and how customers are using your features. This is valuable feedback and addresses another release management red flag, as mentioned earlier in the post.

Summing Up: Experts Can Help You Implement Best Practices

Enterprise release management spans projects and disciplines. It requires skills that range from project management to engineering. Also, it includes a lot of cross-team communication.

All of this may seem daunting! But with the right set of tools and best practices, it’s entirely feasible. Enov8 offers an enterprise release management tool to help you implement best practices. Moreover, its specialists are experienced at guiding enterprises through the adoption of release management best practices. Here’s how you can learn more.

 

Kathrin Paschen

This post was written by Kathrin Paschen. Kathrin is a freelance SRE interested in capacity planning, cost estimation, and monitoring. After a long time at Google, she now has her own small company focused on helping clients use the cloud. She likes figuring out scaling bottlenecks and resource models for cloud architectures.

Relevant Articles

What Makes a Good Enterprise Release Manager?

09 SEPTEMBER, 2022 by Michiel MuldersDo you want your company to scale efficiently? Look for an enterprise release manager (ERM). An ERM protects and manages the movements of releases in multiple environments. This includes build, test, and production environments....

The Pros and Cons of Test Data Synthetics (or Data Fabrication)

22 August, 2022 by Louay Hazami *Update from October 2020Data privacy is one of the most pressing issues in the new digital era. Data holds so much value for normal internet users and for all types of companies that are looking to capitalize on this new resource. To...

What Is Data Fabrication in TDM

16August, 2022 by Carlos Schults *Update from 15 Mar 2021In today's post, we'll answer what looks like a simple question: what is data fabrication in TDM? That's such an unimposing question, but it contains a lot for us to unpack.What is TDM to begin with? Isn't data...

Your Essential Test Environment Management Checklist

08August, 2022 by Carlos Schults *Update from 26 Nov 2019.Your Essential TEM Checklist   "Test Environment Management Checklist." Yep, that sounds like a mouthful, but don't let that discourage you. The idea here is quite simple—adopting a checklist to evaluate...

Sand Castles and DevOps at Scale

03JUNE, 2022 by Niall Crawford & Carlos "Kami" Maldonado. Modified by Eric Goebelbecker.DevOps at scale is what we call the process of implementing DevOps culture at big, structured companies. Although the DevOps term was back in 2009, most organizations still...

Test Environment Management Explained

3JUNE, 2022 by Erik Dietrich, Ukpai Ugochi, and Jane Temov. Modified by Eric GoebelbeckerMost companies spend between 45%-55% of their IT budget on non-production activities like  Training, Development & Testing and lose 20-40% of productivity across their...