Enterprise Release Management (ERM) is a set of end-to-end practices that enable large organizations to effectively manage software releases. ERM is uniquely designed for the challenges of multiple teams building and releasing software simultaneously. ERM establishes a framework that ensures organizations release software changes in a controlled and safe manner, minimizing disruption to normal business operations.

This blog post aims to provide a foundational understanding of ERM, its principles, and its role in facilitating efficient software delivery within large organizations.

What Is Enterprise Release Management?

Enterprise Release Management (ERM) is a framework for planning, building, testing, deploying, and monitoring software releases. ERM is specifically designed for large organizations with many stakeholders and complex dependencies. ​​ERM takes a holistic view of release management, coordinating a portfolio of releases across multiple teams and applications to minimize disruption and ensure everything works together smoothly.

While ERM implementations vary between organizations, ERM typically involves IT governance, configuration management, portfolio management, test-driven development, and project management. ERM often incorporates other frameworks such as Scaled Agile Framework (SAFe), DevOps, continuous delivery, and Release Trains.

By creating and maintaining an Enterprise Release Management strategy, companies can ensure that software teams create software that aligns with the business’s overall objectives, timelines, and resources.

Enterprise Release Management vs. Release Management

Release management is the general term for the process of building, testing, deploying, and monitoring a software release. It typically focuses on a single project or application within a development team.

Some key differences between ERM and release management:

• Scope: ERM considers not just individual projects, but the entire IT portfolio of a company.
• Coordination: ERM requires collaboration across multiple departments, like development, QA, operations, and business stakeholders.
• Risk Management: ERM places a stronger emphasis on risk management due to the complexity of coordinating multiple releases.
• Process: ERM typically relies more heavily on formal processes. For example, ERM typically involves a standardized change management process to ensure smooth transitions and minimize disruptions across the organization.
• Dependencies: ERM goes beyond individual projects, managing how releases rely on each other. It identifies and plans for these dependencies to ensure a smooth, sequenced rollout.

Key Terms in Enterprise Release Management

Before diving into the specifics of Enterprise Release Management (ERM), let’s establish a common ground by defining some key terms you’ll encounter throughout this post. Understanding these core concepts will provide a solid foundation for grasping the functionalities and benefits of ERM within large organizations.

• Enterprise Release Portfolio: The collection of all planned software releases an organization is considering or actively working on at a given time.
• Enterprise Release Plan: An enterprise release plan is a comprehensive roadmap which outlines the sequence and timing of software releases across an organization, considering dependencies and aligning with business objectives.
• Enterprise Release Governance Strategy: Defines the guiding principles and practices for managing the entire ERM process within an organization. It sets the framework for how software releases are planned, prioritized, executed, and controlled.
• Enterprise Release Manager: The enterprise release manager is a specific role overseeing the entire release management process, implementing strategies and best practices, managing the release calendar, and prioritizing releases based on business objectives.
• Release Management Systems (RMS): Software tools designed to centralize and automate tasks within the software release process.

Enterprise Release Management Roles and Responsibilities

ERM involves a coordinated effort across various roles. Here’s a breakdown of some key ERM roles and responsibilities:

• Enterprise Release Manager: A role responsible for coordinating so that the right changes are made to the right systems at the right time in a controlled manner.
• Business Stakeholders: Individuals or groups within an organization who have a stake in the success of a software release, but who are not directly involved in the development process itself. Includes CEOs, CFOs, and corporate strategy, as well as business analysts.
• Software Development Teams: Develop the new features and functionalities targeted for a release. Teams include developers/engineers as well as team managers/project managers.
• Test Teams: Design and execute various testing activities (functional, performance, security) to identify bugs and ensure software quality. Includes test environment managers.
• IT Operations: Prepare the production environment for the deployment of new software releases. Includes deployment, change, and configuration managers.

Enterprise Release Management Tools

Ensuring a smooth and successful software release process requires managing a complex web of tasks and dependencies. Release Management Systems (RMS) such as Enov8’s Enterprise Release Manager are tools which offer functionalities to enhance collaboration, standardize and automate tasks, and improve release governance such as:

• Release Planning and Tracking: Tracking the entire release journey, from initial planning through testing and deployment. This centralized view allows for proactive identification of potential issues and facilitates progress monitoring.
• Deployment Planning and Execution: Creation of detailed deployment plans, outlining the steps involved in rolling out the new software version to production.
• Integration with Existing Processes: RMS tools can integrate seamlessly with existing IT Service Management (ITSM) processes, ensuring adherence to incident and change management procedures during releases.
• Version Tracking and Environment Management: RMS tools can track different software versions across various test environments, components, and microservices. Additionally, they can help identify discrepancies (test environment drift) between test environments and production, minimizing the risk of deployment failures.
• Orchestration of Workflows and Integration with External Tools: An RMS can act as a central hub, coordinating tasks and data flow between various tools within the development ecosystem. This includes alignment of deployment tools, ticketing systems, and CI/CD pipelines, fostering a more streamlined and automated release process.

By implementing a Release Management System, organizations can achieve greater control over the software release process. Improved visibility, standardized workflows, and automated tasks contribute to a more efficient and risk-mitigated development environment.

The Enterprise Release Management Cycle

The Enterprise Release Management (ERM) cycle is a structured, multi-stage process that governs how software changes are planned, built, tested, deployed, and monitored within a large organization. The specific stages and practices within the ERM cycle can vary depending on an organization’s size, development methodology (e.g., agile, waterfall), and the complexity of the software being released. Some companies practice short release cycles, whereas others can have cycles spanning months or years. Some steps in ERM implementations typically include:

1. Defining business objectives: Key stakeholders such as Chief Experience Officers (CxOs), business executives, and strategists work to outline the high-level goals that the organization wants to achieve.
2. Creating an Enterprise Release Plan: This stage involves the enterprise release manager defining the release strategy, including the features and functionalities targeted for the release.
3. Breaking down objectives: This step involves project managers and release teams identifying the specific projects or work streams that need to be completed to deliver the functionalities and features that will ultimately contribute to the broader business objectives.
4. Establishing key milestones: Project managers and release teams work to define key milestones, which act as checkpoints within the overall release plan representing significant events or deliverables that need to be achieved at specific points in time.
5. Establishing gates: Release teams and project managers work together to define gates, which are specific checkpoints with defined criteria that projects must pass before progressing to the next stage.
6. Resource management: Release teams, project managers, test environment managers, and system owners must coordinate to identify key resources and ensure they will be ready when needed.
7. Development and testing: Development teams work on building and testing the new features and functionalities that will be released. Includes solution integration testing and user acceptance testing. Considering the prevalence of packaged solutions and outsourced development in enterprises, some ERM experts advocate renaming this step “delivery” to better reflect the broader scope. This acknowledges that not all projects involve internal development teams.
8. Building for release: The release is prepped for deployment as a build, typically streamlined via standardizing and automating build activities such as packaging, configuration, testing, and deployment.
9. Deployment: The software is carefully deployed to production environments, often following a phased approach to minimize risk.
10. Monitoring: After deployment, the released software is closely monitored for performance, stability, and user feedback. This stage involves bug fixing, addressing user issues, and potentially rolling back the release if critical problems arise.

The Benefits of Enterprise Release Management

Sticking to an Enterprise Release Management schedule can lead to a variety of benefits for an organization such as:

• Improved collaboration and communication: ERM fosters better communication and collaboration between different teams involved in the software development life cycle. This ensures everyone is on the same page and working toward a common goal.
• Increased customer satisfaction: By delivering high-quality software with fewer bugs and downtime, ERM helps improve customer satisfaction. Users receive a more reliable and consistent experience.
• Enhanced resource visibility and control: ERM provides a centralized view of all release activities, giving organizations greater control over the deployment process. This allows for better decision-making and faster troubleshooting if needed.
• Increased efficiency and productivity: ERM streamlines the software release process by establishing a defined workflow. This reduces redundancy and wasted time, allowing teams to deliver updates faster.
• Minimized risks and reduced downtime: ERM helps identify and address potential issues before they impact production. This proactive approach minimizes the risk of bugs and disruptions, leading to a more stable and reliable software environment.

With an optimized Enterprise Release Management system in place, software teams can increase output and avoid common pitfalls—like testing conflicts, quality issues, and security gaffes, among others.

Best Practices for Enterprise Release Management

Building on its core functionalities, this section explores best practices to optimize the ERM process and achieve successful software deployments.

• Plan and communicate: Coordinating complex project releases involves working backward from a target date. To achieve this, you need to map project dependencies and features, establish a timeline with buffers for testing and delays, and gain agreement from all teams involved.
• Manage dependencies: Increased project dependencies within a release heighten the risk of delays and last-minute bugs. Managing these dependencies involves tracking, reserving integration testing time, and ideally, promoting the creation of loosely coupled, independently testable project components. This allows for earlier integration testing and avoids the pitfalls of “big bang” integration at the last minute.
• Understand the importance of a pre-production environment: One of the ways to reduce release risks is to have a quality pre-production environment. Thorough testing in pre-production helps identify issues in the release that might be an expensive failure in production.
• Automate whenever possible: Automating steps such as building, testing, and deployment saves you time and reduces the opportunity for errors.
• Document anything you can’t automate: If a step is too hard to automate, document it. People who aren’t on the project team have to be able to build, test, and package a new version. Such documentation is essential if you ever need to deploy a security update quickly, for example.
• Deploy regularly: Regularly deploying in ERM minimizes risk by releasing smaller, more frequent updates. This fosters faster feedback, improved quality, and greater agility, allowing organizations to adapt and innovate quicker.
• Have standard operating procedures (SOPs): Creating SOPs for timelines, dependencies, and team coordination provides a reusable framework to streamline planning and reduce workload.
• Observe and improve: Observing deployments through monitoring helps isolate issues after releases, enabling targeted rollbacks or fixes while user data on feature usage provides valuable feedback for future releases.

Overcome Enterprise Release Management Challenges With Enov8

It’s very difficult to produce consistent and high-quality software without a central Enterprise Release Management platform in place. Companies often lose control over their production environment due to outdated and inefficient management policies.

This is exactly where Enov8 comes into play. Enov8’s Enterprise Release Management tool provides a central framework that enables you to orchestrate releases and measure progress each step of the way. With Enov8, you can help define and build an enterprise release schedule, onboard projects, identify system requirements, and deploy via DevOps automation.

At the same time, we can also help your organization boost agility by giving you more control and automation throughout your software development life cycle. This, in turn, leads to lower costs, accelerated project timelines, and a supercharged DevOps team.

Ready to unlock the full potential of Enterprise Release Management? Experience Enov8 in action by downloading our “Kick Start” edition today.

Deployment shouldn’t be the most nerve-racking task for IT and DevOps Teams, but many times it is.

If we’re deploying application changes to a beta system, we don’t care that much if the system goes down. Neither will your users—they know that beta means “expect problems from time to time.” The same thing applies to new systems that don’t have a large user base yet.

But what if we’re talking about bigger systems, like those used by a major online retailer or a bank? Users might get frustrated if they have problems shopping online. They may choose to buy from a different store as a result. And they’ll be furious if they aren’t able to access their bank account to make a withdrawal. Even if it’s just for a small fraction of time, companies lose credibility every time their systems are down or malfunctioning.

Deployments tend to get really difficult. Just how difficult is proportional to the complexity of the system you’re working with. The more dependencies the system has, the riskier and more problematic deployments get. But what are the common mistakes that we’ve all made?

#1 Not Enough Automation

Manual deployments lack consistency. So, when a significant portion of your procedures are manual, you wind up with a big ball of mud. I have suffered the consequences of doing deployments manually. I vividly remember one time when the site was down intermittently because I forgot to upload the binary files to 1/4 of the servers in the farm.

With the rise of DevOps, automation is becoming de facto. But don’t get overwhelmed if your process is still largely manual. You don’t need to stop what you’re doing and automate everything. I’ve always loved the boy scout rule to leave things better than you found them. It’s OK to start manually because it’ll help you understand how everything works. But do identify where automation can be used. You might want to be able to go from 0 to 100 to declare the task of automating your process “complete,” but automation should be done progressively and continuously.

The only manual thing you might want to have is someone approving the release, but even that should be a matter of just clicking a button. So, it’s not just about copying and pasting things; it’s also about configuration management, infrastructure, on-demand environments, and so on.

When you automate, you foster communication. If there’s only one way to make changes in deployment, everyone will know the what, why, and how of things. And if everyone is in the know, the guy or gal that solved a complex deployment problem can go on vacation without having their mai-tai-at-the-beach time interrupted by a clueless colleague back at the office.

#2 Different Environment, Different Way

I once fell into the trap of automating deployments by environment. In my case, there were four environments: dev, QA, staging, and production. I took my time to finish with the dev environment. Then I needed to replicate those changes in the other environments. The problem was that the deployment process for each environment was completely different in most cases.

I’m a big fan of the twelve-factor app methodology. Regarding deployments, it says that you need to keep config values in the environment, not in the application or any file. It also says that environments should be as similar as possible—not in size, but in style.

In other words, if you lack production-like environments, you’ll be lying when you say you’re ready to deploy to production. I’ve seen it, I’ve suffered it, and I stopped treating production as some sacred environment to be feared.

When I was part of a project where we Dockerized the system, I understood this very clearly. I used concepts like immutable infrastructure, infrastructure as code, environment configuration management, cloud, and deployment definition files with Kubernetes as pieces of the deployment puzzle to make things easier.

#3 Waiting for Low-Traffic Time Slots

Any time you hear (or say) “Let’s wait for the weekend to make this deployment,” it’s a sign the deployment pipeline isn’t trustworthy. You’re still treating production significantly differently from other environments.

Netflix and the other “unicorns” in the industry say they actually do deployments during regular work hours. After all, that’s when everyone is together and ready to react properly if there’s an incident. I know you might not be as big as the unicorns, but there’s nothing special about them aside from one thing: they experience problems at a really large scale. That forces them to change the way they’re doing things.

As I said before, you probably do deployments during low-traffic hours or over the weekends because you want to prevent any downtime in the system. But instead of living in fear of screwing things up, why don’t you set the foundation to experiment?

Do deployment strategies like feature flags, canary releases, blue/green deployments, or rolling updates ring a bell? Maybe, maybe not. Either way, these are great strategies that help you make deployments without users noticing it. And it’s a way to avoid stealing family time away from the staff.

#4 Waiting Until Enough Changes Accumulate

Do you have changes that are ready to be deployed? Why wait? To take one out of Nike’s book, just do it!

OK, so maybe something official is delaying you. Maybe you’re waiting for the change advisory board (CAB) to approve the changes based on the risk. I’ve been there too. But maybe you just need to improve communication by automating some steps in the process. These changes don’t come overnight—it takes time for the CAB to trust the process.

Another reason why organizations wait to do deployments until enough changes have been accumulated is, again, to reduce downtime. Downtime usually happens because you’re changing something. It rarely happens because of nightmarish things like the cloud provider going down, fires, earthquakes, and so on.

But I’ll say it again: don’t wait! Work to build a culture that deploys frequently and in small batches. If it hurts, do it more often. Every sprint, you can deploy something that can easily be disabled with a feature flag. Once you decide it’s ready to be used by everyone, turn the feature on to release the changes.

When you deploy too many things at once, you don’t know which of those things might be causing problems. And when you don’t know what’s causing problems, you have to roll back everything, including perfectly functional features that may have been really critical for the system.

#5 A Lack of Subsequent Feedback

You might have good monitoring and observability for your systems, but how sure are you that problems are being caused by a recent deployment and not by something else? There are lots of possible factors, such as internet problems, cloud provider issues, and an increase in traffic.

Another problem is that after a deployment, we tend to monitor changes for a while and then forget about it. But if you end up working with feature flags, it can take some time to activate or release a change. For that reason, it’s important that you have a smart way to correlate events, logs, and any other information you’ve found useful as you’ve learned the system’s behavior.

It’s also important that you have feedback after deployments because that’s how you’ll know if something needs to be rolled back. Without reliable feedback, any issues will just continue to affect your users. Let your team know when a deployment and release happen. It could be as simple as a Slack message in a channel. That way, people can “subscribe” to get notified, ask questions, and learn how the team solves problems.

In summation – Shift Deployments to the Left

In a nutshell, don’t treat production differently from your other environments. If you want to have a high rate of successful deployments, practice them (& standardize them) before going live. I’ve even been in the situation of announcing a release to the CAB for a certain date and then having to delay release dates because we found problems when we were launching the B side of our blue/green deployment.

I’ve also come to the conclusion that most of the things I’ve written here are more of a cultural change than anything else—a mindset that has to be adopted by the team that’s involved in building the software, not just developers and operations.

Start small and keep analyzing areas where you’re consistently struggling during deployment. Then improve. Use Enterprise Intelligence solutions like enov8 to continually understand your IT & Test Environments and your Release Management (inc Deployment Management) processes. The end goal should be that, at some point, you might leave releases to marketing or management because you already took care of doing deployments with almost no downtime.

And how do you get there? Practice until you’ve mastered deployments.

Relevant Articles

Independent of your sector, be it banking, telecoms, retail or government, the desire to get there quickly, and without incident, is paramount as once delivered your IT organization, “amongst other things*”, will be delivering 1000s of servers and services in minutes, opposed to the traditional “on-premise” methods which see a handful of services being delivered in months.

*Other things including Cost Optimization, IT Productivity, Operational Resilience & Business Agility.

Sound Compelling? Well yes, however, the journey itself is rarely trivial.

In fact, it is not trivial. Independent of which cloud(s) you’re going to, be it AWS, Google, IBM or Microsoft Azure.

The reality is this: “it is not just about the Infrastructure”.

Go into any large organization and you will see that the Enterprise IT Landscape (Production & Non-Production) are made up of a complex web of Infrastructure, Applications, Data, Processes, Services, Operations, People, and Projects. And of course, underlying dependencies and relationships.

Or more simply: If you change something here, it will impact something over there.

And therein lies in the problem.

Organizations rarely have the required “Enterprise IT Intelligence” to streamline a cloud migration.

In fact, organizations “traditional” methods & tools result in poor transparency, bad decision making, operational heroics and significant change risk. None of which is ideal as you embark on a major migration program. With this in mind, how should one approach a successful Cloud Migration?

Courtesy of team Enov8, here are seven steps we suggest you consider.

The 7 Step Cloud Migration Process

Step 1 – Identify the potential Return on Investment

If you can’t justify the “obvious” benefits, then you will probably struggle to build a robust business case. At a high-level, review the likely benefits of implementing a better IT & Test Environment Management operating model.

1. Consider obvious “cloud-centric” saving like reduction in Infrastructure Costs.And consider related “non-cloud-specific (or better Environments Management) benefits:
   Including:
2. Reduction in Support / Service Costs (through Operational Simplification).
3. Reduction in License Costs through improved housekeeping & accelerated decommissioning.
4. Reduction in Programme/Project Spend through accelerated provisioning & resilience.

Tip: The Enov8 ROI Calculator can provide you with quick insight into the potential benefits of improving your IT & Test Environment Management capability. Most organizations will usually find a compelling reason to move forward.

Step 2 – Enterprise Intelligence – Environment Discovery and Modelling

If you don’t know what your IT landscape looks like you will struggle to manage (or migrate) it effectively. “If you can’t see the forest for the trees” then take a step back to ensure you understand what all your Systems look like and how they relate with each other.

Establish a discovery process to identify all your core systems. Consider capturing information that will support future risk analysis & migration endeavours. For example, for each System (Platform) identify:

• System Name e.g. Group Data Warehouse
• Ownership e.g. Marketing Business Unit
• Subject Matter Expert e.g. Jane Temov
• System Relationships / Dependencies e.g. GDW gets all data from Mainframe & Siebel
• Number of Physical System Instances e.g. 2 in System Test, 1 in UAT, 1 in Production
• For each System Instance, capture the:
  • Components e.g. GDW UAT has 1 x IIS Server and 1 x MSSQL Database
  • Components physical location e.g. London DC1
  • Components performance & usage metrics.

If you don’t already have a robust & up-to-date Configuration Management Database (CMDB) or IT & Test Environment Management solution like Enov8 Environment Manager, then you will probably find this to be a quite time consuming & error-prone process.

Tip: Move away from managing your Environments & Computer Systems through non-scalable and non-Integratable tooling like Email, Spreadsheets, Visio diagrams and debt-ridden CMDB records. There is a lot more to IT Environments Management than simply moving the existing infrastructure layer to “code” (virtual servers). An investment in an Enterprise Intelligence solution now will ensure you can visualize the broader picture and ultimately simplify your immediate and strategic migration objectives. Enov8 Environment Manager comes with advanced mapping capability, discovery capability and integrates with your other favourite discovery and SDLC tools.

Step 3 – Enterprise Intelligence – Environment Operations

Don’t stop at simply understanding what the Systems or Physical Assets look like. Understand & Value Map your existing environment operational capability. Key Operational activities that are important for cloud migrations include:

• Application Operations: Build, Deploy and Test
• Data Operations: Extract, Transform (Subset & Mask) and Load.
• Infrastructure Operations: Provision, Configure & Decommission.

This is important as it will

1. Act as a foundation to support the migration activity itself.
2. Provide you with frame of reference that can be modified, improved and streamlined as you take advantage of cloud specific features and automation.

Contemplate using the Enov8 platform as a central repository to capture your existing operations.

Tip: Use the Enov8 “Runsheet” management feature to replace your static and outdated, Word or Excel documentation. Introduce Runsheet Kanban’s to support your operations definitions, operations standardization, operations tracking and operations automation.

Step 4 – Develop a Business Case

Following on from a “positive” ROI Analysis and enriched by deeper insight, acquired in the Enterprise Intelligence Phases, create a business case that helps to justify the migration journey you are about to embark on.

FYI: For those of you that wonder why we have the Business Case as Step-4, and not Step 2, the answer is as follows: If you don’t have Enterprise Intelligence then you are probably not managing your environments & operations as effectively as you should. By doing Steps 2&3 you are investing in the future of your IT organization and driving a level of insight, independent of cloud, that will support better analysis, decision making & optimization.

Tip: Enov8 dashboards and reports will help you accelerate the development of your business case by helping you aggregate relevant information, identify both financial & productivity benefits of restructuring your landscape, spot the paths of least resistance (e.g. avoid potential risks) and select the appropriate migration strategies.

Step 5 – Migration Planning

With your enterprise IT landscape now mapped, it is time to plan the migration. Ultimately a portfolio (enterprise release) management level activity, this involves breaking the migration effort into logical sprints.

Tip: Consider migrating by family (or mix thereof), for example:

• By System (Platform)
• By Systems Groups (Tightly Coupled Platforms)
• By Business Process
• By Business Division
• By Teams (Ownership)
• By Technology Tier e.g. MSSQL Databases first
• By Location e.g. Rack 24 London
• By Complexity i.e. Low, Medium, High, Very High
• By Cost

Tip: To ensure end-to-end delivery control and visibility, look to use an IT Portfolio Management solution. Enov8 Release Manager provides the holistic capabilities you need to manage an enterprise migration process e.g. Portfolio Migration Management, Sprint Management, Implementation Planning, Deployment Operations, Automation and Configuration Tracking.

Step 6 – Migration Sprints

Refine further, progress from Portfolio level planning to the more granular System level.

For each migration sprint:

• Design Target State,
• Migrate*,
• Validate New Cloud Systems and
• Decommission Legacy

Note*: Migration approach will vary dependent on System(s) under transition.

Referencing Gartner and AWS approach, the migration options are nicely described by the 6Rs.

• Re-host (aka Lift and Shift)
• Re-Platform (move parts of the system/solution to cloud)
• Re-Architect (redesign the system and move to cloud)
• Re-Purchase (decommission and buy something new)
• Re-tire (surplus to requirements)
• Retain (do nothing, leave it alone)

Note: In addition to broad Enterprise Intelligence and Environment Management capabilities, the enov8 platform supports the migration objectives by providing a delivery framework that allows you to define, track and report on all migration activities.

Step 7 – Operate & Manage

Post-migration, harness the benefits of cloud and IT Environment Management.

Use your new Cloud to:

• Rapidly Provision Infrastructure on Demand,
• Rapidly Provision Base Applications on Demand,
• Rapidly Decommission &
• Replace manual operation with streamlined “Operations as Code”.

And Embrace an Environment Management solution to:

• Ensure transparency, control & collaboration across the Enterprise IT Landscape

i.e. Applications, Data, Infrastructure, Operations, People & Tools i.e.

• Infrastructure & Cloud Provisioning Tools
• Application CICD (Build/Deploy) Tools
• Data Integration Tools
• Testing Tools
• ITSM Tools
• ALM Tools

Why use Enov8 Environment Manager for Cloud Migration & Management?

Use Enov8 Environment Manager to:

• Continually Map/Model your Cloud and Non-Cloud (Legacy) Environments in a Visual CMDB
• Monitor Health/Readiness of Your Cloud and Non-Cloud Applications and Data
• Promote Sharing Efficiencies and/or Avoid Conflict through Demand Management
• Centralise Planning & Coordination (scheduling) to ensure proactive & accurate provisioning
• Manage Environment Service Requests (e.g. Change, Release & Incident)
• Standardise & Automate all Environment Operations (e.g. Data, Applications & Infrastructure)
• Track your delivery pipeline, as your solutions move through the life cycle
• Provide end to end auditability and history &
• Use real-time insights (via rich Status Accounting & Reporting) to improve continually

Benefits Enov8 provides

• Centralised Intelligence for all Environment Information
• Control of all Environment Types e.g.
  – On-Premise,
  – Cloud,
  – Hybrid,
  – Monolithic
  – Legacy and Microservices
• Improved Environment & Release Operations,
• Reduced Environment Disruption,
• Program Streamlining and
• Cost Optimization across Cloud and Non-Cloud
  – Infrastructure,
  – Licensing &
  – Services

Want to learn more?

If you are interested in learning more about implementing a mature Environment (including Cloud) Management framework in your organisations then speak to enov8 about enov8 Environment Manager. Enov8 Environment Manager is the only complete platform that takes you across the Environment Management & Release Spectrum.

Niall is the Co-Founder and CIO of Enov8. He has 25 years of experience working across the IT industry from Software Engineering, Architecture, IT & Test Environment Management and Executive Leadership. Niall has worked with, and advised, many global organisations covering verticals like Banking, Defence, Telecom and Information Technology Services.

Relevant Articles

Teams Blocked on External Resources

Development and Release Concerns Intertwined

Broken Loops

Bureaucracy

Wrong Focus

Conclusion

Relevant Articles

Well here at Enov8, over the last few years, we have been building a model called the EMMi (the Environment Management Maturity Index)

Originally called the TEMMi , the EMMi is designed to help organisations understand the big picture, openly discuss capabilities using a standard “frame of reference” and quickly assess themselves i.e. Identify Strengths, Weaknesses, Opportunities and Threats.

The EMMi (Environment Management Maturity Index) can be broken into following eight Key Performance Areas:

(Find the PDF here)

So how do you rate?

Follow the methodology below & use the online maturity calculator to generate yourself a spider diagram baseline report.

EMMi – Score Card

Consider each of the eight Key Performance Area (KPA) from a People (skilling), Process (Repeatability) & Product (Tooling perspective) and score yourself from 1-5.

1. Environment Knowledge Management:
Understanding your IT environments across the lifecycle i.e. across Projects Delivery (e.g. Development, Test, Training) & Production. Without effective intelligence, you will struggle to manage your environments effectively. Intelligence mapping is both top down & bottom up i.e. Top down included understanding Businesses Units, Business Processes, Systems relationships, while Bottom up looks at from the other direction i.e. Components, Interfaces & Instances. Score yourself:

2. Environment Demand Awareness

There is a reason you have these environments, they’re called your consumers (Project Teams, DevTest Teams etc). The ability to understand demand and current usage of one’s IT Environments and shift planning & coordination left. Score yourself:

3. Environment Planning & Coordination

The proactive planning & coordination of environment events & deployments ensures your systems are provided in a timely fashion and ems are correctly configured i.e. fit for purpose and available. Score yourself:

4. Environment IT Service Management

IT Service Management is a customer-focused approach to delivering & supporting information technology. Key aspects of ITSM support include management of Incidents, Change and Releases. ITSM, although potentially leaner, is relevant even in Non-Production as it controls the chaos, invisible change and avoids disruption. Score yourself:

5. Application Release Operations

The implementation of consistent, repeatable & traceable application release operations contributes to the broader needs of IT Environment Management and IT delivery. At a minimum, you should be promoting standard operating procedures & ideally tracking these & automating the most time-consuming tasks. Score yourself:

6. Data Release & Privacy Operations

The implementation of consistent, repeatable & traceable data release operations contributes to the broader needs of IT Environment Management and IT delivery. At a minimum, you should be promoting standard operating procedures & ideally tracking these & automating the most time-consuming tasks. Score yourself:

7. Infrastructure & Cloud Release Operations

The implementation of consistent, repeatable & traceable infrastructure & cloud release operations contributes to the broader needs of IT Environment Management and IT delivery. At a minimum, you should be promoting standard operating procedures & ideally tracking these & automating the most time-consuming tasks. Score yourself:

8. Status Accounting & Reporting:

The ability to capture and present “real-time” environment information in a way that will uplift IT environment analytics, decision making and continual optimization. At any point in time you should be able to visualise Environment Topology, Usage, Health, Activities, Operational Behaviour and System Team Competence. Score yourself:

Want to learn more about how to improve each of the Environment dimensions?

If you are interested in learning more about implementing a mature Environment Management framework in your organisations then speak to enov8 about enov8 Environment Manager. Enov8 Environment Manager is the only complete platform that takes you across the Environment Management & Release Spectrum.

Niall is the Co-Founder and CIO of Enov8. He has 25 years of experience working across the IT industry from Software Engineering, Architecture, IT & Test Environment Management and Executive Leadership. Niall has worked with, and advised, many global organisations covering verticals like Banking, Defence, Telecom and Information Technology Services.

Relevant Articles

The backbone of business today is digital, and every day we rely more and more on the IT services and infrastructures around us. However, when things go wrong, it can go wrong dramatically and lead to loss of service, loss of revenue, unhappy customers, brand damage and potentially even worse i.e. life and death in some cases.

With that in mind, here are our top seven IT environment outages that left an impact on 2017 in chronological order.

1 – GitLab, January 31
Cause: Accidental Removal of Database

GitLab had an outage lasting nearly 18 hours on January 31^st. An outage that impacted myself and team, as we couldn’t push our changes to the repository for a day. Thankfully, it wasn’t during a major or scheduled release and we waited it out without incident.

In our case we lost no data, however, some weren’t so lucky. According to GitLab, they lost “six hours of database data (issues, merge requests, users, comments, snippets, etc.) for GitLab.com.” Nevertheless, GitLab did earn a lot of goodwill for their transparency in handling the outage and to their credit, defended the individual team-member-1 (an experienced DBA) who made the mistake.

2 – Amazon Web Services, February 28
Cause: Wrong Command Accidentally Typed

On February 28^th, an Amazon Web Services engineer was trying to debug an S3 storage system in the provider’s Virginia data center when he accidentally typed a command incorrectly. The result? Well, it felt like half the internet was down for four hours. Affected systems include many enterprise platforms, like Slack, Zendesk, Trello, and consumer-facing ones like Quora.

What made things worse was that the AWS Service Health Dashboard (which AWS customers look at for updates on outages) was itself broken. In this case, it showed that only S3 was having issues. But in reality, multiple AWS services weren’t working properly. AWS reported that the error in the dashboard was because “the icons for the many AWS services were hosted in the Northern Virginia region.” That’s the same region where the outage hit the affected services. Yes, this outage was so bad that AWS couldn’t even get its own health dashboard to work properly.

AWS took preventive measures after this, such as running audits on its operational systems. It even adjusted one such system “to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level.” In other words, Amazon doesn’t want to undergo the embarrassment of the 2017 outage ever again.

3 – WhatsApp, May 4
Cause: Unknown (According to Speculation, Updates Were at Fault)

On the same day that Facebook CEO Mark Zuckerberg announced that WhatsApp had more than 175 million users, WhatsApp went down for four hours. Users were loudly complaining about how little information Facebook was sharing about the outage. Even the official WhatsApp Status Twitter account, which appeared to be unused for at least a year, was silent on the matter.

After WhatsApp went back online, Reuters reported that the company’s spokesman only acknowledged the outage and mentioned that they “have now fixed the issue and apologize for the inconvenience.” WhatsApp gave no reason for the outage, but people speculated that the outage was due to an update.

This outage teaches a valuable lesson. When the makers of a heavily-used service roll out a new release, that should be as carefully planned as the development of the release itself. It’s also worth noting that people can forgive and even empathize with human error, as seen in the GitLab accidental deletion incident, however you must own up to it.

When you deliberately make changes, people have expectations about the smoothness of those changes. It’s a no-win situation, of course. Do it well, and you’re just meeting those expectations. But do it badly, and you incur the wrath of many—in WhatsApp’s case, possibly hundreds of millions.

This means managing your test environment and testing the update there prior to release is especially crucial for a smooth rollout of software updates. If you have an ecosystem like Facebook with multiple interconnected systems, it’s even more challenging to manage a test environment manually. Automated testing and management of the test environment are key.

4 – British Airways, June 2
Cause: Accidental Power Shutoff by Contractor

Unlike the other cases so far, this one seems particularly low-tech. But this outage was just as severe, if not even more so. On June 2, British Airways left 75,000 people stranded in UK airports over the weekend. All flights out of London’s Heathrow and Gatwick were canceled. Computer systems were knocked out. Flight operations were disrupted. Call centers and websites went down due to an overwhelming number of requests from customers.

The cause? A contractor working in the British Airways data center accidentally shut down a power supply unit.

If you think you have difficult customers, try having 75,000 angry passengers with disrupted flight plans. Experts warned that the final cost of this outage would be “significant” for the UK flag carrier. It was a fair assessment. British Airways ended up paying “£200 for a hotel room (with couples expected to share), £50 for transport between the airport and the hotel, and £25 per day for meals and refreshments” for every affected passenger.

5 – Marketo, July 26
Cause: Someone Forgot to Renew the Domain

Maybe you haven’t heard of Marketo. If that’s the case, let me tell you a bit about them. Marketo has been a publically-listed company since 2013. It’s widely considered at the forefront of the market for small business lead management—so much so that by 2017, Gartner Magic Quadrant named Marketo the leader in CRM lead management for the sixth year in a row.

So what happened? Well, they forgot to renew their domain. So businesses using Marketo services sent out email campaigns with links that were completely broken. It was a Marketo customer who spotted the expired domain and renewed it before anyone else could. It’s bad enough to have an outage. Having your customer help you is especially embarrassing.

Even the renewal couldn’t completely solve the issue, as DNS propagation can take 48 hours. The Marketo team probably had to ask ISPs to flush the DNS cache for them.

Marketo’s outage had to be on this list, simply because, for some customers, the outage may have lasted as long as 48 hours.

6 – WhatsApp, November 3
Cause: Unknown. Again.

Yup, it happened again. WhatsApp went down for the second time in less than six months. Thankfully, clocking in at one hour, it didn’t last as long as the May outage. But that’s where the differences end. Everything else is the same, from the lack of information about when things would be back up to the lack of clarity about the cause—and the speculation again that it had to do with software issues. Compare this with GitLab’s constant tweeting about their progress in recovering their database.

WhatsApp had two outages in span of less than six months. That suggests the issues are systemic in nature. And when there are systemic issues, the companies affected can find and solve them permanently. I cannot emphasize enough how important it is to permanently solve preventable issues. As we’re revisiting all these cases of outages, the outcomes are almost always negative. You’ve got to prevent them.

7 – Marketo, November 22
Cause: Firewall Issues from Data Center

Yup, another Marketo outage. Less than four months after they forgot to renew their domain and faced the outage mentioned earlier, they faced another outage lasting several hours. The length of this outage wasn’t as bad, but the timing was awful. It happened when marketers were preparing for the weekend of sales activity following the Thanksgiving holiday—in other words, the weekend of Black Friday.

Once again, this certainly sounds like something that was entirely preventable. The company announced a root-cause analysis into this matter shortly after restoring their services…which should be a basic requirement of every such outage on this list.

Conclusion: Minimize Preventable Outages

Of all these outages, it’s those that are the most preventable that should be the least forgivable. Therefore, systemizing a release management or environment operations process is key towards preventing future preventable outages.

Not sure what features constitute a good release management system? Take a look at the Enov8 release management datasheet. Enov8 offers a holistic approach to release management, ranging from at Scale Enterprise Release Management (Agile Release Trains) through to Implementation Planning and ultimately Agile Deployment Operations and Automation (inc CICD).

If you have your own favorite outages then weigh in! Which do you think were the worst?

Relevant Articles

Your project puts in a request, they grab copies from production, mask them, subset them and deploy them into your Test Environments. Easy Peasy! … Well probably not, probably a lot of paperwork, engineering & provisioning effort. And then the issues really start.

• The data lacks end-to-end integrity i.e. the data is broken.
• The developers & testers can’t easily find the data they are looking for.
• And when the IT teams do find the correct data-points they can use, they all use it.

Causing contention & data related test defects. And it all starts to grind to halt, suddenly Development & Test cycles are being blown out.` And then to add insult to injury, an honest Test Analyst, notices that not all the data has been masked. A serious concern when one considers that’s where your project teams spend 95% of their time, and the opportunity for information to be misplaced or stolen is high. A suboptimal situation, which exposes the customer to:

• Identity Theft & Fraud

And exposes your own organisation to

• Compliance Penalties
• Industry sanctions
• Brand Damage
• Consequent Lawsuits

Not exactly ideal, particularly with data compliance legislation like GDPR that will sting you for 4% of turn-over. Yet I can virtually guarantee, sadly, that the above scenarios describe most organizations today. The reason why data is such a problem – is sixfold:

1. Enterprise architectures are typically diverse & distributed.
2. Environment & data footprints are under constant change.
3. Individual databases are often large, poorly defined or understood.
4. System & data documentation often suffers from technical debt.
5. Easy to make mistakes during data Subsetting (causing integrity health issues).
6. Very easy to make mistakes during data obfuscation exercises (causing PII leakage).

The traditional ETL approach to Test Data Management simply isn’t good enough.

• It is too Slow
• It is too Manual
• It is too Error-Prone &
• It is not customer / user-centric

There is a fundamental need to recognise that successful Test Data Management can’t rely on ETL alone. Instead, organizations must start looking at data a little more broadly and leverage more automation to ensure accuracy, quality, compliance & ease of end-user consumption.

Enter the Holistic Test Data Management (HTDM) Framework

Designed by Enov8, the HTDM Framework is used to contextualise the broader aspects of Test Data Management. A set of “Lego” blocks that call out the broader considerations and needs of an automated test data solution. Build around the traditional ETL, the HTDM promotes supporting capabilities like: Data Requirements Capture – So you have a clear understanding of consumers (testers & projects) needs. Automated Data Profiling – To rapidly understand data structures and PII risks (pre-ETL). Automated Data Validation – To rapidly determine if created data (post-ETL) is free of production patterns and healthy (has integrity). Test Data Mining – So Testers can visualise, understand and find end-to-end (cross-system) data without building complex queries and scripts. Test Data Bookings – So test data can be assigned to test cases or teams and avoid the risk of overwrite.

Key Benefits of creating an HTDM Framework

• Understand your Data
• Improve Compliance.
• Ensure Data Health
• Ease of Consumption (based on your needs, healthy & easy to find)

All of which leads to happy testers and streamlined project delivery.

Learn More or Share Ideas

If you’d like to learn more about Data, Release or Environment Management or perhaps just share your own ideas then feel free to contact the enov8 team. Enov8 provides a complete platform for addressing organisations “DevOps at Scale” requirements. Providing advanced “out of the box” Holistic Test Data Management, IT & Test Environment Management & Release Management capabilities.

Innovate with Enov8, the IT Environment & Data Company.

Specializing in the Governance, Operation & Orchestration of your IT systems and data.

Delivering outcomes like

• Improved visibility of your IT Fabric,
• Streamlined Delivery of IT Projects,
• Operational Standardization,
• Security & Availability,
• DevOps / DataOps Automation,
• Real-Time insights supporting decision making & continuous optimization

Relevant Articles

Ever wonder why your releases take so long?​

After all,  your company just invested a “zillion” dollars on a whole bunch of great ‘agile’ tools and a cloud framework. Tools that allow you to automatically provision your infrastructure, applications, data and ensure that all your security obligations are met.

Hey! With this new” DevOps toolchain”, we should be moving our releases, from request to delivery, in a matter of minutes. You know … full push-button automation! … environments on demand! … And all that stuff!

Yeah! Right! But no! That’s not how it typically plays out.

In fact, a more realistic example might follow a storyline as follows:

• Project manager raises a request for a Test Environment.
• Request sits in it service management queue for a few days.
• Gets approved & assigned by Test Environment Manager & distributed to engineering teams.
• Sits in the team ITSM queues for another few days.
• Apps team build the package in 5 minutes, but can’t deploy as infra not ready.
• Infra team provisions.
• Test team can’t start testing as data not ready.
• Data team ensures Data secure & Compliant.
• Data team provisions.
• …
• Sorry, testing now too busy with another test cycle.
• Test teams spot a defect with the build.
• Higher priority project comes along and acquires environment.
• Go back to go.

I think one gets the point, the issue with DevOps efficiency is rarely down to a single atomic task.

In fact (as illustrated in the diagram below), if you were to take a step back, you would probably realise the inefficiency is not in the operations themselves (like a build task) but in fact the “void” (or the wastage) in between.

In the above multi-process diagram, we can see the Data team takes:

• 180 minutes of operations (real value),
• 5 days of waiting (wastage),
• Or 2.5% Efficiency (Value Operation / [Value Operation + Wastage]).

Not exactly something you want to write home about. However, not an “untypical” in-efficiency, and a serious opportunity for improvement.

Imagine if for each team could move from 2.5% to just 25%. The benefits would be enormous, and over the lifetime of a project we could be saving weeks, maybe months, of time which translates to early “time to market” and significant IT project cost savings.

Enter Value Stream Mapping

Originally employed in the car manufacturing space, Value Stream Mapping (VSM) is a lean method that helps you better define a sequence of activities, identify wastage & ultimately improve your end-to-end processes. A set of methods that can be applied to any type of operation, including of course Test Environments & DevOps.

Some Definitions:

What is a Value Stream

A value stream is a sequence of activities used to create a product or service that customers value. Its purpose is to optimize the flow of value and eliminate waste, reducing lead time. Value stream mapping is a tool used in Lean manufacturing to analyze and optimize processes. This technique can be applied to any industry, including software development, to identify opportunities to improve efficiency, reduce costs and deliver more value to customers.

What is a Value Stream Mapping

Wikipedia Definition: Value-stream mapping is a lean-management method for analyzing the current state and designing a future state for the series of events that take a product or service from its beginning through to the customer.

How do I go about implementing a simple* Value Stream Mapping for DevOps?

1. Select the Product e.g. CRM application
2. Select the Delivery Process of Interest e.g. Build a test environment
3. Gather the SMEs, as VSM is a team event.
4. Visually Map Current State (material flow / operational steps)
5. Identify Non-Value between steps
6. Add a timeline for both Operations (green line above) + Non-Value (red line above)
7. Review Value Stream
8. Design Future State (Optimize)
9. Return to (3).

Tip: When getting started, steps 4 to 8 may initially be completed on whiteboards and simply use guesswork (in place of real data). However, for ongoing improvements, consider using tools that allow you to model your DevOps processes, track the operations and report on stream actuals. As an example, you could use the “Visual Runsheets Manager” functionality inside the Enov8 platform.

Benefits of DevOps Value Stream Mapping

• Baseline existing Operations
• Standardise Operations
• Identify Wastage
• Highlight Operational Bottlenecks *non-automated
• Lift Efficiency *continuous improvement

Learn More about VSM

If you want to know more about how to leverage VSM in your IT delivery process then feel free to contact the Enov8 team. Enov8 is a complete solution that helps you streamline delivery via automating your Release Management, IT Environment Management and DataOps / Data Compliance activites. And offers real-time insights, through customizable ‘information walls’, that help you measure performance and be “agile at scale”.

Enov8 Release Manager, Example Information Wall: Screenshot

Relevant Articles