Marrying SAFe and DevOps

MAY, 2019 by Rodney Smith
If you work in an organization that uses the scaled agile framework (SAFe), chances are it’s not a small company. It’s enterprise-y. It’s probably gone through some growing pains, which is a good problem to have in the business sense. The good news is that the spirit of agile is alive within the organization, but the challenge is that the value stream doesn’t begin and end with your two-pizza team anymore.  

Life in a Scrum Team

Scrum teams are familiar with backlogs. The product backlog is all the stuff we know we want to do someday. The sprint backlog is a curated list of the things we want to do now. Simple, right? If you’re fortunate enough to have all the disciplines necessary to ship software embedded in your scrum team, consider yourself lucky. You can load up tasks for the database administrator, the infrastructure engineer, and the application developers all in one sprint backlog. You may be able to address your security guru in the daily stand-up. Life is good in the small. But at scale (read: in a large organization), things get a little more complicated. The DBA may be a shared resource, serving several scrum teams. Infrastructure is probably its own group—one which doesn’t report to your boss, but whose contributions are still part of the value stream. Complicating matters further, your team isn’t the only team they collaborate with. (For a more in-depth view of “at scale,” see the article Pitfalls with DevOps at Scale.) This probably sounds familiar to anyone who operates on a scrum team in an organization of any size. You have some semblance of process around the work items that are in your immediate control. You’ve got daily stand-ups and pull requests for code review. User stories flow in and are nicely analyzed and broken down. It really feels like the immediate team is mature and finely tuned. But when it’s time to interface with another team, people start to quietly groan.

Cross-Team Collaboration Is Hard

When you need something from another team, it starts to look like the Wild West again. You must ask favors of the infrastructure team. Whether they honor your requests depends on your personal relationships with them. You find yourself using emails rather than ticketing systems to make critical work requests, so there is no way to track the status of these requests. Oftentimes, these requests turn into a hot potato, with no one stepping up to own them. In the image above, we see the old “boxes and arrows” diagram. One of those boxes is your scrum team. Life is good inside that box. The other boxes represent teams that your team must interface with to deliver business value. It’s the arrows that cause all the heartache. Suppose your application development team has created a feature branch to build some new functionality. You’d like to get a test environment provisioned so that you can deploy and test this new functionality. You may fire a request into the abyss and be frustrated by a terse response of “we’ll get to it”—or worse yet, no response at all. Such is life in the siloed organization. But there’s a better way.

SAFe and DevOps Coexisting

The spirit of the DevOps movement has always been to bridge this gap between development teams and operations teams, but in practice, the bridge typically falls down. Why? Often, no significant process or organizational changes actually occur when an organization decides to put its proverbial toe in the DevOps water. If you’re already using SAFe, how do you incorporate DevOps into your Agile Release Trains so that their contributions are reliably tracked and accounted for? Going back to our SAFe principles, we see that principle #7 is “Apply cadence, synchronize with cross-domain planning.” What this means in practice is that we gather all the stakeholders on a regular basis and discuss cross-domain issues—in other words, frequent communication and dependency management. This idea should resonate with those familiar with Scrum of Scrums. SAFe even gives us another fancy term, along with an acronym, for this ritual: Program Increment (PI) Planning. This is the meeting where DevOps and development team representatives are in the same room and addressing the project’s needs. Let’s look back at our earlier example, in which the development team needed a test environment provisioned. If that cross-domain concern hasn’t been addressed, it would be called out in the PI meeting. The more frequently you hold PI meetings, the less time you should spend being blocked by a dependency on another team.

It’s One Big Team

Often in siloed organizations we see an application development team being on the hook for delivery. Meanwhile, the other teams are detached and not accountable. Managers lament that they can’t get the infrastructure team to do the necessary work because they have no authority over that team. SAFe aims to put the focus on delivering business value, irrespective of organizational boundaries. Therefore, every team participating in the release train shares responsibility. In practice, this means that every constituent team should both share in the success and feel the pain of failure. Many large organizations place accountability solely on the shoulders of the leadership, and usually it’s very late in the game, after many missed targets.  To execute large-scale projects successfully, we need high-performing teams. But that’s just step one—that is, table stakes. Looking back to our boxes and arrows diagram, let’s consider the outcome if all of our constituent teams (boxes) are high performing, yet the interactions (arrows) are failing. In this case, the project is probably in trouble. So, we need to drive accountability down into the teams and across teams. There are many management philosophies for motivating and incentivizing teams and individuals, and each organization has its own way of doing things. Suffice it to say, ignoring low-performing team members is a losing strategy, but conversely, refusing to acknowledge high-performing team members is also perilous. The bottom line is this: if someone isn’t doing their job, address it. To keep the Agile Release Trains running on time, we need orchestration. Every player must play their part.

DevOps Is a Key Player

Every organization has its own definition of DevOps. Sometimes it’s a stand-alone team that calls itself DevOps. Other times, it’s a title given to someone on an existing development or infrastructure team. For organizations practicing SAFe, the important part is that DevOps is a recognized entity during cross-domain planning. If DevOps owns the job of maintaining the stability of the production environment, they should be fully engaged in any effort that could impact application behavior. DevOps is one of those terms that’s open to interpretation, both with its form and with its function. The important thing is that they contribute to the value stream, are responsible for certain types of work, and are recognized at the table. DevOps isn’t just “glue” to fill in the gaps; it came about to address deficiencies in organizational structures. SAFe emerged as a way to apply agile principles at the program level and coordinate interactions between smaller teams. These two concepts can coexist harmoniously to provide compounding benefits to an organization.
Rodney Smith This post was written by Rodney Smith. Rodney has deep experience building and deploying software, with a focus on automation and repeat-ability. He’s capable across all layers, from UI to middle tier to database, and from source control to build server on through the deployment pipeline.

Relevant Articles

Sand Castles and DevOps at Scale

03JUNE, 2022 by Niall Crawford & Carlos "Kami" Maldonado. Modified by Eric Goebelbecker.DevOps at scale is what we call the process of implementing DevOps culture at big, structured companies. Although the DevOps term was back in 2009, most organizations still...

Test Environment Management Explained

Test Environment Management Explained3JUNE, 2022 by Erik Dietrich, Ukpai Ugochi, and Jane Temov. Modified by Eric GoebelbeckerMost companies spend between 45%-55% of their IT budget on non-production activities like  Training, Development & Testing and lose 20-40%...

Serverless Computing for Dummies

3JUNE, 2022 by Eric GoebelbeckerWhat Is Serverless Computing? Serverless computing is a cloud architecture where you don’t have to worry about buying, building, provisioning, or maintaining servers. In return for structuring your code around their APIs, your cloud...

Test Environments – The Tracks for Agile Release Trains

25MAY, 2022 by Niall Crawford & Justin Reynolds. Modified by Eric Goebelbecker.So, you’ve decided to implement a Scaled Agile Framework (SAFe) and promote a continuous delivery pipeline by implementing “Agile Release Trains” (ART)*.  Definition: An Agile Release...

What Is Data Masking and How Do We Do It?

24MAY, 2022 by Michiel Mulders. Modified by Eric Goebelbecker.With the cost of data breaches increasing every year, there’s a need for higher security standards. According to IBM’s 2021 security report, the average total cost of a data breach has risen to $4.24...

Test Environments: Why You Need One and How to Set It Up

24MAY, 2022 by Keshav MalikWith the rise of agile development methodologies, the need to quickly test new features is more critical than ever. This is especially true for websites and applications that rely on real-time data and interaction. The only way to ensure...