Environment Resilience – Hiring an SRE Team
by Eric Boersma
Taking on Site Reliability Engineering (SRE) is not an easy task. It doesn’t matter where you’re coming from. Some organizations have done a little DevOps and are trying to break into SRE. Others haven’t even taken that step, and figure they should go all the way with their implementation. Wherever you’re coming from, hiring an SRE team is a big undertaking. There are a lot of pitfalls along the way. With the correct planning, though, you and your team can navigate them successfully.
Adding new Site Reliability Engineers to a team can be stressful. That’s true for both the company, and the employee. If you hire an SRE team, but you’re not prepared to support them, they’re going to have a bad experience. In turn, your experience of the process will be bad, too. In this blog post, we’ll talk about some things you can do before hiring an SRE team, while hiring an SRE team, and after hiring an SRE team to make it a smashing success.
Before Hiring an SRE Team
There are a few steps you can take before hiring an SRE team or engineer that will make your process immeasurably smoother.
Know What Success Looks Like for the Team
You’d think that this goes without saying. However, you’d be surprised at the number of organizations I’ve worked with who don’t know what it takes for a new hire to be successful in a role. That kind of ambiguity makes it difficult to attract good talent, and to evaluate their performance. While you don’t want to have every minute of their first six months on the job planned out, you do want to have a good picture of how they’ll thrive. I try to answer the question “What would a really great first year from this person look like?” If I can’t answer that, I’m probably not ready to hire someone.
Know Your Existing Strengths and Weaknesses
Often times, when writing job descriptions, managers will simply copy and paste the list of requirements they had for the last time they hired someone. This is a bad idea! You don’t want to hire someone who’s going to have the same skills and shortcomings as the people already on your team. For instance, if your team already has a couple engineers and tools that knock Environment and Release Management out of the park, adding another person with skills in that area isn’t likely to improve the team. Instead, you might want to look to someone who has skills in Test Data Management or security. Adding that person to your team means you get stronger.
Even if you’re trying to replace someone who’s leaving, you don’t want a line-for-line replacement for that person. If you’ve been doing your job as a manager, you should have been training other people up to match that person’s skills, so their job description wont’ be what you need any more. Instead, think hard about the skills you need but don’t have. Write a job description that emphasizes the skills you need, not the skills you already have.
While Hiring an SRE team
The process of hiring a new SRE team or engineer can be quite complicated. It’s easy to get lost in the weeds while hiring, and make decisions that you regret later on. Here are a couple things to keep in mind.
Filter for Good Incident Response
Incident response is a significant part of SRE, for both individuals and teams. Rapidly-expanding teams are more likely to experience breaks when new systems fail. That’s OK. That kind of uncertainty is built into the SRE mindset. What’s not OK are those breaks turning into significant outages. This is why it’s important to look for people who have experience responding to outages and incidents. SRE as a discipline is still relatively young. If you only hire for people who have years of experience in SRE, you’re going to be paying through the ears. Instead, you want to identify the kinds of skills that will make someone a valuable SRE engineer without necessarily requiring that they’ve done the job before. Being calm under pressure and able to respond quickly in a crisis are great skills for an Site Reliability Engineer to have.
Good Communication Is a Must
Again, this seems like it’s something that should be obvious when hiring new people. In today’s world, it’s not OK to be bad at communicating! However, there’s a misconception that goes around about tech folks. That misconception says that they’re all reclusive nerds who have difficulty communicating with people who don’t have the same background. That couldn’t be further from the truth. You should expect your Site Reliability Engineers to be just as good at communication as anyone else on your team. You need that good communication because when things break down, you need them to tell you what’s going on. If they’re good communicators, you’ll reduce your team’s overall stress levels. If they’re bad communicators, the opposite will be true.
After Hiring an SRE team
The hard work of building a team doesn’t end when you make a hire.
Regularly Checking In
SRE work, especially for a new team, can feel very stressful. Often times, a SRE engineer is trying to create a whole new system to support an existing process people are used to working with. This new system is going to be better. Both the engineer and the people they’re supporting know that. However, the reality is that getting to that new system requires taking some bruises along the way. That vision you had of a high-quality first year for your new hire? They’re almost certainly not going to cross every item off your list. There are going to be some failures along the way, and this can feel demoralizing. You can support your new employee by regularly checking in with them and providing feedback on their performance. Stress-free employees do better work than stressed employees. It’s your job to make help them manage that stress.
Keep Training Them Up
Another common mistake from teams that hire people to do SRE work is that they don’t know how to train them. Someone comes in, and they point them at a new system, and say “build that.” The new hires are expected to handle learning things all by themselves. Needless to say, this is not a tactic with a spotless track record. Instead, you should be working with your employees, at those regular check-ins, to find out what they don’t know yet. It’s likely that you’re going to put them in position to do things they don’t know how to do yet. If you’re communicating with them about those tasks, you’ll quickly learn what they need to learn. Work with them to make sure they have the resources they need to tackle those new challenges. That might be formal training, or it might be partnering with a more experienced employee. Either way, don’t let your SRE team stagnate.
Empower Your Team to Do Their Best Work
However big your SRE team, following these steps will put them into a position to succeed. Being intentional before, during and after the hiring process gives you the best chance to build a team that transforms your business. Once you’ve hired that team, you want to give them the best tools possible to do their job. That’s where something like Enov8 comes in. Even with talented engineers and a solid support structure, you’ll still need help to manage the myriad new environments you’ll build. Using the right tools makes that a lot easier.
This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.
25 May, 2020 by Daniel Longest Zombie and ghost assets sound exciting, like a late-night movie you’d watch around Halloween. While in reality they may not be that exciting, they’re scary if you don’t understand and prevent them. The good news is the steps you need to...
We often get asked by people “What is TEM (Test Environment Management), well for those of you looking for a quick overview of Test Environment Management, here is Use Case we developed as a way…
19 MARCH, 2020 by Michiel Mulders SRE vs DevOps: Friends or Foes? Nowadays, there’s a lack of clarity about the difference between site reliability engineering (SRE) and development and operations (DevOps). There’s definitely an overlap between the roles, even though...
06 MARCH, 2020 by Arnab Roy Chowdhury Top 10 SRE Practices Do you know what the key to a successful website is? Well, you’re probably going to say that it’s quality coding. However, today, there’s one more aspect that we should consider. That’s reliability. There are...
20 FEBRUARY, 2020 by Arnab Row Chowdhury Technically, the world today has advanced to a level we never could’ve imagined a few years ago. What do you think made it possible? We now understand complexities. And how do you think that became possible? Literacy! Since...
14 FEBRUARY, 2020 by Michiel Mulders A site reliability engineer loves optimizing inefficient processes but also needs coding skills. He or she must have a deep understanding of the software to optimize processes. Therefore, we can say an SRE contributes directly to...