IT & Test Environment Management Anti Patterns #2
By Sylvia Fronczak.
1. Confusing Application Containers with System Containers or VMsSome developers make the mistake of comparing application containers with small VMs. This may seem inconsequential, but it actually lays a dangerous frame of reference.
Anti-PatternThe comparison between containers and VMs leads to developers running multiple processes and services, or even operating systems within application containers. However, containers don’t work like VMs. They should run one process or service. System containers like LXD or OpenVZ explicitly allow developers to install operating systems inside the container. They operate much like virtual machines. However, VMs emulate hardware while system containers do not. What does this mean for us? If you’re looking to run Windows on your Linux, you’ll want a virtual machine. If you want to run multiple types of Linux-based operating systems on your server instead, then you can use system containers. Why is this an anti-pattern? Application containers isolate processes; they don’t virtualize operating systems. They encapsulate parts of your application ecosystem into standalone deployable parts. If I have a Docker container that includes my web services, database, and message bus, I’m unable to scale or deploy different components of my stack. And that results in blowing away one of the main features of containerization.
PatternAn application container should have only one service or process running in it. And that’s it.
2. Creating Tightly Coupled DependenciesSo now you’re ready to use application containers to deploy and run your application. You’ve properly segregated your services and functions into separate containers. What anti-pattern might you fall into next?
Anti-PatternCoupling your containers to each other may seem innocent at first. It’s as simple as adding a depends-on or waits-for-it to your Docker configuration. However, this can result in long deployments, circular dependency, and reduced availability.
PatternTo get the most benefits from containerization, your containers should be independently runnable. This lets you destroy and create containers whenever necessary and lets you scale horizontally. But what does independently runnable mean? Does it mean that my one process can’t be dependent on anything else? That would be fairly difficult to create meaningful apps this way. It means that your container and the process inside should not be dependent on anything else when starting up. Your container should never wait for other containers or dependencies to be up or accessible upon startup. And again, this doesn’t mean that your application can’t depend on anything else. In fact, it should still throw errors if someone is using the container’s app and a critical dependency is down. This can be achieved in different ways. For one, don’t try to connect to another dependency on startup. Alternatively, consider using message buses to keep containers independently of each other.
3. Modifying Running ContainersOur next anti-pattern scolds those of us that like changing things on the fly.
Anti-PatternNow that you have an independently running process in each of your containers, you may be tempted to update application configuration or even the application code inside.
PatternHowever, a much better practice involves deploying a brand new version of the container. You should consider the container and the process running inside to be immutable. If you need to change a configuration, then change it in your Docker file or Helm chart and deploy the new version. If you want to change the application running, then deploy a new container with the new version of your code.
4. Including Unnecessary ServicesEven if you follow the good patterns above, you may be tempted to add just a small service here or there to your containers.
Anti-PatternDevelopers strive to make life easier for themselves. That includes adding tools and small services that are unrelated to the core functionality of a container. For example, a common anti-pattern here involves adding an SSH daemon to your container image. Not only does it encourage developers to change or tweak containers once they’re running, but they create another way for someone to try to gain access to your container. And if you’re not following the anti-patterns above, there should be no need for having an SSH daemon on your container anyway. The more services you include in your container, the more bloated it becomes. Also, those extra services provide additional attack vectors for hackers.
PatternKeep your containers light and simple. Don’t add anything that isn’t necessary to what functionality your service provides.
5. Persisting Data Inside a Container or Image RegistryIt may seem easy and convenient to store data inside our containers. But once again, that’s a big no-no.
Anti-PatternAs a reminder, containers are ephemeral. They spin up and come down fairly frequently. They should never be considered as stable or long-lasting. If you rely on data or logs that are available only in the container, then you’re going to have a bad time. Data doesn’t persist in containers by default. Additionally, you’re introducing another attack vector or security flaw. Anyone that gains access to the container or image registry may also gain access to your data.
PatternData shouldn’t be stored in containers. Instead, use Docker volumes or an external dependency to store your data and logs.
6. Relying on Backups and Restores of ContainersOur next anti-pattern might seem harmless at first. After all, we’ve all been trained to make sure everything is always backed up in case of emergencies. What’s wrong with that?
Anti-PatternIn and of themselves, backups are useful. However, some developers use them as a shortcut to deployments. They take backups of existing containers and restore them on new servers—circumventing their deployment pipeline.
PatternThere are some legitimate reasons why you may want to make a backup of your container. For example, in the event of a security breach in a container, you may want to back it up before destroying it for forensic analysis. However, restoring backups should not be part of your normal CI/CD or disaster recovery process. Instead, build and deploy a new version of your container and related services.
7. Giving Little Thought to Backward CompatibilityWhether your containers are directly dependent on other containers or dependencies like Kafka or RabbitMQ, you should always consider how changes will affect other systems.
Anti-PatternsThe anti-pattern here involves relying on deployment order to coordinate upgrades and changes across containers and dependencies. One big example many of us are guilty of involves forgetting backward compatibility for database changes. If your web application relies on a database, it may be tempting to make changes in both and then coordinate deployments so that both are upgraded at the same time. Another step may involve a maintenance window. Here, services are taken down or made inaccessible. This goes against the high-availability promise of containerization.
PatternsWhen making changes to parts of your system, consider ways in which upgrading one or the other independently will work just fine. This may involve extra future deployments to remove unused code or columns, but it ensures more availability for your services. Give some thought to ways of enhancing and modifying systems so that other older versions of dependencies can still work—even if only temporarily.
8. Assuming Containers Will Fix EverythingI’d like to close out this list with an anti-pattern that’s less technical in nature. As lovers of technology, we’re often looking for the next great thing that will solve the limitations of our current platforms and tools. And this forward-looking experimentation is one of our strengths. However, we also need to be realistic about the limitations of the tools we use, like application containers. They were built to solve a specific set of problems. And if we try to circumvent that by creating workarounds, we’ll end up creating tomorrow’s anti-pattern.
09 SEPTEMBER, 2022 by Michiel MuldersDo you want your company to scale efficiently? Look for an enterprise release manager (ERM). An ERM protects and manages the movements of releases in multiple environments. This includes build, test, and production environments....
22 August, 2022 by Louay Hazami *Update from October 2020Data privacy is one of the most pressing issues in the new digital era. Data holds so much value for normal internet users and for all types of companies that are looking to capitalize on this new resource. To...
16August, 2022 by Carlos Schults *Update from 15 Mar 2021In today's post, we'll answer what looks like a simple question: what is data fabrication in TDM? That's such an unimposing question, but it contains a lot for us to unpack.What is TDM to begin with? Isn't data...
08August, 2022 by Carlos Schults *Update from 26 Nov 2019.Your Essential TEM Checklist "Test Environment Management Checklist." Yep, that sounds like a mouthful, but don't let that discourage you. The idea here is quite simple—adopting a checklist to evaluate...
03JUNE, 2022 by Niall Crawford & Carlos "Kami" Maldonado. Modified by Eric Goebelbecker.DevOps at scale is what we call the process of implementing DevOps culture at big, structured companies. Although the DevOps term was back in 2009, most organizations still...
3JUNE, 2022 by Erik Dietrich, Ukpai Ugochi, and Jane Temov. Modified by Eric GoebelbeckerMost companies spend between 45%-55% of their IT budget on non-production activities like Training, Development & Testing and lose 20-40% of productivity across their...