IT & Test Environment Management Anti Patterns #2 – Containers
By Sylvia Fronczak.
Anti-patterns often arise when developers find creative or outside-the-box uses for a technology – or when they find workarounds for limitations that were specifically put in place by the creators of the technology to prevent misuse.
Application container platforms like Docker have been the hot new thing for several years now, but they’re still fairly new concepts to many IT environment managers. And since they’ve been around for a while, there are a number of anti-patterns that environment managers have fallen into.
1. Confusing Application Containers with System Containers or VMs
Some developers make the mistake of comparing application containers with small VMs. This may seem inconsequential, but it actually lays a dangerous frame of reference.
The comparison between containers and VMs leads to developers running multiple processes and services, or even operating systems within application containers. However, containers don’t work like VMs. They should run one process or service.
System containers like LXD or OpenVZ explicitly allow developers to install operating systems inside the container. They operate much like virtual machines. However, VMs emulate hardware while system containers do not. What does this mean for us? If you’re looking to run Windows on your Linux, you’ll want a virtual machine. If you want to run multiple types of Linux-based operating systems on your server instead, then you can use system containers.
Why is this an anti-pattern? Application containers isolate processes; they don’t virtualize operating systems. They encapsulate parts of your application ecosystem into standalone deployable parts. If I have a Docker container that includes my web services, database, and message bus, I’m unable to scale or deploy different components of my stack.
And that results in blowing away one of the main features of containerization.
An application container should have only one service or process running in it. And that’s it.
2. Creating Tightly Coupled Dependencies
So now you’re ready to use application containers to deploy and run your application. You’ve properly segregated your services and functions into separate containers. What anti-pattern might you fall into next?
Coupling your containers to each other may seem innocent at first. It’s as simple as adding a depends-on or waits-for-it to your Docker configuration. However, this can result in long deployments, circular dependency, and reduced availability.
To get the most benefits from containerization, your containers should be independently runnable. This lets you destroy and create containers whenever necessary and lets you scale horizontally.
But what does independently runnable mean? Does it mean that my one process can’t be dependent on anything else? That would be fairly difficult to create meaningful apps this way. It means that your container and the process inside should not be dependent on anything else when starting up. Your container should never wait for other containers or dependencies to be up or accessible upon startup.
And again, this doesn’t mean that your application can’t depend on anything else. In fact, it should still throw errors if someone is using the container’s app and a critical dependency is down. This can be achieved in different ways. For one, don’t try to connect to another dependency on startup. Alternatively, consider using message buses to keep containers independently of each other.
3. Modifying Running Containers
Our next anti-pattern scolds those of us that like changing things on the fly.
Now that you have an independently running process in each of your containers, you may be tempted to update application configuration or even the application code inside.
However, a much better practice involves deploying a brand new version of the container. You should consider the container and the process running inside to be immutable. If you need to change a configuration, then change it in your Docker file or Helm chart and deploy the new version. If you want to change the application running, then deploy a new container with the new version of your code.
4. Including Unnecessary Services
Even if you follow the good patterns above, you may be tempted to add just a small service here or there to your containers.
Developers strive to make life easier for themselves. That includes adding tools and small services that are unrelated to the core functionality of a container. For example, a common anti-pattern here involves adding an SSH daemon to your container image. Not only does it encourage developers to change or tweak containers once they’re running, but they create another way for someone to try to gain access to your container. And if you’re not following the anti-patterns above, there should be no need for having an SSH daemon on your container anyway.
The more services you include in your container, the more bloated it becomes. Also, those extra services provide additional attack vectors for hackers.
Keep your containers light and simple. Don’t add anything that isn’t necessary to what functionality your service provides.
5. Persisting Data Inside a Container or Image Registry
It may seem easy and convenient to store data inside our containers. But once again, that’s a big no-no.
As a reminder, containers are ephemeral. They spin up and come down fairly frequently. They should never be considered as stable or long-lasting. If you rely on data or logs that are available only in the container, then you’re going to have a bad time.
Data doesn’t persist in containers by default. Additionally, you’re introducing another attack vector or security flaw. Anyone that gains access to the container or image registry may also gain access to your data.
Data shouldn’t be stored in containers. Instead, use Docker volumes or an external dependency to store your data and logs.
6. Relying on Backups and Restores of Containers
Our next anti-pattern might seem harmless at first. After all, we’ve all been trained to make sure everything is always backed up in case of emergencies. What’s wrong with that?
In and of themselves, backups are useful. However, some developers use them as a shortcut to deployments. They take backups of existing containers and restore them on new servers—circumventing their deployment pipeline.
There are some legitimate reasons why you may want to make a backup of your container. For example, in the event of a security breach in a container, you may want to back it up before destroying it for forensic analysis. However, restoring backups should not be part of your normal CI/CD or disaster recovery process. Instead, build and deploy a new version of your container and related services.
7. Giving Little Thought to Backward Compatibility
Whether your containers are directly dependent on other containers or dependencies like Kafka or RabbitMQ, you should always consider how changes will affect other systems.
The anti-pattern here involves relying on deployment order to coordinate upgrades and changes across containers and dependencies. One big example many of us are guilty of involves forgetting backward compatibility for database changes. If your web application relies on a database, it may be tempting to make changes in both and then coordinate deployments so that both are upgraded at the same time. Another step may involve a maintenance window. Here, services are taken down or made inaccessible. This goes against the high-availability promise of containerization.
When making changes to parts of your system, consider ways in which upgrading one or the other independently will work just fine. This may involve extra future deployments to remove unused code or columns, but it ensures more availability for your services. Give some thought to ways of enhancing and modifying systems so that other older versions of dependencies can still work—even if only temporarily.
8. Assuming Containers Will Fix Everything
I’d like to close out this list with an anti-pattern that’s less technical in nature. As lovers of technology, we’re often looking for the next great thing that will solve the limitations of our current platforms and tools. And this forward-looking experimentation is one of our strengths.
However, we also need to be realistic about the limitations of the tools we use, like application containers. They were built to solve a specific set of problems. And if we try to circumvent that by creating workarounds, we’ll end up creating tomorrow’s anti-pattern.
Author: Sylvia Fronczak.
This post was written by Sylvia Fronczak. Sylvia is a software developer that has worked in various industries with various software methodologies. She’s currently focused on design practices that the whole team can own, understand, and evolve over time.
14 FEBRUARY, 2020 by Michiel Mulders A site reliability engineer loves optimizing inefficient processes but also needs coding skills. He or she must have a deep understanding of the software to optimize processes. Therefore, we can say an SRE contributes directly to...
07 February, 2020 by Arnab Roy Chowdhury Do you remember what Uncle Ben said to young Peter Parker? “With great power comes great responsibility.” The same applies to companies. At present, businesses hold a huge amount of data—not only the data of a company but also...
17 JANUARY, 2020 by Sylvia Fronczak Site reliability engineering (SRE) uses techniques and approaches from software engineering to tackle reliability problems with a team’s operations and a site’s infrastructure. Knowing the history of SRE and understanding which...
25 JANUARY, 2020 by Michiel Mulders With the cost of data breaches increasing every year, there’s a huge need for higher security standards. According to IBM’s 2019 security report, the average total cost of a data breach has risen to $3.92 million per breach. It’s no...
08 DECEMBER, 2019 by Arnab Roy Chowdhury In the last few years, people have started to rely less on manual work and more on automation. Internet banking and online shopping portals are some examples of this growing trend of digitalization. Instead of going to the bank...
12 DECEMBER, 2019 by Carlos Schults The software development process today is very different from what it used to be 15, 20, or even more years ago. One of the most dramatic of such differences is undoubtedly the number and frequency of releases. Agile practices have...