3 ms·
Many of the other comments here are kinda generic, like "don't use anything unless you need it". Here are some specific problems with Docker that I've seen bite
by native_samples 5y ago
Many of the other comments here are kinda generic, like "don't use anything unless you need it". Here are some specific problems with Docker that I've seen bite people in the ass over the years, including at startups.
1. DATA DESTRUCTION
Docker is unsafe by default. If you don't take specific measures by setting up bind mounts etc, then it is very easy to get into a state where shutting down the container deletes all the files your server wrote inside it.
This is especially nasty when the program inside the container has generated a private key that was then authorized to do something, and you lose it. Yes, I've seen this happen. What a mess.
2. PERFORMANCE
Docker on Linux is a relatively thin layer over kernel features and is reasonably fast. Docker on everything else is excruciatingly slow to the point that it can break things. In particular, doing anything with Docker on macOS can be so slow that it yields an unusable workflow for developers. It's very common in startups (and these days even bigger firms) for devs to be able to choose their own OS, meaning it's easy for someone to Dockerize stuff in a fit of enthusiasm and then people on Mac or Windows discover that it's no longer pleasant to develop on, or may not even work properly at all. Filesystem bridging is a particular pain point.
3. DISK SPACE LEAKS
Docker likes to download lots of very similar operating systems even when you already have one that works fine, and is very poor at deduplicating files. This can lead to Dockerized systems that appear to work for a while and then one day just go bam because they ran out of disk space.
4. BIZARRE AND UNINTUITIVE CACHING SEMANTICS
What's the difference between these two Dockerfile snippets?
RUN apt-get update
RUN apt-get install xyz abc
and
RUN apt-get update && apt-get install xyz abc
It looks superficially like there's no difference because a Dockerfile is sort of like a shell script that's setting up an OS image. But the first approach is subtly broken in ways that won't become apparent for a few months. The problem is that each and every command you run creates a new snapshot, and Docker assumes that every command is a pure functional transform of the prior state. This is incorrect. So what happens is the apt cache will be snapshotted and when you add another package to the install line, it will try and use an out of date cache, failing because the mirror operators have removed the old versions.
5. SECURITY PROBLEMS
5a. Docker requires fairly complex networking configuration especially once you get into containers talking to each other. It is easy to screw this up and accidentally expose your unprotected database socket to the whole world, whilst believing it's firewalled. For example Docker has been known to actually remove firewall rules in the past, from live production systems.
5b. Docker images snapshot an entire OS meaning it won't get any security updates unless you are careful to continually rebuild them. It's easy to screw this up such that no updates are actually applied (see point 4), but in practice hardly anyone does do this constant upgrading so stale Docker images are such a large problem that whole startups exist to try and tackle it.
6. LACK OF SERVICE MANAGEMENT FEATURES
Docker is metadata poor. It has no good way to express startup dependencies between containers, cannot express sandboxing rules and many other problems. People often think containers are sandboxes, but they aren't actually designed to be so, and this has also led to nasty problems in the past.
7. USER/GROUP FLAKYNESS
Docker containers have their own user/group namespace, although this is rarely what you want. It's easy to end up in a situation where software fails in subtle ways because the uid/gid database inside the container is wrong or missing information, because home directories aren't aligned, or because uids/gids outside the container aren't the same as those inside but they're sharing a filesystem.
Example: the JVM queries the home directory of the user at startup. It can be the case that the user doesn't have one, when using a container, so it sets the system property to '?' which no real Java software checks for. Cue lots of files and directories named '?' appearing. Have fun tracking that one down!
---
A common theme in the above is that Docker makes assumptions about UNIX (really, all operating systems) that don't reflect how they actually work. This is a good sign that it will blow up in some obscure way. So if Docker has these problems, what's the alternative? Well, for what you describe I'd just use systemd with tarballs or if you want a bit more support, DEBs. This is what I do and it works fine. Why, well:
SystemD is conceptually simple and clean. It is metadata rich, so you can do things like say one service depends on another. SystemD sandboxing is not an accident, it's an explicit design feature with plenty of knobs to give you what you need, but it's also very easy to bring up a simple systemd service in a normal UNIX environment as a starting point. Because SystemD is designed to work with normal software it won't do things like delete files your service created, just because it shut down, and it won't delete firewall rules just because it can.
For software distribution, rsync works fine, but DEB/RPM add support for running scripts on upgrade which can be convenient. They can also pull in dependencies, and the operating system can keep those dependencies up to date for you automatically if you want. Or you can pin all your dependencies to specific versions and take complete control. Although automatic upgrades can make the environment less reproducible, if you are small and only have a few machines in the first place it doesn't really matter because you'd be upgrading one machine and letting it soak for a bit before doing the others anyway, so keeping your machines identical is not going to happen.
- juanse 5y agoYou really made a case crystal clear. I appreciate you shared this point of view. Now I have no doubts.