6 ms·

Docker is a poorly engineered and over-hyped technology. The concept is great - and in fact, many companies have built great tooling around Linux cgroups. It l

by nvarsj 10y ago

Docker is a poorly engineered and over-hyped technology. The concept is great - and in fact, many companies have built great tooling around Linux cgroups. It lets you efficiently binpack applications on a single server - which is why 'containers' were created in the first place. The side benefit of letting you define your OS libraries, and other things, is a nice bonus, and way overblown in my opinion. Docker and its tooling is just plain bad. It's so bloody unstable, unless you pick some esoteric combination of versions and storage backends (old docker + old aufs + old ubuntu seems to do the trick) - and even then, you'll run into problems. Documentation won't help you here - it's trial and error. The docker group seems focused on feature, feature, feature, while the basic stability and performance remains poor. It's quite amazing how they get it so wrong. Browse through the github issues for performance and hanging issues. The surrounding tooling is poor as well - registry v2 doesn't even support deletes (because they need to maintain compatibility with the 100000 different storage backends for it). There is no LTS release, bug fixes only happen in latest version, so you're in a constant state of brokenness. And so on.

ChoHag 10y ago
The concept is great but it's also not original. It's called "processes". Docker is little more than a mass of complication laid atop fork+exec. That's why nobody can get it right - because we already did.
- IshKebab 10y ago
  I totally agree. The real issue is dynamic libraries and how hard it is to compile C/C++ code statically with GCC. If you could just pass `-static` to gcc and it actually worked like you expect this would never have happened. Fortunately that seems to be changing somewhat. Go is totally static, and Rust can easily be made totally static using muscl. You can even do totally static C/C++ apps fairly easily with muscl.
  - jjnoakes 10y ago
    Dynamic libraries (in the C/C++ sense) only scratch the surface. Containers give you your own file system namespace (among other namespaces), which means all of the files that make up your complicated application unit can be put together and work together in isolation, separate from the machine's main file system.
    - jalfresi 10y ago
      Isn't that chroot jails?
      - icebraining 10y ago
        Yes, quite literally, in fact: https://github.com/opencontainers/runc/blob/ee992e5ff7143ea3fedb1bb4aa88a41d65a0bd66/libcontainer/rootfs_linux.go#L616 https://github.com/opencontainers/runc/blob/ee992e5ff7143ea3...
    - nix0n 10y ago
      What advantages do file system namespaces have over separating by directories and users?
      - acdha 10y ago
        A big one is managing software which you didn't write: if you have two things which expect to be able to write to /etc/mydaemon.conf etc. you either need to burn a VM for each one, fork the startup scripts or take the Debian-style approach of maintaining patches which make everything configurable, or manage something like maintaining chroots directory hierarchies. (repeat for network namespacing: it's really nice not to need to play games to have your CI server start 3 running jobs which all think they're listening on port 80) None of that is impossible – in the case of chroot there's many years of precedent - but if you do it regularly, there's a strong appeal to automating a common pattern. This is especially true when your goal is supporting development teams: with something like Docker, normal users don't need root just to start a daemon on a privileged port or write to a couple of files. If you work in a large or security-conscious environment, that's a fairly big draw. (Not saying that Docker is perfect or necessarily the long-term winner in this space, only that there's a usability gap which a lot of people fall into).
- jjnoakes 10y ago
  How is "your own network, your own view of the file system, your own view of the process table, your own view of the user IDs, ..." the same as "processes"?
  - elsonrodriguez 10y ago
    On modern Linux distros every process is running in a cgroup and namespace by default. So these days the main difference between a "container" and a regular process is that regular processes are all jumbled together in the same root namespace, and containers are in separate namespaces.
    - jjnoakes 10y ago
      Which Linux distributions do this? I'd like to read how "most" instances of fork and exec end up with unique networks and file system namespaces.
      - elsonrodriguez 10y ago
        I didn't say unique, I said "jumbled together". Now as far as which distros put processes in a namespace and cgroup by default, I know at least CentOS 7 and Ubuntu 15 do this. And those two distros on their own would qualify for "most". To check if your distro does it, one way of checking is just doing a `cat /proc/1/cgroup`. This will show you what cgroups process 1 is in. By default you will be in the "root" cgroup. To check your namespaces, `ls -l /proc/1/ns/`. You'll see the process is in some randomly generated namespace ID per item. I'm sure you could recompile your kernel to disable this behavior, but the default reality of modern Linux is that everything is already running in a "container". Now the question is whether or not people want to take advantage of that reality, and separate out processes in isolation, or keep running everything on a system in such a way that any single process can impact the whole system.
  - zeveb 10y ago
    Plan 9 is knocking on the door and would like to have a word …
- brazzledazzle 10y ago
  Wouldn't it be more akin to jails?
DasIch 10y ago
I think the real problem with Docker is Docker Inc. The pressures and constraints the company is under promotes the creation of new features that can be marketed and new software that competes with products from the competition. They have effectively no incentive on getting rid of bugs in the core product or to test features they do add exhaustively. Sooner or later someone actually using containers will produce a docker replacement that will take over unless Docker focusses on what actually matters.
- xorcist 10y ago
  Docker Inc. desperately wants to be VMware. I don't think it's a coincidence that the official documentation is so full of virtualization-like choice of words you'd be forgiven if you thought they already were. They do however seem to be cutting so many corners technically, that the risk is they get undercut on their core offering.
Leynos 10y ago
Deletion from registry v2 is definitely supported now. It's just a total PITA, and took them a while to implement. But I got my disc space back, so I'm not complaining. :P
- ptman 10y ago
  Any pointers?
  - Leynos 10y ago
    This answer on SO is more or less the approach I used: http://stackoverflow.com/a/37716286/308278 http://stackoverflow.com/a/37716286/308278
vacri 10y ago
We used Docker sort of early on, and got out of it around v1.6. Many problems that you had to work around yourself, but the one you just reminded me of was the repos: There was Dockerhub, a third-party place without 100%(-ish) uptime, and two serve-your-own, one of which was a black box you could never delete from, and the other had stamped "NOT FOR PRODUCTION USE" in big letters on its github page. I remember one of the engineers giving a talk, saying that the problem they had was growing too big too quickly - they didn't have time to properly work out the base architecture in the early days.
nine_k 10y ago
Ok, throw Docker away. Did anybody have better experience with e.g. Rkt?
- senex 10y ago
  I really like: daemontools + static binary + setuidgid and maybe chroot. Or mesos where everything needs to be in a tarball, which is extracted and sole program run. If stuff is statically linked and related files (config, assets, ...) are part of bundle that can be chrooted, what is the value add of a container?
  - brazzledazzle 10y ago
    Containers are easier for folks with less operational experience to understand. Or maybe easier to get started with is a better way to put it. It's easy to underestimate easy to use tooling when you aren't the target demographic.
  - JdeBP 10y ago
    It's not hard to use control groups in daemontools family style, either. * http://jdebp.eu./Softwares/nosh/guide/move-to-control-group.html http://jdebp.eu./Softwares/nosh/guide/move-to-control-group....

by nvarsj 10y ago

ChoHag 10y ago

The concept is great but it's also not original. It's called "processes". Docker is little more than a mass of complication laid atop fork+exec. That's why nobody can get it right - because we already did.

IshKebab 10y ago
I totally agree. The real issue is dynamic libraries and how hard it is to compile C/C++ code statically with GCC. If you could just pass `-static` to gcc and it actually worked like you expect this would never have happened. Fortunately that seems to be changing somewhat. Go is totally static, and Rust can easily be made totally static using muscl. You can even do totally static C/C++ apps fairly easily with muscl.
- jjnoakes 10y ago
  Dynamic libraries (in the C/C++ sense) only scratch the surface. Containers give you your own file system namespace (among other namespaces), which means all of the files that make up your complicated application unit can be put together and work together in isolation, separate from the machine's main file system.
  - jalfresi 10y ago
    Isn't that chroot jails?
    - icebraining 10y ago
      Yes, quite literally, in fact: https://github.com/opencontainers/runc/blob/ee992e5ff7143ea3fedb1bb4aa88a41d65a0bd66/libcontainer/rootfs_linux.go#L616 https://github.com/opencontainers/runc/blob/ee992e5ff7143ea3...
  - nix0n 10y ago
    What advantages do file system namespaces have over separating by directories and users?
    - acdha 10y ago
      A big one is managing software which you didn't write: if you have two things which expect to be able to write to /etc/mydaemon.conf etc. you either need to burn a VM for each one, fork the startup scripts or take the Debian-style approach of maintaining patches which make everything configurable, or manage something like maintaining chroots directory hierarchies. (repeat for network namespacing: it's really nice not to need to play games to have your CI server start 3 running jobs which all think they're listening on port 80) None of that is impossible – in the case of chroot there's many years of precedent - but if you do it regularly, there's a strong appeal to automating a common pattern. This is especially true when your goal is supporting development teams: with something like Docker, normal users don't need root just to start a daemon on a privileged port or write to a couple of files. If you work in a large or security-conscious environment, that's a fairly big draw. (Not saying that Docker is perfect or necessarily the long-term winner in this space, only that there's a usability gap which a lot of people fall into).
jjnoakes 10y ago
How is "your own network, your own view of the file system, your own view of the process table, your own view of the user IDs, ..." the same as "processes"?
- elsonrodriguez 10y ago
  On modern Linux distros every process is running in a cgroup and namespace by default. So these days the main difference between a "container" and a regular process is that regular processes are all jumbled together in the same root namespace, and containers are in separate namespaces.
  - jjnoakes 10y ago
    Which Linux distributions do this? I'd like to read how "most" instances of fork and exec end up with unique networks and file system namespaces.
    - elsonrodriguez 10y ago
      I didn't say unique, I said "jumbled together". Now as far as which distros put processes in a namespace and cgroup by default, I know at least CentOS 7 and Ubuntu 15 do this. And those two distros on their own would qualify for "most". To check if your distro does it, one way of checking is just doing a `cat /proc/1/cgroup`. This will show you what cgroups process 1 is in. By default you will be in the "root" cgroup. To check your namespaces, `ls -l /proc/1/ns/`. You'll see the process is in some randomly generated namespace ID per item. I'm sure you could recompile your kernel to disable this behavior, but the default reality of modern Linux is that everything is already running in a "container". Now the question is whether or not people want to take advantage of that reality, and separate out processes in isolation, or keep running everything on a system in such a way that any single process can impact the whole system.
- zeveb 10y ago
  Plan 9 is knocking on the door and would like to have a word …
brazzledazzle 10y ago
Wouldn't it be more akin to jails?

DasIch 10y ago

I think the real problem with Docker is Docker Inc. The pressures and constraints the company is under promotes the creation of new features that can be marketed and new software that competes with products from the competition. They have effectively no incentive on getting rid of bugs in the core product or to test features they do add exhaustively. Sooner or later someone actually using containers will produce a docker replacement that will take over unless Docker focusses on what actually matters.

xorcist 10y ago
Docker Inc. desperately wants to be VMware. I don't think it's a coincidence that the official documentation is so full of virtualization-like choice of words you'd be forgiven if you thought they already were. They do however seem to be cutting so many corners technically, that the risk is they get undercut on their core offering.

Leynos 10y ago

Deletion from registry v2 is definitely supported now. It's just a total PITA, and took them a while to implement. But I got my disc space back, so I'm not complaining. :P

ptman 10y ago
Any pointers?
- Leynos 10y ago
  This answer on SO is more or less the approach I used: http://stackoverflow.com/a/37716286/308278 http://stackoverflow.com/a/37716286/308278

vacri 10y ago

We used Docker sort of early on, and got out of it around v1.6. Many problems that you had to work around yourself, but the one you just reminded me of was the repos: There was Dockerhub, a third-party place without 100%(-ish) uptime, and two serve-your-own, one of which was a black box you could never delete from, and the other had stamped "NOT FOR PRODUCTION USE" in big letters on its github page. I remember one of the engineers giving a talk, saying that the problem they had was growing too big too quickly - they didn't have time to properly work out the base architecture in the early days.

nine_k 10y ago

Ok, throw Docker away. Did anybody have better experience with e.g. Rkt?

senex 10y ago
I really like: daemontools + static binary + setuidgid and maybe chroot. Or mesos where everything needs to be in a tarball, which is extracted and sole program run. If stuff is statically linked and related files (config, assets, ...) are part of bundle that can be chrooted, what is the value add of a container?
- brazzledazzle 10y ago
  Containers are easier for folks with less operational experience to understand. Or maybe easier to get started with is a better way to put it. It's easy to underestimate easy to use tooling when you aren't the target demographic.
- JdeBP 10y ago
  It's not hard to use control groups in daemontools family style, either. * http://jdebp.eu./Softwares/nosh/guide/move-to-control-group.html http://jdebp.eu./Softwares/nosh/guide/move-to-control-group....