12 ms·

Docker. I use it and love it every day in both dev and prod, but I also really kind of hate it. I'll keep my complaints short. There should not be a system-w

Docker. I use it and love it every day in both dev and prod, but I also really kind of hate it. I'll keep my complaints short. There should not be a system-wide daemon. (Or any daemon). It should not require root at all (no setuid either). From outside the container, the container and its processes should be a single process (with threads). (Like glueing a bunch of processes together.) The containers should be nest-able to arbitrary depth without performance loss (at least to say, hundreds of nestings deep.) Docker-compose should not exist, instead it should be replaced by nesting of containers. Basically, I think it needs to follow the UNIX philosophy better by providing simple abstractions that can be combined easily. The containers would visually look a bit more like an old virtual machine (single process) than our current containers. These changes probably require a bunch of kernel hacking, but I think it would be worth it long-term for a cleaner architecture. It appears there are some movements into this direction thanks to podman, but it's really not there yet, especially with nesting. Also, it wouldn't really be a product at all but just a built-in tool on Linux systems.

heliosfire 6y ago
Some good stuff here, however... Strong disagree on compose. I think it’s amazing.
- _-___________-_ 6y ago
  I find people only love compose if it's the first exposure they had to container orchestration. Orchestration is a great concept but basically everything else that solves this problem solves it much better than docker-compose.
  - lifty 6y ago
    Can you mention what are the tools that you use as alternatives to compose?
GordonS 6y ago
I agree about the daemon. Podman is a daemonless alternative, though I've never used it myself. Strongly disagree about Docker Compose though - I actually really like the ability to compose a stack of different containers together with some simple yaml.
- etaioinshrdlu 6y ago
  You'd still be able to compose a bunch of containers together, but it would result in a new, single container due to nesting. It could even be compatible with docker-compose and it's yaml.
- fsociety 6y ago
  Haven’t used Podman in production but at home it’s a huge improvement over Docker and enjoyable to use
- mceachen 6y ago
  I enjoyed docker compose as well (enough to use it for PhotoStructure), but was bit by breaking changes even when I had specified a version in my docker-compose.yml. It meant that a bunch of my beta users suddenly had broken PhotoStructure configurations because their docker-compose implementation had received a minor update. Why require a version to your configuration file and not increment it on breaking changes? I ended up tearing out the script that helped people create their own docker-compose.yml file, and replaced the installation instructions with an annotated call to `docker run`. And don't get me started on how janky it is to update existing containers to new images without docker-compose: there seems to only be one third-party tool to assist with this automatically (lighthouse), but is essentially abandoned. I'd love to be wrong about this, please point me to other solutions if they exist!
  - BrandoElFollito 6y ago
    I use watchtower with docker compose and the update is seamless.
kuiro5 6y ago
Strongly agree about this. Docker is conceptually on the right track, but it's fundamentally the wrong abstraction.
- still_grokking 6y ago
  Even I would like to tune in to Docker bashing (in this case one can actually say with confidence: "Hitler was right"[1]) the fundamental architectural problem is on the OS side. UNIX, and especially Linux, is a monolithic design. Even such an OS is able to separate user processes form each other all system parts run by concept in the form of a "big ball of mud", with "god-like" capabilities available to them by default. Sure, some internal "barriers" have been added, and per process capability dropping has been retrofitted, but this is backwards form the architectural point of view. Cutting things in peaces after the fact is almost always way more complicated and awkward compared to designing things in a modular way form the get go. This is related as virtualizing a modular OS is almost a no-brainer (conceptually). You just need to start additional instances of the required system servers / modules / whatever-you-call-that-parts. Compared to that virtualizing a monolith is like trying to construct a kind of Ouroboros: It needs to run itself (with an altered, usually constrained view on the 'outside' world) from inside of itself; and it can't just globally drop the "god-like capabilities" its execution context provides—like it would be possible with an external process. It needs to "hide or manipulate things in front of its own eyes" even "it" has the "all seeing eye". Or to put it even more metaphorical: "A God tries to use his divine powers to constrain his omnipotence so he can lie to himself about the things he sees, without himself ever being able to look through this jugglery". Formulated like that the architectural issue is obvious, I guess. [1] https://www.youtube.com/watch?v=PivpCKEiQOQ https://www.youtube.com/watch?v=PivpCKEiQOQ , and I just learned it seems he was also a Kubernetes fan. :-)
  - neillyons 6y ago
    I was thinking recently all these layers of abstraction is crazy. I would be simpler to write a Python web server using micro python and deploy to a server farm of esp32 micro controllers. No idea if this is possible.
ypcx 6y ago
For me one of the worst were Docker for Desktop on both Mac and Windows, especially when used for local Kubernetes. I fixed this with a project running Kubernetes directly on a local virtual machine(s) and local Docker (without polluting the machine with Docker for Desktop) is a bonus that comes with that[1]. [1] https://github.com/youurayy/hyperctl https://github.com/youurayy/hyperctl
- rikroots 6y ago
  When Docker for Desktop came out, I refused to move from Docker Toolbox. My reasoning was probably illogical but my view was that if you're maintaining a number of projects, each of which runs in their own Docker environment and some of which had duplicate container names, then having the power to start/stop different Docker engines for each just made a lot more sense to me. I don't miss Docker.
baddox 6y ago
Docker is also essentially completely broken on MacOS and has been for years. The performance penalty on anything doing I/O is like 5x, and it tends to completely hammer my CPU. There are tons of internet discussions, so it appears to not be just me.
- yjftsjthsd-h 6y ago
  Isn't that inherent because Docker needs a Linux kernel, so running it on Darwin has a hard requirement on virtualization and running an entire guest operating system?
  - jlokier 6y ago
    Virtualization and even running a guest kernel and OS isn't 5x expensive on the CPU, so the problem is not virtualization per se. Perhaps the problem is Darwin.
    - yjftsjthsd-h 6y ago
      I know it's not bad on CPU, but I thought it took more of a hit on I/O? Granted, I suspect that some of this is indeed some issue with Darwin itself, or at least poor integration with it.
      - jlokier 6y ago
        I agree, I/O can be pretty expensive. It depends quite a lot on the kind of I/O and how it's implemented, so design choices matter. If there's a full Linux kernel inside the VM, then you may as well do I/O inside the VM as well, using something like virtio and a ringbuffer of async block device commands to the host, or at least batching them. That will be quicker than relaying every POSIX file operation synchronously to the host, because the number of VM exits is much lower in the former case.
    - arp242 6y ago
      Docker abstracts and re-implements a lot of the filesystem and IO stuff; almost all of the issues (often outright bugs) I've had with Docker on Linux are related to that, too.
alhirzel 6y ago
Give Singularity a look
- walleeee 6y ago
  Yep. Doesn't have a daemon or require root.
btilly 6y ago
If you want more complaints, and well informed ones at that, read https://www.linkedin.com/pulse/containers-future-ian-eyberg/ https://www.linkedin.com/pulse/containers-future-ian-eyberg/. I particularly love the quote, The kernel developers view of the docker community is that in the rare case they can actually formulate the question correctly they usually don't understand the answer. There is only so much that you can say to clarify things to someone who is thinking about everything wrong and doesn't realize it. :-(
- breatheoften 6y ago
  Interesting thread. Is there a microvm that can run chromium with puppeteer? I've been thinking that server side chromium might actually turn into a pretty badass application server platform ... security, async, remote debug, webasm for cross platform secure binaries ... Some efficient infrastructure for deploying is needed -- but should be far easier to create a fast server runtime for puppeteer+chromium than it is to create a generic container execution environment ... -- so the microvm approach seems like the right one for what i want ...
  - reichardt 6y ago
    Something like https://workers.cloudflare.com/ https://workers.cloudflare.com/ ?
- aloer 6y ago
  That article seems overly critical about young developers that don't know it any better because they grew up on containers. I guess I am one of those so I got to ask, is the proposed solution of unikernels something we had before but lost in favor of containers, or is it something completely new anyways? It does look like it might be the latter so why blame developers for using containers due to lack of choice? If unikernels are better and just as easy to use then I am sure people will convert. He blames a lot on marketing and marketing lies but his company (https://nanovms.com/ https://nanovms.com/) seems to make it just as hard to figure out what's going on with the apparently only option being a schedule a demo button. Come on, I remember Docker being that fancy new thing that people at university taught themselves and to each other around ~2014/2015. That hype was well deserved and if you want to compete with that you can't just decide to brush it off as wrong and misguided. At the risk of pointing out that I also might be one of those that the quote above is referring to, I gotta ask: Is there a technical reason why I shouldn't be able to eventually just replace Docker with a micro or unikernel? Same or similar style of image definition, completely different runtime technology? Isn't it up to the kernel and platform developers to build the tools to make that happen comfortably for all of us naive container users?
pbar 6y ago
The reality is that containers via `runc` really _are_ just bundles of processes with some sugar to control Linux namespaces. Using another runtime (kata, etc) would get closer to the tighter abstraction you mention, but it would truly be a VM, just a small one.
hinkley 6y ago
I feel like the Dockerfile format was very tight and simple to use, and the tools somewhat usable, but over time they keep bolting things onto it by committee. It is better than git, but that's pretty faint praise. When I'm staring at the worst of it (unsticking myself or worse, trying to explain why it's like this to a coworker who is stuck), I keep thinking that there's a standard for making these containers, won't someone get around to rewriting the user-facing bits with the modern requirements designed in from the start? But it's good enough, so we are probably stuck with it until someone comes up with a better idea to base application compartmentalization upon. Like an OS that actually does what I was promised 25 years ago and am still waiting for.
- m463 6y ago
  I think the idea of a docker file is brilliant. But one pattern I see all the time is: RUN foo && bar && bletch && ... They should have a way of achieving the same thing (just one layer) without multiple commands and added to the same line
AsyncAwait 6y ago
I agree except for compose. I actually like compose and how simple it is, but am curious about container nesting. Could you elaborate? How would dependencies work there for example?
regularfry 6y ago
I was thinking through how you might do this, and my brain dredged up User-Mode Linux. A UML wrapper around those docker containers would behave almost exactly like you describe. You should (if I remember correctly) be able to nest them, too. I'm pretty sure this is doable today. It's a monstrous hack, and I've got no idea what the performance overheads would look like, but as a way of hiding a mess behind a clean facade, I'm not aware of any reason it shouldn't work.
- etaioinshrdlu 6y ago
  Nowadays, it seems like gVisor is doing something fairly similar to UML but a good deal more lightweight.
  - regularfry 6y ago
    Yeah, it looks like this is in the category of "things that could reasonably be done with some elbow-grease" rather than "things that are made impossible by some aspect of the technology".
ForHackernews 6y ago
You might want to check out Flockport: https://thenewstack.io/flockport-time-to-start-all-over-again-and-return-to-lxc-containers/ https://thenewstack.io/flockport-time-to-start-all-over-agai... They're trying to use built-in Linux LXC container features.
nikisweeting 6y ago
I like docker-compose so much that I'd use it even if the underlying containerization technology didn't exist. I think it's one of the things they got most right, and I wish there was a bigger ecosystem around managing compose-based projects.
- Rapzid 6y ago
  I think it could be better but its features became coupled with swarm. That and the primary maintainer was super hostile to feature requests..
the8472 6y ago
> It should not require root at all (no setuid either). The only thing that really needs setuid are network namespaces to setup the bridges. Userspace workarounds are clunky and slow. If you can do without network isolation then this would be possible. > The containers should be nest-able to arbitrary depth without performance loss (at least to say, hundreds of nestings deep.) Multiple levels of nesting are ppossible if you disable seccomp. I don't know if it scales to hundreds though. Overlayfs has hard limits and btrfs snapeshots don't scale infinitely either. > Also, it wouldn't really be a product at all but just a built-in tool on Linux systems. Well, there's systemd-nspawn and machinectl
- _-___________-_ 6y ago
  > > Also, it wouldn't really be a product at all but just a built-in tool on Linux systems. > Well, there's systemd-nspawn and machinectl There's also podman, which is a drop-in replacement for docker, and buildah, which does daemonless container builds. I switched to them from Docker recently and will never look back.
thrwaway69 6y ago
Try https://podman.io https://podman.io
musicale 6y ago
Yeah, docker is an anti-pattern in so many ways. Linux is partially to blame as well, since the container/isolation APIs can be hard to use correctly, and many people have latched on to docker as something that sort of works. It also seems to me that the failed security and isolation designs, and painful management and administration designs, of every mainstream operating system have been primary factors in pushing us toward VMs and containers in the first place.
- arp242 6y ago
  The isolation stuff is a bit arcane and confusing, but also not that hard to figure out or implement as a wrapper. I think the biggest reason people like Docker is that Docker makes it so easy to distribute containers.
- neop1x 6y ago
  You are probably forgetting about dependency hell, reproducible builds and automated testing. Waaaay tooo often some software won't compile because a bad version of the library is installed in the system or an incompatible compiler is used. Or two perl/python apps require different dependency version. All those problems are quickly solved by Docker. There were no better way to solve it, other than packing and versioning everything like in Chrome build process...

heliosfire 6y ago

Some good stuff here, however... Strong disagree on compose. I think it’s amazing.

_-___________-_ 6y ago
I find people only love compose if it's the first exposure they had to container orchestration. Orchestration is a great concept but basically everything else that solves this problem solves it much better than docker-compose.
- lifty 6y ago
  Can you mention what are the tools that you use as alternatives to compose?

GordonS 6y ago

I agree about the daemon. Podman is a daemonless alternative, though I've never used it myself. Strongly disagree about Docker Compose though - I actually really like the ability to compose a stack of different containers together with some simple yaml.

etaioinshrdlu 6y ago
You'd still be able to compose a bunch of containers together, but it would result in a new, single container due to nesting. It could even be compatible with docker-compose and it's yaml.
fsociety 6y ago
Haven’t used Podman in production but at home it’s a huge improvement over Docker and enjoyable to use
mceachen 6y ago
I enjoyed docker compose as well (enough to use it for PhotoStructure), but was bit by breaking changes even when I had specified a version in my docker-compose.yml. It meant that a bunch of my beta users suddenly had broken PhotoStructure configurations because their docker-compose implementation had received a minor update. Why require a version to your configuration file and not increment it on breaking changes? I ended up tearing out the script that helped people create their own docker-compose.yml file, and replaced the installation instructions with an annotated call to `docker run`. And don't get me started on how janky it is to update existing containers to new images without docker-compose: there seems to only be one third-party tool to assist with this automatically (lighthouse), but is essentially abandoned. I'd love to be wrong about this, please point me to other solutions if they exist!
- BrandoElFollito 6y ago
  I use watchtower with docker compose and the update is seamless.

kuiro5 6y ago

Strongly agree about this. Docker is conceptually on the right track, but it's fundamentally the wrong abstraction.

still_grokking 6y ago
Even I would like to tune in to Docker bashing (in this case one can actually say with confidence: "Hitler was right"[1]) the fundamental architectural problem is on the OS side. UNIX, and especially Linux, is a monolithic design. Even such an OS is able to separate user processes form each other all system parts run by concept in the form of a "big ball of mud", with "god-like" capabilities available to them by default. Sure, some internal "barriers" have been added, and per process capability dropping has been retrofitted, but this is backwards form the architectural point of view. Cutting things in peaces after the fact is almost always way more complicated and awkward compared to designing things in a modular way form the get go. This is related as virtualizing a modular OS is almost a no-brainer (conceptually). You just need to start additional instances of the required system servers / modules / whatever-you-call-that-parts. Compared to that virtualizing a monolith is like trying to construct a kind of Ouroboros: It needs to run itself (with an altered, usually constrained view on the 'outside' world) from inside of itself; and it can't just globally drop the "god-like capabilities" its execution context provides—like it would be possible with an external process. It needs to "hide or manipulate things in front of its own eyes" even "it" has the "all seeing eye". Or to put it even more metaphorical: "A God tries to use his divine powers to constrain his omnipotence so he can lie to himself about the things he sees, without himself ever being able to look through this jugglery". Formulated like that the architectural issue is obvious, I guess. [1] https://www.youtube.com/watch?v=PivpCKEiQOQ https://www.youtube.com/watch?v=PivpCKEiQOQ , and I just learned it seems he was also a Kubernetes fan. :-)
- neillyons 6y ago
  I was thinking recently all these layers of abstraction is crazy. I would be simpler to write a Python web server using micro python and deploy to a server farm of esp32 micro controllers. No idea if this is possible.

ypcx 6y ago

For me one of the worst were Docker for Desktop on both Mac and Windows, especially when used for local Kubernetes. I fixed this with a project running Kubernetes directly on a local virtual machine(s) and local Docker (without polluting the machine with Docker for Desktop) is a bonus that comes with that[1]. [1] https://github.com/youurayy/hyperctl https://github.com/youurayy/hyperctl

rikroots 6y ago
When Docker for Desktop came out, I refused to move from Docker Toolbox. My reasoning was probably illogical but my view was that if you're maintaining a number of projects, each of which runs in their own Docker environment and some of which had duplicate container names, then having the power to start/stop different Docker engines for each just made a lot more sense to me. I don't miss Docker.

baddox 6y ago

Docker is also essentially completely broken on MacOS and has been for years. The performance penalty on anything doing I/O is like 5x, and it tends to completely hammer my CPU. There are tons of internet discussions, so it appears to not be just me.

yjftsjthsd-h 6y ago
Isn't that inherent because Docker needs a Linux kernel, so running it on Darwin has a hard requirement on virtualization and running an entire guest operating system?
- jlokier 6y ago
  Virtualization and even running a guest kernel and OS isn't 5x expensive on the CPU, so the problem is not virtualization per se. Perhaps the problem is Darwin.
  - yjftsjthsd-h 6y ago
    I know it's not bad on CPU, but I thought it took more of a hit on I/O? Granted, I suspect that some of this is indeed some issue with Darwin itself, or at least poor integration with it.
    - jlokier 6y ago
      I agree, I/O can be pretty expensive. It depends quite a lot on the kind of I/O and how it's implemented, so design choices matter. If there's a full Linux kernel inside the VM, then you may as well do I/O inside the VM as well, using something like virtio and a ringbuffer of async block device commands to the host, or at least batching them. That will be quicker than relaying every POSIX file operation synchronously to the host, because the number of VM exits is much lower in the former case.
  - arp242 6y ago
    Docker abstracts and re-implements a lot of the filesystem and IO stuff; almost all of the issues (often outright bugs) I've had with Docker on Linux are related to that, too.

alhirzel 6y ago

Give Singularity a look

walleeee 6y ago
Yep. Doesn't have a daemon or require root.

btilly 6y ago

If you want more complaints, and well informed ones at that, read https://www.linkedin.com/pulse/containers-future-ian-eyberg/ https://www.linkedin.com/pulse/containers-future-ian-eyberg/. I particularly love the quote, The kernel developers view of the docker community is that in the rare case they can actually formulate the question correctly they usually don't understand the answer. There is only so much that you can say to clarify things to someone who is thinking about everything wrong and doesn't realize it. :-(

breatheoften 6y ago
Interesting thread. Is there a microvm that can run chromium with puppeteer? I've been thinking that server side chromium might actually turn into a pretty badass application server platform ... security, async, remote debug, webasm for cross platform secure binaries ... Some efficient infrastructure for deploying is needed -- but should be far easier to create a fast server runtime for puppeteer+chromium than it is to create a generic container execution environment ... -- so the microvm approach seems like the right one for what i want ...
- reichardt 6y ago
  Something like https://workers.cloudflare.com/ https://workers.cloudflare.com/ ?
aloer 6y ago
That article seems overly critical about young developers that don't know it any better because they grew up on containers. I guess I am one of those so I got to ask, is the proposed solution of unikernels something we had before but lost in favor of containers, or is it something completely new anyways? It does look like it might be the latter so why blame developers for using containers due to lack of choice? If unikernels are better and just as easy to use then I am sure people will convert. He blames a lot on marketing and marketing lies but his company (https://nanovms.com/ https://nanovms.com/) seems to make it just as hard to figure out what's going on with the apparently only option being a schedule a demo button. Come on, I remember Docker being that fancy new thing that people at university taught themselves and to each other around ~2014/2015. That hype was well deserved and if you want to compete with that you can't just decide to brush it off as wrong and misguided. At the risk of pointing out that I also might be one of those that the quote above is referring to, I gotta ask: Is there a technical reason why I shouldn't be able to eventually just replace Docker with a micro or unikernel? Same or similar style of image definition, completely different runtime technology? Isn't it up to the kernel and platform developers to build the tools to make that happen comfortably for all of us naive container users?

pbar 6y ago

The reality is that containers via `runc` really _are_ just bundles of processes with some sugar to control Linux namespaces. Using another runtime (kata, etc) would get closer to the tighter abstraction you mention, but it would truly be a VM, just a small one.

hinkley 6y ago

I feel like the Dockerfile format was very tight and simple to use, and the tools somewhat usable, but over time they keep bolting things onto it by committee. It is better than git, but that's pretty faint praise. When I'm staring at the worst of it (unsticking myself or worse, trying to explain why it's like this to a coworker who is stuck), I keep thinking that there's a standard for making these containers, won't someone get around to rewriting the user-facing bits with the modern requirements designed in from the start? But it's good enough, so we are probably stuck with it until someone comes up with a better idea to base application compartmentalization upon. Like an OS that actually does what I was promised 25 years ago and am still waiting for.

m463 6y ago
I think the idea of a docker file is brilliant. But one pattern I see all the time is: RUN foo && bar && bletch && ... They should have a way of achieving the same thing (just one layer) without multiple commands and added to the same line

AsyncAwait 6y ago

I agree except for compose. I actually like compose and how simple it is, but am curious about container nesting. Could you elaborate? How would dependencies work there for example?

regularfry 6y ago

I was thinking through how you might do this, and my brain dredged up User-Mode Linux. A UML wrapper around those docker containers would behave almost exactly like you describe. You should (if I remember correctly) be able to nest them, too. I'm pretty sure this is doable today. It's a monstrous hack, and I've got no idea what the performance overheads would look like, but as a way of hiding a mess behind a clean facade, I'm not aware of any reason it shouldn't work.

etaioinshrdlu 6y ago
Nowadays, it seems like gVisor is doing something fairly similar to UML but a good deal more lightweight.
- regularfry 6y ago
  Yeah, it looks like this is in the category of "things that could reasonably be done with some elbow-grease" rather than "things that are made impossible by some aspect of the technology".

ForHackernews 6y ago

You might want to check out Flockport: https://thenewstack.io/flockport-time-to-start-all-over-again-and-return-to-lxc-containers/ https://thenewstack.io/flockport-time-to-start-all-over-agai... They're trying to use built-in Linux LXC container features.

nikisweeting 6y ago

I like docker-compose so much that I'd use it even if the underlying containerization technology didn't exist. I think it's one of the things they got most right, and I wish there was a bigger ecosystem around managing compose-based projects.

Rapzid 6y ago
I think it could be better but its features became coupled with swarm. That and the primary maintainer was super hostile to feature requests..

the8472 6y ago

> It should not require root at all (no setuid either). The only thing that really needs setuid are network namespaces to setup the bridges. Userspace workarounds are clunky and slow. If you can do without network isolation then this would be possible. > The containers should be nest-able to arbitrary depth without performance loss (at least to say, hundreds of nestings deep.) Multiple levels of nesting are ppossible if you disable seccomp. I don't know if it scales to hundreds though. Overlayfs has hard limits and btrfs snapeshots don't scale infinitely either. > Also, it wouldn't really be a product at all but just a built-in tool on Linux systems. Well, there's systemd-nspawn and machinectl

_-___________-_ 6y ago
> > Also, it wouldn't really be a product at all but just a built-in tool on Linux systems. > Well, there's systemd-nspawn and machinectl There's also podman, which is a drop-in replacement for docker, and buildah, which does daemonless container builds. I switched to them from Docker recently and will never look back.

thrwaway69 6y ago

Try https://podman.io https://podman.io

musicale 6y ago

Yeah, docker is an anti-pattern in so many ways. Linux is partially to blame as well, since the container/isolation APIs can be hard to use correctly, and many people have latched on to docker as something that sort of works. It also seems to me that the failed security and isolation designs, and painful management and administration designs, of every mainstream operating system have been primary factors in pushing us toward VMs and containers in the first place.

arp242 6y ago
The isolation stuff is a bit arcane and confusing, but also not that hard to figure out or implement as a wrapper. I think the biggest reason people like Docker is that Docker makes it so easy to distribute containers.
neop1x 6y ago
You are probably forgetting about dependency hell, reproducible builds and automated testing. Waaaay tooo often some software won't compile because a bad version of the library is installed in the system or an incompatible compiler is used. Or two perl/python apps require different dependency version. All those problems are quickly solved by Docker. There were no better way to solve it, other than packing and versioning everything like in Chrome build process...