10 ms·

Containers Don't Solve Everything

98 points by kendru 5y ago

kendru 5y ago
Author here. I have been developing Docker applications for years now, and while the experience is better than it used to be, it's still not great. I work for Deref, which is working on developer tooling that is more amenable to modern development workflows. We'd love to hear what pains you have with the current state of development environments.
- ryanmarsh 5y ago
  How do "modern development workflows" differ?
  - kendru 5y ago
    When I started in software development, I was mostly working on monoliths. All dependencies were vendored or were expected to be dynamically linked on the system where they were deployed. Next, I started working with Docker and languages with better package management. Dependencies were fetched in CI and were either statically linked or packaged in a container with the application I was working on. Still, these were mostly monoliths or applications with simple API boundaries and a known set of clients. In the past few years, almost everything I have written has involved cloud services, and when I deploy an application, I do not know what other services may soon depend on the one I just wrote. This type of workflow that depends on runtime systems that I do not own - and where my app may become a runtime dependency of someone else - is what I am referring to as a "modern development workflow".
    - ryanmarsh 5y ago
      Thank you. That’s a worthwhile distinction to make. I often hear the word “modern” bandied about by developers trying to advocate for something novel. If you haven’t already it might be rewarding to fully articulate this in long form, including the consequences and implications. I wish I was a better writer or I’d write this myself.
  - mikepurvis 5y ago
    Not sure about the GP, but for me, it's a pretty big difference jumping from tools like docker to ones like buildah, microk8s— I feel like k8s really makes good on many of the original promises of the "container revolution" in terms of automatic provisioning/scaling/image lifecycle management, declaratively defining relationships between containers, adding a much needed upper layer of abstraction in terms of the pod, etc. I know docker has made it part of the way there over the years with Compose and so on, but it's all felt pretty ad-hoc, whereas k8s feels like a cohesive system designed against a clear vision (which makes sense, since it was designed as borg 2.0)— no one else working in this space had the benefit of having already built a giant system for it and used it at scale for years beforehand.
- johnchristopher 5y ago
  I am not a 10x engineer or a linux wizard. I wish someone would rewrite docker-compose in a single go or rust binary so that I don't have to deal with the python3 crypto package being out of date or something when simply configuring docker/docker-compose for another user (usually me on a different machine or new account).
  - raesene9 5y ago
    https://docs.docker.com/compose/cli-command/ https://docs.docker.com/compose/cli-command/ ^ There's an rc of a compose command built into the standard docker CLI.
    - johnchristopher 5y ago
      Haha, nice :D. Thanks. Now I wish the company I work for would drop their plan to bring me back in office next week and just settle instead for a day or two of mandatory presence in the office per month (crossing fingers while you do your magic).
superkuh 5y ago
It depends on the context, I don't know about corporate persons with profit incentives but if we're talking human persons then containers don't solve anything. They're just the symptom of the disease that is future shock. The underlying libraries we depend on just change too fast now and no devs care about forwards compatibility so we end up with all OS/Distros having libs that stop working in about a year (or more like 3 months with Rust/JS/etc). The solution has to either come in the form of static compilation, or, even less feasible, getting devs to actually care if their software runs on platforms more than a year old. Containers just make everything worse in all cases beyond the contrived "it just worked and I never need to change anything".
- encryptluks2 5y ago
  I don't think dependencies is the only benefit of containers. I personally like the isolation they provide and generally prefer running services in containers, even if they are using the same dependencies as my OS. I run Linux too, so I don't have to worry about any virtualization framework overhead.
  - zozbot234 5y ago
    Namespacing and isolation also unlocks additional features, such as VM-style checkpointing and migration (via the CRIU featureset, which AIUI is now part of the mainline kernel). Moreover, the 'container' workflow provides a common interface that the various sorts of orchestration/deployment/management platforms can then rely on.
  - rualca 5y ago
    > I personally like the isolation they provide and generally prefer running services in containers, even if they are using the same dependencies as my OS. I would also not downplay the importance of Docker's support for software-defined networks and it's ability to arbitrarily configure networking at the container level. I firmly believe that networking doesn't pop up so often while discussing Docker because Docker solves that problem so fantastically well that a complex problem simply ceases to exist and completely abandons everyone's mental model.
vahid4m 5y ago
What solves everything?
- ISV_Damocles 5y ago
  The heat death of the universe?
  - klysm 5y ago
    Possibly the only maybe valid answer.
  - 1_player 5y ago
    No, that'd be like closing all outstanding bug reports with a WONTFIX.
    - deleted 5y ago
      [deleted]
    - yencabulator 5y ago
      Heat death of the universe is just nature's stalebot.
- the-dude 5y ago
  Silver bullets
  - ozim 5y ago
    Well you are onto something. Even if it is a joke, people want to have silver bullets. Those are killing the hairy problems which can be named werewolves. Downside is hairy problems just like werewolves come from people. So it in the end it is people problems not some container tech or other stack problem. There are no werewolves without people :)
  - jtolmar 5y ago
    Only if you have physical access to the server.
encryptluks2 5y ago
This looks more like an advertisement than a useful blog post. Also: > Consider also that Docker relies on Linux kernel-specific features to implement containers, so users of macOS, Windows, FreeBSD, and other operating systems still need a virtualization layer. First, FreeBSD has its own native form of containers and Windows has its own native implementation. Docker != containers. I really don't see how Docker (or containers as we mostly know them) relying on kernel-features from an open source operating system in order to run Linux OS images as something to even complain about, and there is nothing preventing Mac from implementing their own form of containers.
- kendru 5y ago
  I am familiar with FreeBSD jails (and IMO, they are actually superior to Linux containers in most respects). My point is not so much that other systems don't have the tech to make containers work - or that OS vendors are not capable of adding containers to their kernels - but that having container technology is not the same as having a smooth devex for containerized applications.
  - encryptluks2 5y ago
    The fact is Linux containers are probably hotter than anything else. Almost every enterprise are using them to some larger extent, and Kubernetes has become the platform of choice. Is vanilla Kubernetes easy for new developers? No, but there is an entire ecosystem offering tools and platforms to make development using containers a seamless as possible. Microsoft saw this, so they really had no choice but to adopt the container terminology and partner with Docker to try to stay relevant. My guess is without containers, Microsoft would have never even built WSL. If you want smooth developer experience with containers then that is what solutions like GitLab offer. Even Microsoft's GitLab is essentially built around running various actions inside containers. I personally welcome the change. I can spin up a local Kubernetes cluster and test an entire cluster of applications locally if I want, or integrate it into Skaffold or whatever else and test live in the cloud. It really is a lot better than what we had before. I think the solutions though really come down to documentation and resources to help train new employees and acclimate them.
jmartens 5y ago
Follow up article: Containers don't solve anything.
- 1_player 5y ago
  6 months later, the containerless movement takes root. 2 years later, the seminal article: Perhaps containers were not such a bad idea.
  - hinkley 5y ago
    And a fringe group pushing for thin VMs.
cfors 5y ago
Yes containers don't solve for dealing with the mess of third party saas that every company is built around. But that's why anytime you integrate with one of these tools you should be aware that there is a cost for maintaining that integration.
nimbius 5y ago
the one thing containers addressed was their use as a countermeasure to rising costs from greedy VPS providers, and as an agile framework to quickly evacuate from a toxic provider (cost, politics, performance, etc...) providers in turn responded by shilling their 'in house' containerization products and things like Lambda for lock-in.
asim 5y ago
I spent 6+ years fighting this exact battle. It's hard. It's resource intensive. And timing is everything. It requires either one company to front all the development cost and bring it to the world after validating it or it needs an ecosystem to emerge through a shared pain and understanding. We're not there yet. My efforts => https://micro.mu https://micro.mu Oh and prior efforts https://github.com/asim/go-micro https://github.com/asim/go-micro
Zababa 5y ago
Since the author mentionned it, is the 12 factor app still a best practice? Was it a best practice? I saw the website a few times and all of it makes sense for me, but I haven't seen much discussion about it.
dekhn 5y ago
the one problem containers solved for me better than anything I ever used in previous UNIX/LINUX is heirarchical resource tracking. I work with many codes that fork from their main binary and do their work in subprocesses. If your resource manager isn't scraping /proc to invert the process tree, it needs a way to assign resources to process trees such that the entire tree sum cannot exceed the resource limitation.
forgotmypw17 5y ago
My container is POSIX :)
- ebiester 5y ago
  What is your strategy that works for every packaging system and every version of every library you depend on? Nothing says love like realizing that you are segfaulting due to a library version you didn't test against subtly changing its behavior.
  - forgotmypw17 5y ago
    I use almost no libraries and stick to languages which have prioritized stability for 20+ years. This amounts to using Perl, bash, and POSIX. On the client side, of course, it is HTML and JS, which I use a very limited subset of to improve compatibility.
tracker1 5y ago
I think the next step(s) will be something closer to what the combination of Cloudflare Workers + KV + Durable Objects gives you... I think there also needs to be some implementation of PubSub added to the mix as well as a more robust database store. Fastly has similar growing options, and there are more being advanced/developed. In the end, there's only a few missing pieces to offer a more robust solution. I do think that making it all webassembly will be the way to go, assuming the WASI model(s) get more flushed out (Sockets, Fetch, etc). The Multi-user web doom on cloudflare[1] is absolutely impressive to say the least. I kind of wonder if Cloudflare could take what FaunaDB, CockroachDB or similar offers and push this more broadly... At least a step beyond k/v which could be database queries/indexes against multiple fields. Been thinking on how I could use the existing Cloudflare system for something like a forum or for live chat targeting/queries... I think that the Durable Objects might be able to handle this, but could get very ugly. 1. https://blog.cloudflare.com/doom-multiplayer-workers/ https://blog.cloudflare.com/doom-multiplayer-workers/
KingMachiavelli 5y ago
Containers don't solve anything more than virtual machines. Containers are 'better' than virtual machines because they have less overhead and are 100% open source. Containers and VMs let you divide and solve problems in isolation in a convenient manner. You still have the same problems inside each container. Firstly, Docker & k8s made using containers easy. Minimal distros like alpine simplify containers to a set of one or more executable. You could implement the same thing with a system of systemd services & namespaces. But now that everything was a container, you need a way to manage what & where containers are running and how they communicate with each other. It looks like 90% of the stuff different container tools and gadgets try to solve is the issues they created. You can no longer install a LAMP stack via 'apt install mysql apache php7.4' so instead you need a tool that sets up 3 containers with the necessary network & filesystem connections. It certainly better because it is all decoratively defined but it is still the same problem. This is why I mostly stayed out of containers until recently. The complexity of containers really only helps if you need to replicate certain server/application. You will still need to template all of your configuration files even if you use Docker, etc. What is changing everything IMO is NixOS because it solves the same issues without jumping all the way to Docker or k8s. Dependencies are isolated like containers but the system itself whether it is a host/standalone or a container can be defined in the same manner. This means that going from n=1 to n>1 is super easy and migrating from a multi-application server (i.e a pet server) to a containerized environment (i.e to a 'cattle' server/container) is straightforward. It's still more complex and a bit rough compared to Docker & k8s but using the same configuration system everywhere makes it worthwhile.
mikewarot 5y ago
Virtual Machines gained popularity as are kludge to get around the remarkably horrible state of operating systems. The inability to reliably save and restore the state of a computer grew to be so costly that it became worthwhile to pay the performance penalty of a layer of emulation/virtualization to route around it. Containers were the next logical step, as each virtual machine vendor tried to lock in their users. Containers allowed routing around it. Both of these steps could be eliminated if a well behaved operating system similar to those in mainframes could be deployed, so that each application sat in its own runtime, had its own resources, and no other default access. There's a market opportunity here, it just needs to be found.

98 points by kendru 5y ago

Author here. I have been developing Docker applications for years now, and while the experience is better than it used to be, it's still not great. I work for Deref, which is working on developer tooling that is more amenable to modern development workflows. We'd love to hear what pains you have with the current state of development environments.

ryanmarsh 5y ago
How do "modern development workflows" differ?
- kendru 5y ago
  When I started in software development, I was mostly working on monoliths. All dependencies were vendored or were expected to be dynamically linked on the system where they were deployed. Next, I started working with Docker and languages with better package management. Dependencies were fetched in CI and were either statically linked or packaged in a container with the application I was working on. Still, these were mostly monoliths or applications with simple API boundaries and a known set of clients. In the past few years, almost everything I have written has involved cloud services, and when I deploy an application, I do not know what other services may soon depend on the one I just wrote. This type of workflow that depends on runtime systems that I do not own - and where my app may become a runtime dependency of someone else - is what I am referring to as a "modern development workflow".
  - ryanmarsh 5y ago
    Thank you. That’s a worthwhile distinction to make. I often hear the word “modern” bandied about by developers trying to advocate for something novel. If you haven’t already it might be rewarding to fully articulate this in long form, including the consequences and implications. I wish I was a better writer or I’d write this myself.
- mikepurvis 5y ago
  Not sure about the GP, but for me, it's a pretty big difference jumping from tools like docker to ones like buildah, microk8s— I feel like k8s really makes good on many of the original promises of the "container revolution" in terms of automatic provisioning/scaling/image lifecycle management, declaratively defining relationships between containers, adding a much needed upper layer of abstraction in terms of the pod, etc. I know docker has made it part of the way there over the years with Compose and so on, but it's all felt pretty ad-hoc, whereas k8s feels like a cohesive system designed against a clear vision (which makes sense, since it was designed as borg 2.0)— no one else working in this space had the benefit of having already built a giant system for it and used it at scale for years beforehand.
johnchristopher 5y ago
I am not a 10x engineer or a linux wizard. I wish someone would rewrite docker-compose in a single go or rust binary so that I don't have to deal with the python3 crypto package being out of date or something when simply configuring docker/docker-compose for another user (usually me on a different machine or new account).
- raesene9 5y ago
  https://docs.docker.com/compose/cli-command/ https://docs.docker.com/compose/cli-command/ ^ There's an rc of a compose command built into the standard docker CLI.
  - johnchristopher 5y ago
    Haha, nice :D. Thanks. Now I wish the company I work for would drop their plan to bring me back in office next week and just settle instead for a day or two of mandatory presence in the office per month (crossing fingers while you do your magic).

superkuh 5y ago

It depends on the context, I don't know about corporate persons with profit incentives but if we're talking human persons then containers don't solve anything. They're just the symptom of the disease that is future shock. The underlying libraries we depend on just change too fast now and no devs care about forwards compatibility so we end up with all OS/Distros having libs that stop working in about a year (or more like 3 months with Rust/JS/etc). The solution has to either come in the form of static compilation, or, even less feasible, getting devs to actually care if their software runs on platforms more than a year old. Containers just make everything worse in all cases beyond the contrived "it just worked and I never need to change anything".

encryptluks2 5y ago
I don't think dependencies is the only benefit of containers. I personally like the isolation they provide and generally prefer running services in containers, even if they are using the same dependencies as my OS. I run Linux too, so I don't have to worry about any virtualization framework overhead.
- zozbot234 5y ago
  Namespacing and isolation also unlocks additional features, such as VM-style checkpointing and migration (via the CRIU featureset, which AIUI is now part of the mainline kernel). Moreover, the 'container' workflow provides a common interface that the various sorts of orchestration/deployment/management platforms can then rely on.
- rualca 5y ago
  > I personally like the isolation they provide and generally prefer running services in containers, even if they are using the same dependencies as my OS. I would also not downplay the importance of Docker's support for software-defined networks and it's ability to arbitrarily configure networking at the container level. I firmly believe that networking doesn't pop up so often while discussing Docker because Docker solves that problem so fantastically well that a complex problem simply ceases to exist and completely abandons everyone's mental model.

vahid4m 5y ago

What solves everything?

ISV_Damocles 5y ago
The heat death of the universe?
- klysm 5y ago
  Possibly the only maybe valid answer.
- 1_player 5y ago
  No, that'd be like closing all outstanding bug reports with a WONTFIX.
  - deleted 5y ago
    [deleted]
  - yencabulator 5y ago
    Heat death of the universe is just nature's stalebot.
the-dude 5y ago
Silver bullets
- ozim 5y ago
  Well you are onto something. Even if it is a joke, people want to have silver bullets. Those are killing the hairy problems which can be named werewolves. Downside is hairy problems just like werewolves come from people. So it in the end it is people problems not some container tech or other stack problem. There are no werewolves without people :)
- jtolmar 5y ago
  Only if you have physical access to the server.

encryptluks2 5y ago

This looks more like an advertisement than a useful blog post. Also: > Consider also that Docker relies on Linux kernel-specific features to implement containers, so users of macOS, Windows, FreeBSD, and other operating systems still need a virtualization layer. First, FreeBSD has its own native form of containers and Windows has its own native implementation. Docker != containers. I really don't see how Docker (or containers as we mostly know them) relying on kernel-features from an open source operating system in order to run Linux OS images as something to even complain about, and there is nothing preventing Mac from implementing their own form of containers.

kendru 5y ago
I am familiar with FreeBSD jails (and IMO, they are actually superior to Linux containers in most respects). My point is not so much that other systems don't have the tech to make containers work - or that OS vendors are not capable of adding containers to their kernels - but that having container technology is not the same as having a smooth devex for containerized applications.
- encryptluks2 5y ago
  The fact is Linux containers are probably hotter than anything else. Almost every enterprise are using them to some larger extent, and Kubernetes has become the platform of choice. Is vanilla Kubernetes easy for new developers? No, but there is an entire ecosystem offering tools and platforms to make development using containers a seamless as possible. Microsoft saw this, so they really had no choice but to adopt the container terminology and partner with Docker to try to stay relevant. My guess is without containers, Microsoft would have never even built WSL. If you want smooth developer experience with containers then that is what solutions like GitLab offer. Even Microsoft's GitLab is essentially built around running various actions inside containers. I personally welcome the change. I can spin up a local Kubernetes cluster and test an entire cluster of applications locally if I want, or integrate it into Skaffold or whatever else and test live in the cloud. It really is a lot better than what we had before. I think the solutions though really come down to documentation and resources to help train new employees and acclimate them.

jmartens 5y ago

Follow up article: Containers don't solve anything.

1_player 5y ago
6 months later, the containerless movement takes root. 2 years later, the seminal article: Perhaps containers were not such a bad idea.
- hinkley 5y ago
  And a fringe group pushing for thin VMs.

cfors 5y ago

Yes containers don't solve for dealing with the mess of third party saas that every company is built around. But that's why anytime you integrate with one of these tools you should be aware that there is a cost for maintaining that integration.

nimbius 5y ago

the one thing containers addressed was their use as a countermeasure to rising costs from greedy VPS providers, and as an agile framework to quickly evacuate from a toxic provider (cost, politics, performance, etc...) providers in turn responded by shilling their 'in house' containerization products and things like Lambda for lock-in.

asim 5y ago

I spent 6+ years fighting this exact battle. It's hard. It's resource intensive. And timing is everything. It requires either one company to front all the development cost and bring it to the world after validating it or it needs an ecosystem to emerge through a shared pain and understanding. We're not there yet. My efforts => https://micro.mu https://micro.mu Oh and prior efforts https://github.com/asim/go-micro https://github.com/asim/go-micro

Zababa 5y ago

Since the author mentionned it, is the 12 factor app still a best practice? Was it a best practice? I saw the website a few times and all of it makes sense for me, but I haven't seen much discussion about it.

dekhn 5y ago

the one problem containers solved for me better than anything I ever used in previous UNIX/LINUX is heirarchical resource tracking. I work with many codes that fork from their main binary and do their work in subprocesses. If your resource manager isn't scraping /proc to invert the process tree, it needs a way to assign resources to process trees such that the entire tree sum cannot exceed the resource limitation.

forgotmypw17 5y ago

My container is POSIX :)

ebiester 5y ago
What is your strategy that works for every packaging system and every version of every library you depend on? Nothing says love like realizing that you are segfaulting due to a library version you didn't test against subtly changing its behavior.
- forgotmypw17 5y ago
  I use almost no libraries and stick to languages which have prioritized stability for 20+ years. This amounts to using Perl, bash, and POSIX. On the client side, of course, it is HTML and JS, which I use a very limited subset of to improve compatibility.

tracker1 5y ago

I think the next step(s) will be something closer to what the combination of Cloudflare Workers + KV + Durable Objects gives you... I think there also needs to be some implementation of PubSub added to the mix as well as a more robust database store. Fastly has similar growing options, and there are more being advanced/developed. In the end, there's only a few missing pieces to offer a more robust solution. I do think that making it all webassembly will be the way to go, assuming the WASI model(s) get more flushed out (Sockets, Fetch, etc). The Multi-user web doom on cloudflare[1] is absolutely impressive to say the least. I kind of wonder if Cloudflare could take what FaunaDB, CockroachDB or similar offers and push this more broadly... At least a step beyond k/v which could be database queries/indexes against multiple fields. Been thinking on how I could use the existing Cloudflare system for something like a forum or for live chat targeting/queries... I think that the Durable Objects might be able to handle this, but could get very ugly. 1. https://blog.cloudflare.com/doom-multiplayer-workers/ https://blog.cloudflare.com/doom-multiplayer-workers/

KingMachiavelli 5y ago

Containers don't solve anything more than virtual machines. Containers are 'better' than virtual machines because they have less overhead and are 100% open source. Containers and VMs let you divide and solve problems in isolation in a convenient manner. You still have the same problems inside each container. Firstly, Docker & k8s made using containers easy. Minimal distros like alpine simplify containers to a set of one or more executable. You could implement the same thing with a system of systemd services & namespaces. But now that everything was a container, you need a way to manage what & where containers are running and how they communicate with each other. It looks like 90% of the stuff different container tools and gadgets try to solve is the issues they created. You can no longer install a LAMP stack via 'apt install mysql apache php7.4' so instead you need a tool that sets up 3 containers with the necessary network & filesystem connections. It certainly better because it is all decoratively defined but it is still the same problem. This is why I mostly stayed out of containers until recently. The complexity of containers really only helps if you need to replicate certain server/application. You will still need to template all of your configuration files even if you use Docker, etc. What is changing everything IMO is NixOS because it solves the same issues without jumping all the way to Docker or k8s. Dependencies are isolated like containers but the system itself whether it is a host/standalone or a container can be defined in the same manner. This means that going from n=1 to n>1 is super easy and migrating from a multi-application server (i.e a pet server) to a containerized environment (i.e to a 'cattle' server/container) is straightforward. It's still more complex and a bit rough compared to Docker & k8s but using the same configuration system everywhere makes it worthwhile.

mikewarot 5y ago

Virtual Machines gained popularity as are kludge to get around the remarkably horrible state of operating systems. The inability to reliably save and restore the state of a computer grew to be so costly that it became worthwhile to pay the performance penalty of a layer of emulation/virtualization to route around it. Containers were the next logical step, as each virtual machine vendor tried to lock in their users. Containers allowed routing around it. Both of these steps could be eliminated if a well behaved operating system similar to those in mainframes could be deployed, so that each application sat in its own runtime, had its own resources, and no other default access. There's a market opportunity here, it just needs to be found.