12 ms·

Docker in Production: A History of Failure (2016)

112 points by flanfly 5y ago

debarshri 5y ago
It was year 2016, kubernetes did not have jobs, cronjobs, statefulsets. Pods would get stuck in terminating state or container creating state. Networking in kubernetes was wonky. AWS did not have support for EKS. It used be painful. It is year 2021, 1000s of new startups around kubernetes, more features, more resource types. Pods would still get stuck in terminating state or container creating state. It still pretty painful.
- HelloNurse 5y ago
  > Networking in kubernetes was wonky. Can you elaborate? Does Kubernetes add some thrills to the relatively simple Docker network configuration?
  - krab 5y ago
    Yes, Kubernetes (actually its add-ons) provide a virtual network that unifies communication within the cluster so you don't need to care on which computer your service runs.
  - jrockway 5y ago
    You can make it as complicated as you want it to be; part of setting up a cluster is picking the networking system ("CNI"). Cloud providers often have their own IPAM (i.e. on Amazon, you get this: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html; https://docs.aws.amazon.com/eks/latest/userguide/pod-network... each Pod gets an IP from your VPC, resulting in weird limits like 17 pods per instance because that's how many IP addresses you can have for that particular instance type throughout EC2).
  - kaidon 5y ago
    Please take a seat and let the joys of k8s networking overwhelm your senses: https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n... And yes... Kubernetes network configuration is on a whole different level from docker networking.
    - theptip 5y ago
      To be fair, multi-node networking of any sort is on a different level than single-host docker networking. If you ever tried to use Docker Swarm to network multiple nodes, god help you. Also worth noting that almost all users of K8s don't actually need to operate a cluster, the hosted offerings handle all of that for you. You just need to understand the Service object, and maybe Ingress if you're trying to do some more advanced cert management or API gateway stuff. It's a common meme around here to point in horror to the complexity that is abstracted away under the K8s cluster API, and claim that k8s is really hard to use. I think that's mostly misguided, the hosted offerings like GKE really do a good job of hiding away all that complexity from you. Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.
      - kazen44 5y ago
        > Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes. How is routing from the internet to multiple servers a problem? usually, you have either one of these setups: - you run a loadbalancer that distributes traffic across your nodes. (This loadbalancer could even be distributed thanks to BGP). - you either run your own firewall or have a managed one, in which you either announce your IP prefix yourself, or they are announced for you by your uplink provider. - you run an anycast setup (for, for example, globally distributed DNS). and announce multiples of the same prefix across the globe. Routing in the DFZ does the rest for you. Streched L2 across the globe/internet is also possible (although not very performant) either by doing IPsec tunneling, or by buying/setting up L2VPN services. (either MPLS or VXLAN based).
        theptip 5y ago
        I didn't say it was a problem. My claim was just that it's easier in GKE than in GCE/EC2. I only mentioned multi-node because exposing a single VM to the internet is trivial -- just give it a public IP -- and thus is not an apples-to-apples comparison with the multi-node load balancing that you get from the entry-level k8s configuration of Service > Pod < Deployment.
  - kazen44 5y ago
    the largest issue with kubernetes networking seems to be the lack of integration with modern datacenter networking technology. Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance). Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).
- jrockway 5y ago
  I haven't seen these failure modes in 2021. We do managed clusters at work and have created around 100,000 of them, and basically all the pods we intend to start start, and all the pods we intend to kill die -- even with autoscaling that provisions new instance types. Our biggest failure mode is TLS certificates failing to provision through Let's Encrypt, but that has nothing to do with Kubernetes (that is a layer above; what we run in Kubernetes). EKS continues to be painful. It has gotten better over the years, but it is a chore compared to GKE. I like to imagine that Jeff Bezos walked into someone's office, said "you're all fired if I don't have Kubersomethings in two weeks", and that's what they launched.
  - shaicoleman 5y ago
    What are the EKS pain points?
    - jrockway 5y ago
      The biggest pain point is having to manually use cloudformation to create node pools. This is especially irritating when you just need to roll the Linux version on nodes -- takes half a day to do right. In GKE, it's just a button in the UI (or better, an easy-to-use API), and you can schedule maintenance windows for security updates (which are typically zero downtime anyway, assuming you have the right PodDisruptionBudgets). I think AWS fixed that. I remember when I used it, they said they had some new tool that would handle that, but you had to re-create the cluster from scratch. This was a couple years ago, and is probably decent nowadays. There are other warts, like certain storage classes being unavailable by default (gp3), the whole ENI thing for Pod IPs, the supported version being way out of date, etc. EKS has always felt like "minimum viable product" to me -- they really want you to use their proprietary stuff like ECS/Fargate, CloudFormation, etc. If you're already on AWS and want Kubernetes, it's just what you need. If you could pick any cloud provider for mainly Kubernetes, it wouldn't be my first choice. Having used EKS, GKE, and DOKS, I definitely prefer GKE. GKE is very feature-rich, and the API for managing clusters works well. The nodes are also cheaper than AWS. (I use DOKS for my personal stuff and I haven't had any problems, and it is free, but it's missing features like regional clusters that you probably want for things you make money off of.)
- dehrmann 5y ago
  Around 2015, I was at Spotify, and we were using a container orchestrator build in-house named Helios. They didn't build it because Kubernetes wasn't invented there; they built it because Kubernetes didn't exist, yet.
  - lacion 5y ago
    kubernetes was released in 2014, in late 2015 I already had a cluster marked as released candidate that was put into production early 2016. so im guessing here that spotify wrote their own becouse they had a spesific requirement? nomad was also early days in 2015
    - dehrmann 5y ago
      I'm pretty sure the work started before Kubernetes was released, and even then, it wasn't clear that was going to be the de factor orchestrator.
- halfmatthalfcat 5y ago
  I have used Kubernetes extensively over the past couple years and have never seen pods stuck in terminating or creating state that didn't have to do with errors in container creation (your Dockerfile/bootstrapping is messed up) or issues with healthchecks.
- CSDude 5y ago
  We spawn thousands of pods per day for jobs and never get those stuck and it was not the case in 2018 either. Not sure what is it you are doing causes this.
clipradiowallet 5y ago
I know this article is from 2016...but my feelings about it(the article) are unchanged. Some people do not like new things, and they will blog about it in some form or fashion. Maybe their reasoning is valid, maybe it's not - it doesn't matter. Meanwhile...businesses have, and continue, to pay top $$$ for people that will help them do these things. If you want to collect this $$$, get on board. In a few years, the things businesses want to pay $$$ will change. New blog articles about "this new stuff is bad!" will appear, and new job postings paying above-market $$$ will appear also. You can either rail on about the bad(or good) changes, and how it's just everything-old-is-new-again....or you can get with the program, and get paid. In another few years, rinse and repeat.
- deleted 5y ago
  [deleted]
- jjnoakes 5y ago
  It's all anecdotal. For example I know many folks who make $$$ doing the boring old thing because it is reliable, it gets results quickly with low risk, the engineers know the tech inside and out, and not many other folks want to work with "boring tech".
  - cpach 5y ago
    Just out of curiosity, what are some examples of boring tech in this case?
    - isoskeles 5y ago
      Makes me think of stuff like managing WordPress although I'm not sure if that's an example they had in mind.
    - aprdm 5y ago
      Django, Rails, Postgres...
      - 5y ago
manishsharan 5y ago
This is a blogpost from 2016 . However if we switch to more recent times, my experience with AWS ECS and Fargate has been fairly boring. There was a learning curve to get it to work with cloudformation, vpcs, iam and load balancer .
- esotericimpl 5y ago
  Agreed, dont see why ECS and fargate isnt used everywhere. It takes a bit of a learning curve to understand how tasks, task definitions ,clusters, services all fir together but once you do, it's pretty straight forward. I've ran ECS in production for over 5 years, can count on one hand where we had any issue related to docker or availability, all were based on code updates we didn't test properly.
_joel 5y ago
In 2016 I started at a company that had no build procedures and deployed to a variety of linux versions, developed on windows. It was a nightmare for administration, no automation, no monitoring. I implemented containers and most of the process was getting the developers on board. Having technical sessions with them to understand what they needed and ease them into the plan so they felt enfranchished. Doing this vastly increased productivity, devs could take off the shelf compose files that were written for common projects (it was a GIS shop) and meant they could concentrate on delivering code. It helped no end. Sure there's issues (albeit a lot fewer as time progressed) with docker but for what it gained in productivity and developer's sanity, it was very welcome.
- lmm 5y ago
  > In 2016 I started at a company that had no build procedures and deployed to a variety of linux versions, developed on windows. It was a nightmare for administration, no automation, no monitoring. It's no surprise that implementing pretty much any process would be a massive improvement from that. The more interesting question is whether containers offer any advantages for people who already have a (probably significantly more lightweight) build and deployment process.
theamk 5y ago
Back in 2016 during the original discussion of this article, amount said it very well in [0]: "If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area." - Yes, "clean old images" was not implemented back then. His hack is not that bad, and one can filter out in-use images if they want to pretty easily. Anyway, docker does have "docker image prune" now. - Storage driver history discussion is entirely incorrect. No, docker did not invent overlayfs nor overlayfs2. There was a whole big drama of aufs not mainlining, but it was mostly in context of live cd's, not docker. But the big missing thing is: you should not store important data in docker images, Docker is designed to work with transient container. If you have a database, or a high-performance data store, you use volumes, and those _bypass_ docker storage drivers completely. - The database story is completely crazy... judging by their comments, they decided to store the database data in the docker container for some reason and got all the expected problems (unable to recover, hard to migrate, etc....). It is not clear why they didn't put database data on the volume, there is a 2016 StackOverflow question discussing it [0]. Also, "Docker is locking away [...] files through its abstraction [...] It prevents from doing any sort of recovery if something goes wrong." Really? I did recovery with docker, the files are under /var/lib/docker in the directory named with guid, a simple "find" command can locate them. - By default, Docker uses Linux networking and yes, the configuration is complex so it adds overhead. That's why there is --net=host option (which was there for a long time) which just bypasses that all. [0] https://news.ycombinator.com/item?id=12872636 https://news.ycombinator.com/item?id=12872636 [1] https://stackoverflow.com/questions/40167245/how-to-persist-data-using-postgres-docker-image https://stackoverflow.com/questions/40167245/how-to-persist-...
pbecotte 5y ago
Even in 2016, I had been running production services in Docker successfully. Its interesting to me that they see the problem "Docker isn't designed to store data" without also seeing the solution "the docker copy-on-write filesystem isn't designed to be written to production- but volume mounts are". I hadn't seen docker crashing hosts (still haven't) - but I'm guessing that was caused by using the storage drivers. The complaints about their development practices are valid (and haven't really improved), but even then the technology worked well so long as you understood its limitations.
mberning 5y ago
Docker is the chaos monkey incarnate.
crummybowley 5y ago
The issue is not docker, the issue is you treat your servers like pets. Folks need to start building systems that destroy all and re-image fresh. Any other way you are just setting your self up for failure.
- esotericimpl 5y ago
  except the Database, don't re-image the database.
- abraxas 5y ago
  And at the heart of this there will be a database or some other data store that will be very much a pet. A very precious, important and fickle pet. Even if it's "serverless" it will have its own needs and wants. This whole "cattle not pets" charade gets on my nerves. Yeah, it's easy to "scale" that which is stateless. Not so fast with stateful. I don't care that I can spin up gazillions of web server instances. My data store is still one and very much stateful.
  - sureglymop 5y ago
    I think stateless is good but don't forget that the state is just in another place, another stateful "gear" in your machine. And that, as you said, has to be treated like the pet it is.
  - crummybowley 5y ago
    Even so, the topic is docker. Your storage is else where in docker -- not in docker. If it is in docker, then you are already in trouble. Simple solution is storage is a secondary drive/partition -- still automate OS install.
  - tene 5y ago
    Persistent identity is fine, it just doesn't belong on individual servers. If the data matters, then it's better for it to be reproducibly managed, durably persisted, etc. For many database needs, there are great options that natively support a highly available cluster. For those needs that don't have a good clustering option, there are great network storage systems that are easy to deploy and use. You don't need to treat hardware as a pet to have a database with persistent identity.
stevebmark 5y ago
> Docker is meant to be stateless. Containers have no permanent disk storage, whatever happens is ephemeral and is gone when the container stops. It's interesting that this misconception made it into a clearly knowledgeable article. Containers have state on the writeable layer that is persisted between container stops and starts.
- plainnoodles 5y ago
  But I think most people and tools consider "containers" to be volatile storage, like RAM. Non-volatile storage would be volumes. Honestly I think there is a lot to be said for making the writable layer of a container read-only. It makes sure that things like logging, if you care about them, go somewhere safe, or if you don't, get turned off explicitly. And also prevents gotchas like "oops wrote important data to /var/lib/notavolume when I meant to write to /var/lib/therightvolume" that show up at the worst times.
- beermonster 5y ago
  I’m not sure it’s a misconception. That’s how they’re intended to be used. Cattle not pets. If you don’t get used to not treating them as throw away you can end up accidentally relying on some state. As you say, the top layer is read/write but that doesn’t mean you should be relying on what you write there. Quite the opposite - that state should be somewhere else unless you can afford to lose it. I usually start mine with —-rm so they’re removed on shutdown. I’ve seen people apply security updates via ‘apt update; apt upgrade’ within a running container. Guess what happens when that container is eventually destroyed?
bfrog 5y ago
Heh I just ran into an issue the other day with a coworker where Ubuntu had a patched kernel auto update and break everything. Yep, it’s a sand castle
Theodores 5y ago
Symfony console broke Magento 2 today. Same story.
belter 5y ago
Posted many times before but this is the only one with comments: https://news.ycombinator.com/item?id=12872304 https://news.ycombinator.com/item?id=12872304
zz865 5y ago
Our big project has moved from physical servers to Openshift. Its taken a lot of work, much more than expected. The best thing is that developers like it on their resume, which is a bigger benefit than you'd think as we've kept some good people on the team. For users I see zero benefit. CI pipeline is just more complicated and probably slower. Cost wise it was cheaper for a while but now RedHat are bumping up licensing costs so now I think is about the same costs. Overall it seems like a waste of time, but has been interesting.
- geerlingguy 5y ago
  OpenShift is about 10x more complex than basic Docker / containers, and probably 2-4x more complex than plain old Kubernetes. I've seen more success from organizations running smaller K3s or K8s clusters (if they need the orchestration) or just running small apps via Docker/Docker Compose separately, using a CI system (even as simple as GitHub Actions) to manage deployments.
  - zz865 5y ago
    Yeah its also a problem that our org has infrastructure teams that manage the openshift clusters and they are under resourced so dont help or often can't figure out how to fix problems. Linux sysadmins know what they're doing as the core infrastructure has been mostly the same for the last few decades.
- mixermachine 5y ago
  Moving from classic servers to containers you get: - Builds with fixed dependencies that never change. Rollback is easy - Easy deployment of a prod environment on a local machine - Fast deployment - Easy automation (use version X with config Y) With Kubernetes (or other derivates like Openshift) you get: - Auto scaling - Fail over - Better resource usage if multiple environments are executed - Abstraction of infrastructure - Zero downtime deployment (biggest point for my company, we deploy >3 times per week) There are applications that do not need Kubernetes or even containers, but is this list really nothing oO? I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things.
rubyist5eva 5y ago
Podman and Kubernetes are like a match made in heaven. Docker was a good first try for most people, but there is so much better technology that exists now.
ChrisArchitect 5y ago
Anything new since this? A history of re-submitted, previously discussed posts: https://news.ycombinator.com/item?id=12872304 https://news.ycombinator.com/item?id=12872304
KronisLV 5y ago
The article seems to mention problems with AUFS, overlay and possibly overlay2 as well. However one of the things that i haven't quite understood, is why people use Docker volumes that much in the first place, or even think that they need to use additional volume plugins in most deployments? If it's a relatively simple deployment, that has some persistent data and it's clear on which nodes the containers could be scheduled (either by label or by hostname), what would prevent someone from just using bind mounts ( https://docs.docker.com/storage/bind-mounts/ https://docs.docker.com/storage/bind-mounts/ )? And if you need to store it on a separate machine, why not just use NFS on the host OS to mount the directory which you will bind mount? Or, alternatively, why not just use GlusterFS or Ceph for that sort of stuff, instead of making Docker attempt to manage it? For example, Docker Swarm fails to launch containers if the bind mount path doesn't exist, but that bit can also be addressed by creating the necessary directory structure with something like Ansible - and then you're not only able to not worry about volumes and the risk of them ever becoming corrupt, but you also have the ability to inspect the contents of the container storage on the actual host. Say, if there are some configuration files that need altering (seeing as not all of the containerized software out there follows 12 Factor principles with environment configuration either), or you just want to do some backups for the data that you've stored in a granular fashion.
jwildeboer 5y ago
Guy who claims to run systems in the hft space, responsible for millions of trades with high values, can’t be bothered to actually pay for support, relies on community and blames everyone but himself for being left alone with his mess. Not sorry.
user5994461 5y ago
Original author from 5 years ago. Surprised to see this here 5 years later. Docker really used to crash a lot back in the days, mostly due to buggy storage drivers. If you were on Debian or CentOS it's very likely that you experienced crashes (though a lot of developers didn't care or didn't understand the reasons the system went unresponsive). There was notably a new version of Debian (with a newer kernel) published the year after my experience. It's a lot more stable now. My experience is that by 2018-2019, Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience. edit: at that time Docker didn't have a way to clear images/containers, it was added after the article and follow up articles, I will never know if it was a coincidence but I like to think there is a link. I think writing the article was worth it if only for this reason.
- MuffinFlavored 5y ago
  > Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience. I don't know if we are doing it wrong at my job but... I feel like Kubernetes is just a way to orchestrate Docker images running as pods (amongst many other things I'm sure). I know Kubernetes doesn't require Docker images but... how many shops are using k8s without Dockerfile build + tag + publish of images for containers?
  - aflag 5y ago
    My understanding is that you still use docker images, you just don't use docker, but containerd instead.
    - sureglymop 5y ago
      Yes, you use OCI images. The OCI format is a specification for container images based on the Docker Image Manifest Version 2, Schema 2 format. Now, there are multiple ways to build these images, some which are better than using Dockerfiles, though mostly people still use Dockerfiles.
    - hmsimha 5y ago
mianos 5y ago
It seems a lot has not changed: Docker gradually exhausts disk space on BTRFS Open ghost opened this issue on 23 Oct 2016 https://github.com/moby/moby/issues/27653 https://github.com/moby/moby/issues/27653 Still comments this week showing it happens still.
robertwt7 5y ago
I felt the same way when deploying to production with docker manually back then. But honestly after we used k8s we everything just works. Although I realise that GKE is actually moving to containerd for newer clusters? Not sure what made the decision, but last time I had to restart a container manually due to my stupid mistake, the api doesn’t seem to be much different
akanet 5y ago
I once gave a lunch talk at Docker, inc, which included several slides of suggestions of features to add to Docker. Prominently featured was the request for a native command to clean old images. An engineer in attendance remarked incredulously that he could not believe that users would not know how to pipe docker ls into xargs docker rmi.
- chazu 5y ago
  Talk about myopic.
- dboreham 5y ago
  Hence the need for either a) product managers or b) a no sociopath hiring policy.
- easton 5y ago
  Guess he relented or isn't there anymore. docker images prune is really nice.
- yongjik 5y ago
  Hah, that's nothing. As far as I know, the command to update an existing configmap in Kubernetes looks like: kubectl create configmap foo --from-file foo.json -o yaml --dry-run | kubectl replace -f -
- sureglymop 5y ago
  I hate this mentality because.. what if there is no xargs? Don't just assume the standard scenario that everyone has this on their machine. Especially if your program is a statically linked "portable" go binary.
  - jclulow 5y ago
    It's a sad, sad UNIX machine that doesn't even have xargs(1).
    - sureglymop 5y ago
cygned 5y ago
We had a fun issue with Docker yesterday: suddenly, services in our Swarm did not start, apparently because a config could not be mounted. They were running fine for over two years and nobody had touched anything in the Swarm config. Turned out, AWS had decided to upgrade Docker on the server and that version (20.x) is not able to launch services in the Swarm. We have downgraded to 18 now, which works, but is not a long-term solution.
aaccount 5y ago
Any sort of containerization is for people who don't know how to properly package software.

112 points by flanfly 5y ago

It was year 2016, kubernetes did not have jobs, cronjobs, statefulsets. Pods would get stuck in terminating state or container creating state. Networking in kubernetes was wonky. AWS did not have support for EKS. It used be painful. It is year 2021, 1000s of new startups around kubernetes, more features, more resource types. Pods would still get stuck in terminating state or container creating state. It still pretty painful.

HelloNurse 5y ago
> Networking in kubernetes was wonky. Can you elaborate? Does Kubernetes add some thrills to the relatively simple Docker network configuration?
- krab 5y ago
  Yes, Kubernetes (actually its add-ons) provide a virtual network that unifies communication within the cluster so you don't need to care on which computer your service runs.
- jrockway 5y ago
  You can make it as complicated as you want it to be; part of setting up a cluster is picking the networking system ("CNI"). Cloud providers often have their own IPAM (i.e. on Amazon, you get this: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html; https://docs.aws.amazon.com/eks/latest/userguide/pod-network... each Pod gets an IP from your VPC, resulting in weird limits like 17 pods per instance because that's how many IP addresses you can have for that particular instance type throughout EC2).
- kaidon 5y ago
  Please take a seat and let the joys of k8s networking overwhelm your senses: https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n... And yes... Kubernetes network configuration is on a whole different level from docker networking.
  - theptip 5y ago
    To be fair, multi-node networking of any sort is on a different level than single-host docker networking. If you ever tried to use Docker Swarm to network multiple nodes, god help you. Also worth noting that almost all users of K8s don't actually need to operate a cluster, the hosted offerings handle all of that for you. You just need to understand the Service object, and maybe Ingress if you're trying to do some more advanced cert management or API gateway stuff. It's a common meme around here to point in horror to the complexity that is abstracted away under the K8s cluster API, and claim that k8s is really hard to use. I think that's mostly misguided, the hosted offerings like GKE really do a good job of hiding away all that complexity from you. Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.
    - kazen44 5y ago
      > Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes. How is routing from the internet to multiple servers a problem? usually, you have either one of these setups: - you run a loadbalancer that distributes traffic across your nodes. (This loadbalancer could even be distributed thanks to BGP). - you either run your own firewall or have a managed one, in which you either announce your IP prefix yourself, or they are announced for you by your uplink provider. - you run an anycast setup (for, for example, globally distributed DNS). and announce multiples of the same prefix across the globe. Routing in the DFZ does the rest for you. Streched L2 across the globe/internet is also possible (although not very performant) either by doing IPsec tunneling, or by buying/setting up L2VPN services. (either MPLS or VXLAN based).
      - theptip 5y ago
        I didn't say it was a problem. My claim was just that it's easier in GKE than in GCE/EC2. I only mentioned multi-node because exposing a single VM to the internet is trivial -- just give it a public IP -- and thus is not an apples-to-apples comparison with the multi-node load balancing that you get from the entry-level k8s configuration of Service > Pod < Deployment.
- kazen44 5y ago
  the largest issue with kubernetes networking seems to be the lack of integration with modern datacenter networking technology. Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance). Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).
jrockway 5y ago
I haven't seen these failure modes in 2021. We do managed clusters at work and have created around 100,000 of them, and basically all the pods we intend to start start, and all the pods we intend to kill die -- even with autoscaling that provisions new instance types. Our biggest failure mode is TLS certificates failing to provision through Let's Encrypt, but that has nothing to do with Kubernetes (that is a layer above; what we run in Kubernetes). EKS continues to be painful. It has gotten better over the years, but it is a chore compared to GKE. I like to imagine that Jeff Bezos walked into someone's office, said "you're all fired if I don't have Kubersomethings in two weeks", and that's what they launched.
- shaicoleman 5y ago
  What are the EKS pain points?
  - jrockway 5y ago
    The biggest pain point is having to manually use cloudformation to create node pools. This is especially irritating when you just need to roll the Linux version on nodes -- takes half a day to do right. In GKE, it's just a button in the UI (or better, an easy-to-use API), and you can schedule maintenance windows for security updates (which are typically zero downtime anyway, assuming you have the right PodDisruptionBudgets). I think AWS fixed that. I remember when I used it, they said they had some new tool that would handle that, but you had to re-create the cluster from scratch. This was a couple years ago, and is probably decent nowadays. There are other warts, like certain storage classes being unavailable by default (gp3), the whole ENI thing for Pod IPs, the supported version being way out of date, etc. EKS has always felt like "minimum viable product" to me -- they really want you to use their proprietary stuff like ECS/Fargate, CloudFormation, etc. If you're already on AWS and want Kubernetes, it's just what you need. If you could pick any cloud provider for mainly Kubernetes, it wouldn't be my first choice. Having used EKS, GKE, and DOKS, I definitely prefer GKE. GKE is very feature-rich, and the API for managing clusters works well. The nodes are also cheaper than AWS. (I use DOKS for my personal stuff and I haven't had any problems, and it is free, but it's missing features like regional clusters that you probably want for things you make money off of.)
dehrmann 5y ago
Around 2015, I was at Spotify, and we were using a container orchestrator build in-house named Helios. They didn't build it because Kubernetes wasn't invented there; they built it because Kubernetes didn't exist, yet.
- lacion 5y ago
  kubernetes was released in 2014, in late 2015 I already had a cluster marked as released candidate that was put into production early 2016. so im guessing here that spotify wrote their own becouse they had a spesific requirement? nomad was also early days in 2015
  - dehrmann 5y ago
    I'm pretty sure the work started before Kubernetes was released, and even then, it wasn't clear that was going to be the de factor orchestrator.
halfmatthalfcat 5y ago
I have used Kubernetes extensively over the past couple years and have never seen pods stuck in terminating or creating state that didn't have to do with errors in container creation (your Dockerfile/bootstrapping is messed up) or issues with healthchecks.
CSDude 5y ago
We spawn thousands of pods per day for jobs and never get those stuck and it was not the case in 2018 either. Not sure what is it you are doing causes this.

clipradiowallet 5y ago

I know this article is from 2016...but my feelings about it(the article) are unchanged. Some people do not like new things, and they will blog about it in some form or fashion. Maybe their reasoning is valid, maybe it's not - it doesn't matter. Meanwhile...businesses have, and continue, to pay top $$$ for people that will help them do these things. If you want to collect this $$$, get on board. In a few years, the things businesses want to pay $$$ will change. New blog articles about "this new stuff is bad!" will appear, and new job postings paying above-market $$$ will appear also. You can either rail on about the bad(or good) changes, and how it's just everything-old-is-new-again....or you can get with the program, and get paid. In another few years, rinse and repeat.

deleted 5y ago
[deleted]
jjnoakes 5y ago
It's all anecdotal. For example I know many folks who make $$$ doing the boring old thing because it is reliable, it gets results quickly with low risk, the engineers know the tech inside and out, and not many other folks want to work with "boring tech".
- cpach 5y ago
  Just out of curiosity, what are some examples of boring tech in this case?
  - isoskeles 5y ago
    Makes me think of stuff like managing WordPress although I'm not sure if that's an example they had in mind.
  - aprdm 5y ago
    Django, Rails, Postgres...
    - 5y ago

manishsharan 5y ago

This is a blogpost from 2016 . However if we switch to more recent times, my experience with AWS ECS and Fargate has been fairly boring. There was a learning curve to get it to work with cloudformation, vpcs, iam and load balancer .

esotericimpl 5y ago
Agreed, dont see why ECS and fargate isnt used everywhere. It takes a bit of a learning curve to understand how tasks, task definitions ,clusters, services all fir together but once you do, it's pretty straight forward. I've ran ECS in production for over 5 years, can count on one hand where we had any issue related to docker or availability, all were based on code updates we didn't test properly.

_joel 5y ago

In 2016 I started at a company that had no build procedures and deployed to a variety of linux versions, developed on windows. It was a nightmare for administration, no automation, no monitoring. I implemented containers and most of the process was getting the developers on board. Having technical sessions with them to understand what they needed and ease them into the plan so they felt enfranchished. Doing this vastly increased productivity, devs could take off the shelf compose files that were written for common projects (it was a GIS shop) and meant they could concentrate on delivering code. It helped no end. Sure there's issues (albeit a lot fewer as time progressed) with docker but for what it gained in productivity and developer's sanity, it was very welcome.

lmm 5y ago
> In 2016 I started at a company that had no build procedures and deployed to a variety of linux versions, developed on windows. It was a nightmare for administration, no automation, no monitoring. It's no surprise that implementing pretty much any process would be a massive improvement from that. The more interesting question is whether containers offer any advantages for people who already have a (probably significantly more lightweight) build and deployment process.

theamk 5y ago

Back in 2016 during the original discussion of this article, amount said it very well in [0]: "If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area." - Yes, "clean old images" was not implemented back then. His hack is not that bad, and one can filter out in-use images if they want to pretty easily. Anyway, docker does have "docker image prune" now. - Storage driver history discussion is entirely incorrect. No, docker did not invent overlayfs nor overlayfs2. There was a whole big drama of aufs not mainlining, but it was mostly in context of live cd's, not docker. But the big missing thing is: you should not store important data in docker images, Docker is designed to work with transient container. If you have a database, or a high-performance data store, you use volumes, and those _bypass_ docker storage drivers completely. - The database story is completely crazy... judging by their comments, they decided to store the database data in the docker container for some reason and got all the expected problems (unable to recover, hard to migrate, etc....). It is not clear why they didn't put database data on the volume, there is a 2016 StackOverflow question discussing it [0]. Also, "Docker is locking away [...] files through its abstraction [...] It prevents from doing any sort of recovery if something goes wrong." Really? I did recovery with docker, the files are under /var/lib/docker in the directory named with guid, a simple "find" command can locate them. - By default, Docker uses Linux networking and yes, the configuration is complex so it adds overhead. That's why there is --net=host option (which was there for a long time) which just bypasses that all. [0] https://news.ycombinator.com/item?id=12872636 https://news.ycombinator.com/item?id=12872636 [1] https://stackoverflow.com/questions/40167245/how-to-persist-data-using-postgres-docker-image https://stackoverflow.com/questions/40167245/how-to-persist-...

pbecotte 5y ago

Even in 2016, I had been running production services in Docker successfully. Its interesting to me that they see the problem "Docker isn't designed to store data" without also seeing the solution "the docker copy-on-write filesystem isn't designed to be written to production- but volume mounts are". I hadn't seen docker crashing hosts (still haven't) - but I'm guessing that was caused by using the storage drivers. The complaints about their development practices are valid (and haven't really improved), but even then the technology worked well so long as you understood its limitations.

mberning 5y ago

Docker is the chaos monkey incarnate.

crummybowley 5y ago

The issue is not docker, the issue is you treat your servers like pets. Folks need to start building systems that destroy all and re-image fresh. Any other way you are just setting your self up for failure.

esotericimpl 5y ago
except the Database, don't re-image the database.
abraxas 5y ago
And at the heart of this there will be a database or some other data store that will be very much a pet. A very precious, important and fickle pet. Even if it's "serverless" it will have its own needs and wants. This whole "cattle not pets" charade gets on my nerves. Yeah, it's easy to "scale" that which is stateless. Not so fast with stateful. I don't care that I can spin up gazillions of web server instances. My data store is still one and very much stateful.
- sureglymop 5y ago
  I think stateless is good but don't forget that the state is just in another place, another stateful "gear" in your machine. And that, as you said, has to be treated like the pet it is.
- crummybowley 5y ago
  Even so, the topic is docker. Your storage is else where in docker -- not in docker. If it is in docker, then you are already in trouble. Simple solution is storage is a secondary drive/partition -- still automate OS install.
- tene 5y ago
  Persistent identity is fine, it just doesn't belong on individual servers. If the data matters, then it's better for it to be reproducibly managed, durably persisted, etc. For many database needs, there are great options that natively support a highly available cluster. For those needs that don't have a good clustering option, there are great network storage systems that are easy to deploy and use. You don't need to treat hardware as a pet to have a database with persistent identity.

stevebmark 5y ago

> Docker is meant to be stateless. Containers have no permanent disk storage, whatever happens is ephemeral and is gone when the container stops. It's interesting that this misconception made it into a clearly knowledgeable article. Containers have state on the writeable layer that is persisted between container stops and starts.

plainnoodles 5y ago
But I think most people and tools consider "containers" to be volatile storage, like RAM. Non-volatile storage would be volumes. Honestly I think there is a lot to be said for making the writable layer of a container read-only. It makes sure that things like logging, if you care about them, go somewhere safe, or if you don't, get turned off explicitly. And also prevents gotchas like "oops wrote important data to /var/lib/notavolume when I meant to write to /var/lib/therightvolume" that show up at the worst times.
beermonster 5y ago
I’m not sure it’s a misconception. That’s how they’re intended to be used. Cattle not pets. If you don’t get used to not treating them as throw away you can end up accidentally relying on some state. As you say, the top layer is read/write but that doesn’t mean you should be relying on what you write there. Quite the opposite - that state should be somewhere else unless you can afford to lose it. I usually start mine with —-rm so they’re removed on shutdown. I’ve seen people apply security updates via ‘apt update; apt upgrade’ within a running container. Guess what happens when that container is eventually destroyed?

bfrog 5y ago

Heh I just ran into an issue the other day with a coworker where Ubuntu had a patched kernel auto update and break everything. Yep, it’s a sand castle

Theodores 5y ago

Symfony console broke Magento 2 today. Same story.

belter 5y ago

Posted many times before but this is the only one with comments: https://news.ycombinator.com/item?id=12872304 https://news.ycombinator.com/item?id=12872304

zz865 5y ago

Our big project has moved from physical servers to Openshift. Its taken a lot of work, much more than expected. The best thing is that developers like it on their resume, which is a bigger benefit than you'd think as we've kept some good people on the team. For users I see zero benefit. CI pipeline is just more complicated and probably slower. Cost wise it was cheaper for a while but now RedHat are bumping up licensing costs so now I think is about the same costs. Overall it seems like a waste of time, but has been interesting.

geerlingguy 5y ago
OpenShift is about 10x more complex than basic Docker / containers, and probably 2-4x more complex than plain old Kubernetes. I've seen more success from organizations running smaller K3s or K8s clusters (if they need the orchestration) or just running small apps via Docker/Docker Compose separately, using a CI system (even as simple as GitHub Actions) to manage deployments.
- zz865 5y ago
  Yeah its also a problem that our org has infrastructure teams that manage the openshift clusters and they are under resourced so dont help or often can't figure out how to fix problems. Linux sysadmins know what they're doing as the core infrastructure has been mostly the same for the last few decades.
mixermachine 5y ago
Moving from classic servers to containers you get: - Builds with fixed dependencies that never change. Rollback is easy - Easy deployment of a prod environment on a local machine - Fast deployment - Easy automation (use version X with config Y) With Kubernetes (or other derivates like Openshift) you get: - Auto scaling - Fail over - Better resource usage if multiple environments are executed - Abstraction of infrastructure - Zero downtime deployment (biggest point for my company, we deploy >3 times per week) There are applications that do not need Kubernetes or even containers, but is this list really nothing oO? I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things.

rubyist5eva 5y ago

Podman and Kubernetes are like a match made in heaven. Docker was a good first try for most people, but there is so much better technology that exists now.

ChrisArchitect 5y ago

Anything new since this? A history of re-submitted, previously discussed posts: https://news.ycombinator.com/item?id=12872304 https://news.ycombinator.com/item?id=12872304

KronisLV 5y ago

The article seems to mention problems with AUFS, overlay and possibly overlay2 as well. However one of the things that i haven't quite understood, is why people use Docker volumes that much in the first place, or even think that they need to use additional volume plugins in most deployments? If it's a relatively simple deployment, that has some persistent data and it's clear on which nodes the containers could be scheduled (either by label or by hostname), what would prevent someone from just using bind mounts ( https://docs.docker.com/storage/bind-mounts/ https://docs.docker.com/storage/bind-mounts/ )? And if you need to store it on a separate machine, why not just use NFS on the host OS to mount the directory which you will bind mount? Or, alternatively, why not just use GlusterFS or Ceph for that sort of stuff, instead of making Docker attempt to manage it? For example, Docker Swarm fails to launch containers if the bind mount path doesn't exist, but that bit can also be addressed by creating the necessary directory structure with something like Ansible - and then you're not only able to not worry about volumes and the risk of them ever becoming corrupt, but you also have the ability to inspect the contents of the container storage on the actual host. Say, if there are some configuration files that need altering (seeing as not all of the containerized software out there follows 12 Factor principles with environment configuration either), or you just want to do some backups for the data that you've stored in a granular fashion.

jwildeboer 5y ago

Guy who claims to run systems in the hft space, responsible for millions of trades with high values, can’t be bothered to actually pay for support, relies on community and blames everyone but himself for being left alone with his mess. Not sorry.

user5994461 5y ago

Original author from 5 years ago. Surprised to see this here 5 years later. Docker really used to crash a lot back in the days, mostly due to buggy storage drivers. If you were on Debian or CentOS it's very likely that you experienced crashes (though a lot of developers didn't care or didn't understand the reasons the system went unresponsive). There was notably a new version of Debian (with a newer kernel) published the year after my experience. It's a lot more stable now. My experience is that by 2018-2019, Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience. edit: at that time Docker didn't have a way to clear images/containers, it was added after the article and follow up articles, I will never know if it was a coincidence but I like to think there is a link. I think writing the article was worth it if only for this reason.

MuffinFlavored 5y ago
> Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience. I don't know if we are doing it wrong at my job but... I feel like Kubernetes is just a way to orchestrate Docker images running as pods (amongst many other things I'm sure). I know Kubernetes doesn't require Docker images but... how many shops are using k8s without Dockerfile build + tag + publish of images for containers?
- aflag 5y ago
  My understanding is that you still use docker images, you just don't use docker, but containerd instead.
  - sureglymop 5y ago
    Yes, you use OCI images. The OCI format is a specification for container images based on the Docker Image Manifest Version 2, Schema 2 format. Now, there are multiple ways to build these images, some which are better than using Dockerfiles, though mostly people still use Dockerfiles.
  - hmsimha 5y ago

mianos 5y ago

It seems a lot has not changed: Docker gradually exhausts disk space on BTRFS Open ghost opened this issue on 23 Oct 2016 https://github.com/moby/moby/issues/27653 https://github.com/moby/moby/issues/27653 Still comments this week showing it happens still.

robertwt7 5y ago

I felt the same way when deploying to production with docker manually back then. But honestly after we used k8s we everything just works. Although I realise that GKE is actually moving to containerd for newer clusters? Not sure what made the decision, but last time I had to restart a container manually due to my stupid mistake, the api doesn’t seem to be much different

akanet 5y ago

I once gave a lunch talk at Docker, inc, which included several slides of suggestions of features to add to Docker. Prominently featured was the request for a native command to clean old images. An engineer in attendance remarked incredulously that he could not believe that users would not know how to pipe docker ls into xargs docker rmi.

chazu 5y ago
Talk about myopic.
dboreham 5y ago
Hence the need for either a) product managers or b) a no sociopath hiring policy.
easton 5y ago
Guess he relented or isn't there anymore. docker images prune is really nice.
yongjik 5y ago
Hah, that's nothing. As far as I know, the command to update an existing configmap in Kubernetes looks like: kubectl create configmap foo --from-file foo.json -o yaml --dry-run | kubectl replace -f -
sureglymop 5y ago
I hate this mentality because.. what if there is no xargs? Don't just assume the standard scenario that everyone has this on their machine. Especially if your program is a statically linked "portable" go binary.
- jclulow 5y ago
  It's a sad, sad UNIX machine that doesn't even have xargs(1).
  - sureglymop 5y ago

cygned 5y ago

We had a fun issue with Docker yesterday: suddenly, services in our Swarm did not start, apparently because a config could not be mounted. They were running fine for over two years and nobody had touched anything in the Swarm config. Turned out, AWS had decided to upgrade Docker on the server and that version (20.x) is not able to launch services in the Swarm. We have downgraded to 18 now, which works, but is not a long-term solution.

aaccount 5y ago

Any sort of containerization is for people who don't know how to properly package software.