5 ms·

Original author from 5 years ago. Surprised to see this here 5 years later. Docker really used to crash a lot back in the days, mostly due to buggy storage dri

Original author from 5 years ago. Surprised to see this here 5 years later. Docker really used to crash a lot back in the days, mostly due to buggy storage drivers. If you were on Debian or CentOS it's very likely that you experienced crashes (though a lot of developers didn't care or didn't understand the reasons the system went unresponsive). There was notably a new version of Debian (with a newer kernel) published the year after my experience. It's a lot more stable now. My experience is that by 2018-2019, Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience. edit: at that time Docker didn't have a way to clear images/containers, it was added after the article and follow up articles, I will never know if it was a coincidence but I like to think there is a link. I think writing the article was worth it if only for this reason.

MuffinFlavored 5y ago
> Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience. I don't know if we are doing it wrong at my job but... I feel like Kubernetes is just a way to orchestrate Docker images running as pods (amongst many other things I'm sure). I know Kubernetes doesn't require Docker images but... how many shops are using k8s without Dockerfile build + tag + publish of images for containers?
- aflag 5y ago
  My understanding is that you still use docker images, you just don't use docker, but containerd instead.
  - sureglymop 5y ago
    Yes, you use OCI images. The OCI format is a specification for container images based on the Docker Image Manifest Version 2, Schema 2 format. Now, there are multiple ways to build these images, some which are better than using Dockerfiles, though mostly people still use Dockerfiles.
  - hmsimha 5y ago
    Containerd is the docker runtime, so it's still used to run/manage containers on the kubelet. Crio is another option: https://kubernetes.io/docs/setup/production-environment/container-runtimes/ https://kubernetes.io/docs/setup/production-environment/cont... . I think the distinction in the docs there is that with just containerd, you don't get all the docker tooling My understanding is that docker was the most common, but I'm not actually sure if that's still the case. CRI is the interface k8s uses to manage the container engine, and I think CRI-O is the most lightweight, but also gives you less/worse tooling for debugging if you need to drop into a kubelet. The images are just OCI-spec now, but if memory serves, Docker used a different image format in the past, and it was another containerization company (coreos/rkt) working on an open standard which eventually become OCI
theamk 5y ago
Hey, always wondered: how come you tried to store the DB data on the docker’s system disk instead of the external volume? This would bypass all the driver stability problems, make the data recoverable trivially, decrease cpu usage, etc... I know that there are posts from 2016 explaining how to do so, but the article had no mention of this?
- user5994461 5y ago
  That seems to be a common misconception. I don't know why anybody thought I wasn't aware of external volumes, I was. It made little difference, the entire ecosystem was highly unstable. Containers could still fail, the docker daemon could hang, and the host could kernel panic any minute. For databases that meant downtime and potential data corruption. Besides, there is a major use case to run temporary databases for CI testing. I remember a lot of issues when running performance tests or seeding the database with initial data, basically anything that is performance intensive. I think the unstable filesystem played a role but it was far from the only root cause. Honestly it's been 6 years now and I've never seen a company running any critical databases inside docker (but I've seen it for testing). I've known a lot of companies that said they did or they had plans to move existing databases, but that wasn't real. At the end of the day there was no sysadmin/DBA/devops who would do it, they understood that it would come back to bite them (many had enough troubles just running ephemeral web services). Maybe it's the harder part to grasp but database is really a different mindset from web development, you cannot risk losing customer data, this would be an extinction level event for your job and for the company.
  - dilyevsky 5y ago
    We run almost all of our DBs - pg, cockroachdb, clickhouse and etcd/kafka/redis (if you can consider those a database) inside docker/crio inside k8s. In production under high load. Works really well. We’ve had more crashes of the db itself than anything container/node related
  - hodgesrm 5y ago
    If you mean using Docker containers, well those are pretty stable. There are hundreds of companies running ClickHouse on Kubernetes, which deploys using Docker containers. Some of them run on very large K8s clusters. We've seen very few problems. The one issue I've seen specific to Docker is that you can run into configuration errors that keep them from coming up. That happens occasionally and it can be tricky to debug. Disclaimer: My company Altinity wrote the ClickHouse Kubernetes Operator, which is also a Docker container.

MuffinFlavored 5y ago

> Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience. I don't know if we are doing it wrong at my job but... I feel like Kubernetes is just a way to orchestrate Docker images running as pods (amongst many other things I'm sure). I know Kubernetes doesn't require Docker images but... how many shops are using k8s without Dockerfile build + tag + publish of images for containers?

aflag 5y ago
My understanding is that you still use docker images, you just don't use docker, but containerd instead.
- sureglymop 5y ago
  Yes, you use OCI images. The OCI format is a specification for container images based on the Docker Image Manifest Version 2, Schema 2 format. Now, there are multiple ways to build these images, some which are better than using Dockerfiles, though mostly people still use Dockerfiles.
- hmsimha 5y ago
  Containerd is the docker runtime, so it's still used to run/manage containers on the kubelet. Crio is another option: https://kubernetes.io/docs/setup/production-environment/container-runtimes/ https://kubernetes.io/docs/setup/production-environment/cont... . I think the distinction in the docs there is that with just containerd, you don't get all the docker tooling My understanding is that docker was the most common, but I'm not actually sure if that's still the case. CRI is the interface k8s uses to manage the container engine, and I think CRI-O is the most lightweight, but also gives you less/worse tooling for debugging if you need to drop into a kubelet. The images are just OCI-spec now, but if memory serves, Docker used a different image format in the past, and it was another containerization company (coreos/rkt) working on an open standard which eventually become OCI

theamk 5y ago

Hey, always wondered: how come you tried to store the DB data on the docker’s system disk instead of the external volume? This would bypass all the driver stability problems, make the data recoverable trivially, decrease cpu usage, etc... I know that there are posts from 2016 explaining how to do so, but the article had no mention of this?

user5994461 5y ago
That seems to be a common misconception. I don't know why anybody thought I wasn't aware of external volumes, I was. It made little difference, the entire ecosystem was highly unstable. Containers could still fail, the docker daemon could hang, and the host could kernel panic any minute. For databases that meant downtime and potential data corruption. Besides, there is a major use case to run temporary databases for CI testing. I remember a lot of issues when running performance tests or seeding the database with initial data, basically anything that is performance intensive. I think the unstable filesystem played a role but it was far from the only root cause. Honestly it's been 6 years now and I've never seen a company running any critical databases inside docker (but I've seen it for testing). I've known a lot of companies that said they did or they had plans to move existing databases, but that wasn't real. At the end of the day there was no sysadmin/DBA/devops who would do it, they understood that it would come back to bite them (many had enough troubles just running ephemeral web services). Maybe it's the harder part to grasp but database is really a different mindset from web development, you cannot risk losing customer data, this would be an extinction level event for your job and for the company.
- dilyevsky 5y ago
  We run almost all of our DBs - pg, cockroachdb, clickhouse and etcd/kafka/redis (if you can consider those a database) inside docker/crio inside k8s. In production under high load. Works really well. We’ve had more crashes of the db itself than anything container/node related
- hodgesrm 5y ago
  If you mean using Docker containers, well those are pretty stable. There are hundreds of companies running ClickHouse on Kubernetes, which deploys using Docker containers. Some of them run on very large K8s clusters. We've seen very few problems. The one issue I've seen specific to Docker is that you can run into configuration errors that keep them from coming up. That happens occasionally and it can be tricky to debug. Disclaimer: My company Altinity wrote the ClickHouse Kubernetes Operator, which is also a Docker container.