5 ms·

We did at feeder.co. We ran into so many quirks with Docker and Kubernetes that we decided to go back ”bare VM”. It was great! We’re a small team, so any resour

We did at feeder.co. We ran into so many quirks with Docker and Kubernetes that we decided to go back ”bare VM”. It was great! We’re a small team, so any resources save are a great thing. It always felt like we containerized to be able to ”scale up quickly” or ”iterate faster”. That of course was never needed, what was needed however was to scale up infrastructure to keep up with the resources required by Docker Swarm, k8s or just Docker in general. Also, running services is hard already. Adding another layer that caused existing assumptions to break (networking, storage, etc) made it even harder. Bare VMs are crazy fast and running apt install is crazy easy. Not looking back at the moment!

murkt 6y ago
Yep, storage and networking are two things that I haven't mentioned in the post but they definitely annoy. Sometimes (rarely but it happens) network breaks and Docker Swarm falls apart, we then have to restart it. Storage is ughh.
- erikrothoff 6y ago
  It's also hard to understand how the network works in K8s and Docker Swarm. Sometimes we'd hit random slowdowns that were impossible to understand (I'm definitely no networking expert) Just restarting the server or moving to another node would fix it. I really want to use K8s, because it's a cool promise, but for us at least, it was too complicated in reality.
- gtsteve 6y ago
  It depends how you designed your app. My app uses a RDS instance and a S3 bucket for data and file storage. I fel it is a best practice that your containers should be stateless (except perhaps in development). Docker is not very good at storage and I wouldn't recommend using it in that way.
- physicles 6y ago
  If you only have a few VMs, consider ditching swarm and just using docker (with compose) with host mode networking. I’m not sure swarm ever got stable enough to use in prod; I migrated our stuff from swarm to k8s a couple years ago due to similar issues. K8s has been solid but it’s a beast. For storage, why not just mount host dirs into the container for stuff you want to persist? Then you’re no worse off than you were before.
saargrin 6y ago
maybe scale is not needed ,but how do you achieve resiliency with bare metal VMs without adding LB and watchdog layers (which is what k8s is anyway)?
- erikrothoff 6y ago
  We use DigitalOcean's loadbalancer product, we've also tried Cloudflare's for pure HTTP loadbalancing. We also use a single Redis instance for job queues. We use Graphite and Grafana for monitoring system metrics (running a Bitnami Graphite/Grafana instance on AWS because we had credits) And for the rest I guess just keeping the services simple? When we do need to scale up and add a new web server or task runner, it takes about an hour of my day. One thing I realised with going bare-VM is that most services today are insanely stable. MySQL almost never crashes, Redis definitely almost never crashes, Rails/Passenger/Nginx never have any issues. The things that do happen is disks filling up, application bugs causing issues, or actual VM downtime, which is rare but happens when you have 30 VMs. With Docker or K8s it added a super complex layer that is in development and has issues. The 4 months we ran our web servers on K8s, I spent at least 1 month debugging issues that ended at an existing open ticket on Github.
  - tlear 6y ago
    A lot of it been like this for a long time. Postgre, MySQL, Redis, Nginx are bulletproof solid. Sqlite might as well be a hammer and so many businesses could easily run on it, if only there was a way to drop a column :o Amount of $$ that was spent on Docker infra to run couple dozen servers at the last place I used to contract for.. oh and more fun when those machines have GPUs on them(a lot more fun) then they decided to support Singularity as well because.. I don’t know.
  - mwcampbell 6y ago
    > When we do need to scale up and add a new web server or task runner, it takes about an hour of my day. So you manually create and configure your VMs? Do you have some kind of HA for your database? If a VM goes offline, how quickly can you replace it? Maybe you don't need to go back to Docker or Kubernetes, but at least consider using one of the hyperscale cloud providers, with its auto-scaling groups and multiple data centers per region, so you can have a system thatheals itself even while you're asleep or on a plane.
deleted 6y ago
[deleted]

murkt 6y ago

Yep, storage and networking are two things that I haven't mentioned in the post but they definitely annoy. Sometimes (rarely but it happens) network breaks and Docker Swarm falls apart, we then have to restart it. Storage is ughh.

erikrothoff 6y ago
It's also hard to understand how the network works in K8s and Docker Swarm. Sometimes we'd hit random slowdowns that were impossible to understand (I'm definitely no networking expert) Just restarting the server or moving to another node would fix it. I really want to use K8s, because it's a cool promise, but for us at least, it was too complicated in reality.
gtsteve 6y ago
It depends how you designed your app. My app uses a RDS instance and a S3 bucket for data and file storage. I fel it is a best practice that your containers should be stateless (except perhaps in development). Docker is not very good at storage and I wouldn't recommend using it in that way.
physicles 6y ago
If you only have a few VMs, consider ditching swarm and just using docker (with compose) with host mode networking. I’m not sure swarm ever got stable enough to use in prod; I migrated our stuff from swarm to k8s a couple years ago due to similar issues. K8s has been solid but it’s a beast. For storage, why not just mount host dirs into the container for stuff you want to persist? Then you’re no worse off than you were before.

saargrin 6y ago

maybe scale is not needed ,but how do you achieve resiliency with bare metal VMs without adding LB and watchdog layers (which is what k8s is anyway)?

erikrothoff 6y ago
We use DigitalOcean's loadbalancer product, we've also tried Cloudflare's for pure HTTP loadbalancing. We also use a single Redis instance for job queues. We use Graphite and Grafana for monitoring system metrics (running a Bitnami Graphite/Grafana instance on AWS because we had credits) And for the rest I guess just keeping the services simple? When we do need to scale up and add a new web server or task runner, it takes about an hour of my day. One thing I realised with going bare-VM is that most services today are insanely stable. MySQL almost never crashes, Redis definitely almost never crashes, Rails/Passenger/Nginx never have any issues. The things that do happen is disks filling up, application bugs causing issues, or actual VM downtime, which is rare but happens when you have 30 VMs. With Docker or K8s it added a super complex layer that is in development and has issues. The 4 months we ran our web servers on K8s, I spent at least 1 month debugging issues that ended at an existing open ticket on Github.
- tlear 6y ago
  A lot of it been like this for a long time. Postgre, MySQL, Redis, Nginx are bulletproof solid. Sqlite might as well be a hammer and so many businesses could easily run on it, if only there was a way to drop a column :o Amount of $$ that was spent on Docker infra to run couple dozen servers at the last place I used to contract for.. oh and more fun when those machines have GPUs on them(a lot more fun) then they decided to support Singularity as well because.. I don’t know.
- mwcampbell 6y ago
  > When we do need to scale up and add a new web server or task runner, it takes about an hour of my day. So you manually create and configure your VMs? Do you have some kind of HA for your database? If a VM goes offline, how quickly can you replace it? Maybe you don't need to go back to Docker or Kubernetes, but at least consider using one of the hyperscale cloud providers, with its auto-scaling groups and multiple data centers per region, so you can have a system thatheals itself even while you're asleep or on a plane.

deleted 6y ago

[deleted]