7 ms·

I recently migrated a small Rails project (16,000 monthly users) from Heroku to Helm/Kubernetes, after having tried Fly.io and everything else. It was easy, and

by bkazez 4y ago

I recently migrated a small Rails project (16,000 monthly users) from Heroku to Helm/Kubernetes, after having tried Fly.io and everything else. It was easy, and I like the peace of mind of zero-downtime deploys and non-beta infrastructure. Cost is comparable to Heroku. My writeup, in case it helps others: https://www.vmii.org/blog/2023/03/12/kubernetes/ https://www.vmii.org/blog/2023/03/12/kubernetes/

number6 4y ago
Thanks for your writeup I am thinking of moving our Django Apps to kubernetes
robertlagrant 4y ago
Thanks for this. Having tried similar with AWS, and having to specify security groups, static IPs, etc etc with many annotations in EKS, that Google Cloud version looks refreshingly simple.
selcuka 4y ago
I was going to ask "why", but you already explained it in your writeup: > The prevailing wisdom is that Kubernetes is overkill for a small Rails/Elasticsearch app, but overkill is my life philosophy.
- loveparade 4y ago
  I'll never understand these arguments against Kubernetes and its complexity that are so prevalent on HN. Yup, k8s has a learning curve, just like any new technology. You'll need to spend a couple days understanding it. But once you've grasped the abstractions it's actually quite easy to setup, operate, and manage, even for small side projects. I'd pick it any day over "third party magic" that is a black box and I have no control over. It feels like many developers these days are so spoiled by magic services that they are unwilling to even spend a few days going deep into something. Everything has to work in minutes. Next, what inevitably happens is that services shut down, pricing changes, or something stops working, and they have no way to debug it or move off it. And then we get customer support complaints and posts on HN about it. These developers look for the next shiny 3rd party service that solves the problem immediately and repeat the cycle, without ever learning anything that helps in the long term.
  - turtlebits 4y ago
    The problem is that the surface area for k8s is too large for one person to truly understand it. Sure, you can get up and running in a few days, but good luck when your cluster mysteriously stops working and ignores all kubectl commands.
    - dewey 4y ago
      In the past 6 years of working with Kubernetes in production every day, I've never had that happen to me, so it doesn't sound like the error case I should optimize for.
  - bshipp 4y ago
    My issue is that I have 40 dockerized containers set up behind a reverse proxy and, although I want to play with k8s to learn, I've already dumped so much time and effort into docker-compose that I'm cautious about migrating to a new system. Is the architecture substantially different?
    - margorczynski 4y ago
      Well it's distributed. I would say single machine vs distributed system is a really big difference. A lot problems and complexity.
    - 8n4vidtmkvmk 4y ago
      I went down that road a few years ago. Dockerized my app because I figured that was the first step. Then naturally docker-compose to bring the pieces together, right? No. Was upset that was not the next step to Kubernetifying an app. Use Kustomize instead. It's not that bad. Did take days or a few weekends but it requires very little maintenance now and its easy to spin up new apps.
  - rapind 4y ago
    > It feels like many developers these days are so spoiled by magic services that they are unwilling to even spend a few days going deep into something. Quite the opposite. K8s are a big opaque ball of magical complexity to most of us devs for sure (as is heroku). However, for 100% of my use cases nothing should be more complex than the database. I’m not opposed to learning about it, but reducing complexity wherever reasonable has paid off for me.
alexellisuk 4y ago
The cost of Heroku was 50 USD / mo for 1GB of application RAM. I can't imagine that a GKE cluster, plus nodes, plus cloud SQL (and you refer to AWS S3 too?), plus bandwidth all adds up to less than 50 USD / mo. What's your cost breakdown like? And how are you finding it so far?
- vasco 4y ago
  Around 8 years ago I migrated a full company from heroku to AWS and cut costs by 3/4. At that time I remember all the startups around us doing the same, with similar results.
- bkazez 4y ago
  50 USD/mo was for 1 production Heroku app dyno (no db!), with no Redis or Sidekiq. On GKE we added Redis and Sidekiq, all duplicated for staging (on Spot VMs), for 58 USD/mo. (One single-zone cluster is free on GCP.)
foldr 4y ago
>If you are still with me, this is about the point where things start to get incomprehensible. Just close your eyes and keep copying/pasting yaml. Excellent writeup. This line seems to track with what other people have told me about maintaining Kubernetes deployments. Almost no-one really understands the YAML (beyond the basics), so you end up with services defined via tens of thousands of lines of unmaintainable copy pasta YAML. This problem probably isn't going to surface so quickly (if ever) for a smaller project.

by bkazez 4y ago

number6 4y ago

Thanks for your writeup I am thinking of moving our Django Apps to kubernetes

robertlagrant 4y ago

Thanks for this. Having tried similar with AWS, and having to specify security groups, static IPs, etc etc with many annotations in EKS, that Google Cloud version looks refreshingly simple.

selcuka 4y ago

I was going to ask "why", but you already explained it in your writeup: > The prevailing wisdom is that Kubernetes is overkill for a small Rails/Elasticsearch app, but overkill is my life philosophy.

loveparade 4y ago
I'll never understand these arguments against Kubernetes and its complexity that are so prevalent on HN. Yup, k8s has a learning curve, just like any new technology. You'll need to spend a couple days understanding it. But once you've grasped the abstractions it's actually quite easy to setup, operate, and manage, even for small side projects. I'd pick it any day over "third party magic" that is a black box and I have no control over. It feels like many developers these days are so spoiled by magic services that they are unwilling to even spend a few days going deep into something. Everything has to work in minutes. Next, what inevitably happens is that services shut down, pricing changes, or something stops working, and they have no way to debug it or move off it. And then we get customer support complaints and posts on HN about it. These developers look for the next shiny 3rd party service that solves the problem immediately and repeat the cycle, without ever learning anything that helps in the long term.
- turtlebits 4y ago
  The problem is that the surface area for k8s is too large for one person to truly understand it. Sure, you can get up and running in a few days, but good luck when your cluster mysteriously stops working and ignores all kubectl commands.
  - dewey 4y ago
    In the past 6 years of working with Kubernetes in production every day, I've never had that happen to me, so it doesn't sound like the error case I should optimize for.
- bshipp 4y ago
  My issue is that I have 40 dockerized containers set up behind a reverse proxy and, although I want to play with k8s to learn, I've already dumped so much time and effort into docker-compose that I'm cautious about migrating to a new system. Is the architecture substantially different?
  - margorczynski 4y ago
    Well it's distributed. I would say single machine vs distributed system is a really big difference. A lot problems and complexity.
  - 8n4vidtmkvmk 4y ago
    I went down that road a few years ago. Dockerized my app because I figured that was the first step. Then naturally docker-compose to bring the pieces together, right? No. Was upset that was not the next step to Kubernetifying an app. Use Kustomize instead. It's not that bad. Did take days or a few weekends but it requires very little maintenance now and its easy to spin up new apps.
- rapind 4y ago
  > It feels like many developers these days are so spoiled by magic services that they are unwilling to even spend a few days going deep into something. Quite the opposite. K8s are a big opaque ball of magical complexity to most of us devs for sure (as is heroku). However, for 100% of my use cases nothing should be more complex than the database. I’m not opposed to learning about it, but reducing complexity wherever reasonable has paid off for me.

alexellisuk 4y ago

The cost of Heroku was 50 USD / mo for 1GB of application RAM. I can't imagine that a GKE cluster, plus nodes, plus cloud SQL (and you refer to AWS S3 too?), plus bandwidth all adds up to less than 50 USD / mo. What's your cost breakdown like? And how are you finding it so far?

vasco 4y ago
Around 8 years ago I migrated a full company from heroku to AWS and cut costs by 3/4. At that time I remember all the startups around us doing the same, with similar results.
bkazez 4y ago
50 USD/mo was for 1 production Heroku app dyno (no db!), with no Redis or Sidekiq. On GKE we added Redis and Sidekiq, all duplicated for staging (on Spot VMs), for 58 USD/mo. (One single-zone cluster is free on GCP.)

foldr 4y ago

>If you are still with me, this is about the point where things start to get incomprehensible. Just close your eyes and keep copying/pasting yaml. Excellent writeup. This line seems to track with what other people have told me about maintaining Kubernetes deployments. Almost no-one really understands the YAML (beyond the basics), so you end up with services defined via tens of thousands of lines of unmaintainable copy pasta YAML. This problem probably isn't going to surface so quickly (if ever) for a smaller project.