7 ms·

Moving from Heroku to Google Kubernetes Engine

225 points by shosti 8y ago

neya 8y ago
I'm interested to know why they didn't move to Google AppEngine instead, which offers a better experience and more advanced features overall. Especially considering that AppEngine is a direct competitor to Heroku than Kubernetes engine.
- tlrobinson 8y ago
  The post has a whole section called "Why Kubernetes?"
  - neya 8y ago
    It doesn't explain "Why not AppEngine"? Which was my question..
    - tlrobinson 8y ago
      Presumably because of all the reasons they chose Kubernetes? FTA: * Kubernetes has a huge amount of traction in the DevOps landscape, with managed implementations from all the major cloud vendors and virtually endless training materials and complementary technologies. * Kubernetes is open source, which was a major plus: it meant that we could avoid vendor lock-in and implement local development environments that mimic production. * Kubernetes has a large feature set that fit well with our requirements, including our more exotic necessities like autoscaling based on custom metrics. AFAIK none of those are true of AppEngine.
      - riku_iki 8y ago
        > Kubernetes has a huge amount of traction in the DevOps landscape the appeal of AppEngine is that you don't need DevOps, google manages and monitors services for you.
        tlrobinson 8y ago
        That's also true of "managed" Kubernetes platforms like Google Kubernetes Engine and Amazon EKS, the difference being no (or at least less) vendor lock-in.
        ukd1 8y ago
        ^ this is why it won
        riku_iki 8y ago
        But you still need to write your app deployment scripts, integrate logs and monitoring.
      - spyspy 8y ago
        GAE comes with a full local dev setup out of the box. You can even run multiple apps with one command.
      - auslander 8y ago
        > Kubernetes has a huge amount of traction in the DevOps landscape Definition of 'hype' here :)
- alasdair_ 8y ago
  >I'm interested to know why they didn't move to Google AppEngine instead, which offers a better experience and more advanced features overall. Especially considering that AppEngine is a direct competitor to Heroku than Kubernetes engine. AppEngine has an enormous number of limitations that you only hit once you scale up and gets very expensive very quickly.
  - rlancer 8y ago
    Is standard more expensive in reality? Any data on this, flex is certainly very expensive
    - haimez 8y ago
      Flex is not at all very expensive. Per instance cost is about 1/2 the cost compared to heroku, putting it on par with the likes of ECS in terms of cost but ships with usable monitoring, logging, and metrics out of the box with extremely generous free tiers that stay free way longer and that are still marginally cheaper than what you’d pay in AWS land if you exceed them.
  - neya 8y ago
    > AppEngine has an enormous number of limitations I have used AppEngine in production for about 50+ clients and I am genuinely really curious to know what these limitations are. Maybe it is dependent on the programming language/framework? I run Phoenix/Elixir with Vue.JS + PostgreSQL as standard for most of my clients, it's really a breeze to work with. In addition, these are the advantages of working with AppEngine: https://news.ycombinator.com/item?id=17516530 https://news.ycombinator.com/item?id=17516530
    - clhodapp 8y ago
      The AppEngine Standard Java 8 API is severely limited because it is coupled to the Servlet API, which severely screwed up on the design of its async API. As a Scala developer, this limitation pretty much sinks the platform for me, since the flexible environment is really expensive and has a worse value proposition vs managed Kubernetes.
mahgnous 8y ago
That's not much of a stretch there.
kerng 8y ago
Crossing my fingers for you that GCP won't shut down your account because some mysterious Google AI decides so.
- ec109685 8y ago
  One of the reasons for choosing GKE was reducing vendor lock in.
- jasonvorhe 8y ago
  That should no longer happen since they have added manual oversight to account closures.
wrs 8y ago
We are happy with GKE, but have gone from Cloud SQL PostgreSQL back to self-managed PostgreSQL VMs. Cloud SQL is still stuck on version 9.6, and still has no point-in-time recovery ability. It's disappointing because the rest of the GCP offering is pretty well thought out and making rapid progress but Cloud SQL seems not to be getting much love.
- philliphaydon 8y ago
  AWS is definitely better for managed PostgreSQL compared to GCE/Azure.
  - wapoamspomw 8y ago
    Aurora Postgres is pretty fantastic.
    - jdreaver 8y ago
      We just switched to Aurora Postgres, and I agree wholeheartedly. The main DB client is a web app with ~4000 txns/second, and we instantly got a ~30% performance boost. We also got much more consistent performance for our slower queries thanks to Aurora's fast, custom storage engine.
      - riku_iki 8y ago
        > The main DB client is a web app with ~4000 txns/second curious what is your bill for this
        jdreaver 8y ago
        For Aurora we are on a db.r4.8xlarge, which is ~$3340 per month for the instance alone. Aurora also charges for IOPS against storage, and we pay about $350 per month for those IOPS. Our DB is ~400 GB, so ~$40 per month for that as well. I am pretty sure we could live comfortably on a db.r4.4xlarge, but we have some bursty analytics applications that sometimes bring our load up for about a minute, and we like the peace of mind of headroom during peak hours (we are in edtech with super predictable traffic throughout US school hours). Total side note: I just went to the pricing page and it looks like they just released db.r5 instances for Aurora (https://aws.amazon.com/rds/aurora/pricing/ https://aws.amazon.com/rds/aurora/pricing/)! It might be time to try those out. I don't see an announcement just yet though...
deleted 8y ago
[deleted]
zenlot 8y ago
For some reason those blogs become boring. Whoever moves to Kubernetes, or chooses to use some service of major cloud provider feels a need to write a blog about it having a same theme as everyone else. Bonus points if it explains what containers are and differences between EKS, ECS and others.
- techslave 8y ago
  well you are missing the point. the tech details are irrelevant in posts like this. no one, but no one, is throwing down knowledge and insight that anyone and their mama doesn’t know or can’t easily acquire. people just don’t give away their secret sauce that easily. the 2 points are a. a recruiting signal. it lets candidates know “we are like you”. we care about the tech stack and what we do. we care about elegant and “good” solutions. b. it gives company devs a public sounding board. a chance to have a bigger voice than in house obscurity. if you think these blogs are really about the tech, well then yes they are quite boring.
  - nitinreddy88 8y ago
    I guess it's becoming a trend "we moved from here to here" and let's make it to top of HN. Unless current provider has horrible support/issues which you want to specify to help others, these articles are useless. HN community making these articles to front page which neither describes architect challenges/scalability challenges is pointless
    - staticassertion 8y ago
      I found the section on their evaluation of other potential systems like ECS to be interesting - it's something I'm currently considering myself.
      - ukd1 8y ago
        glad it was useful!
mcguireio 8y ago
Now if only they could quit spamming every inbox of every company I've worked at, id be impressed. Their sales automation is out of control. Honestly...I write them off before ever looking at their services.
markbnj 8y ago
We've been running in production on GKE for a little over two years and it's been a solid platform since day one. It's nice to read articles like this and see others coming to the same conclusions we've come to. If your practices and workflow are oriented to containers and you outgrow your PAAS then k8s is the logical place to land. With respect to the choice of helm: we started out rolling our own pipeline with sed and awk and the usual suspects. When that became too complex helm was just taking off and we moved to that. We still use it to install various charts from stable that implement infrastructure things. For our own applications we found that there was just too much cognitive dissonance between the helm charts and the resulting resources. Essentially the charts and the values we plugged into them became a second API to kubernetes, obfuscating the actual API below. The conventions around the "package manager" role that the tool has taken for itself also contribute to lessen readability due to scads of boilerplate and name mangling. We recently started deploying things in a new pipeline based on kustomize. We keep base yaml resources in the application repo and apply patches from a config repo to finalize them for a given environment. So far it's working out quite well and the applications engineers like it much better. Now with kubectl 1.14 kustomize's features have been pulled in to that tool, something I have mixed feelings about, but at least the more declarative approach does seem to be the way the wind is blowing.
- zapita 8y ago
  Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14?
  - markbnj 8y ago
    > Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14? It's nothing too surprising :). Simply a preference for simple, compose-able tools. I personally feel like patching yaml client-side is outside the scope of a "ctl" tool.
- physicles 8y ago
  Perhaps I don’t understand the design decisions behind Helm, but it’s always struck me as having a severe impedance mismatch with k8s itself. It defines another entirely different schema, and relies on an agent running in your cluster that’s also trying to reconcile desired state with actual state (which k8s itself is also doing). I’m skeptical that you could use it extensively without also understanding the k8s stuff underneath it. kustomize came along just as it’s become untenable for us to copy/paste config to multiple environments. I like that it’s pretty much the simplest possible way to customize yaml, and plan to dive in soon.
autotune 8y ago
Basically moved to my dream tech stack, jealous of the SRE’s over there.
- maciejgryka 8y ago
  We are hiring :) https://rainforestqa.com/careers https://rainforestqa.com/careers
  - autotune 8y ago
    Ha, would love to as a remote candidate out of NYC, don't see anything for SRE though.
shay_ker 8y ago
> At one point we attempted to migrate to Heroku Shield to address some of these issues, but we found that it wasn’t a good fit for our application. This part seems very hand wavy, given that Heroku Shield would've solved many (all?) of their problems. > We were also running into limitations with Heroku on the compute side: some of our newer automation-based features involve running a large number of short-lived batch jobs, which doesn’t work well on Heroku (due to the relatively high cost of computing resources). How much memory did their batch jobs actually need? If they're using Rails, then I'm assuming they're just running a bunch of Sidekiq jobs that are querying PG. I'm surprised that they'd need that much in terms of compute resources. They should be able to get very, very far by making PG do a lot of the work, or by streaming data from PG and not holding a lot of data in memory. Even if they did need all this, the following two options seem WAY easier to manage: 1) Use dokku to run your super-intense Sidekiq batch jobs on beefy EC2 instances. You can still schedule them in your Rails app in Heroku, no big deal. Many engineering teams have to do this type of split-up anyway when it comes to Application Engineers and Data Engineers, this is just a simpler way to do it. 2) Similar to 1), use a different language runtime for the batch jobs. If you really need to run CPU intensive jobs, why are you using Ruby? If the jobs aren't so intense to mandate maintaining two languages (fwiw, not that hard), why will moving to k8s solve the issue? Personally, I'm not sold on their decision to move to Kubernetes, and I use Kubernetes for my job.
- shosti 8y ago
  > This part seems very hand wavy, given that Heroku Shield would've solved many (all?) of their problems. Author here; I don’t want to go into too much detail, but we tried Shield early on and had a negative experience that made us wary about using the platform (it seems to use a different tech stack under the hood from “normal” Heroku and lacks a lot of the things that make Heroku great). Also it’s very expensive compared to VPC-based solutions on AWS and GCP. W.R.T. the batch jobs, I think I didn’t explain super well—we are using a different language and runtime from our “normal” background processing jobs (which use worker queues in Rails), it’s just that Heroku isn’t very well suited for the use case (which is basically FaaS-like but with long-lived jobs). The “split” workflow you described is basically what we were doing (but with AWS Batch instead of Dokku); it’s just that it’s more cost-efficient to consolidate everything into one cluster (especially with preemptible gke nodes) and also better to have a common set of tooling for the Ops team. To be fair, we haven’t yet completed the move from Batch to k8s so it’s possible that part of the plan won’t pan out as expected.
_0nac 8y ago
If you can't wait for their teaser of "In a future post, I’ll cover the migration process itself", the GCP site has a hands-on tutorial of migrating an app which may prove interesting: https://cloud.google.com/solutions/migrating-ruby-on-rails-apps-on-heroku-to-gke https://cloud.google.com/solutions/migrating-ruby-on-rails-a... Disclaimer: I work for GCP and wrote most of that :D At the end of the day, Heroku and GKE are rather different beasts with different philosophies, so migrations are never going to be 1:1. I expect this to become simpler over time as tooling matures though, eg. using https://buildpacks.io https://buildpacks.io to build Docker images instead of having to craft them by hand seems promising.
- ZeroCool2u 8y ago
  I've been playing around with GCP, mostly app engine, the past couple weeks making a toy application to get a little more comfortable with building and designing serverless apps. Just wanted to say you, and whoever else, is writing those docs does an amazing job. Seriously, I've tried all 3 platforms, because of work and GCP is by far the easiest to become productive with and the main reason is the glorious documentation. Keep up the great work!
  - devlance 7y ago
    Great to hear about your experience with the docs and appreciate the call out. You can use the "Send Feedback" button on any of the doc pages to let us know about any feedback you have. Real people read it.
    - ZeroCool2u 7y ago
      I'll keep that in mind!
NightlyDev 8y ago
I've always wondered: Is kubernetes hard to host on your own on a couple of servers for production? I've never tried, but I've heard a lot of people saying it's very hard, but people are often complaining about the most basic stuff as being hard.. Sooo..?
- bauerd 8y ago
  Depends on what you mean by hard. Bootstrapping a K8s control plane? Not so much, kubeadm does all the heavylifting for you. Keeping your install recent, etcd performant and backed up, maintaining and scaling underlying filesystems etc. is still a full-time job for production usage imho. Essentially you'll find yourself operating a number of distributed systems below your application stack.
- physicles 8y ago
  More important than whether it’s easy or hard, is the fact that it’s unnecessary. GKE is great and you only pay for your compute (as opposed to EKS, which is over $140/mo for the control plane, a move which always struck me as a terrible business decision). I maintain two prod and two test environments: AWS+kops (since before EKS existed), Alibaba+kubeadm (because China), and two local ones that used to be minikube but are now just kubeadm. I spent a total of about two months getting the knowledge to do that, and these days I spend about 4 hours a week on maintenance. I do minor version upgrades but put off major ones because there’s not much business value and upgrades can break stuff. It’s my least favorite part of my job. I’d rather be coding or doing architecture, which is what I spend most of my time doing. We’ll be moving the AWS stuff to GKE in a couple months. Still haven’t heard much on the quality of Alibaba’s offering. Their IaaS is solid but some of their other services will sometimes throw errors that return 0 hits on google, no documentation, which scares me.
- flurdy 8y ago
  You would have to follow this the whole way: https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
  - 8y ago
rdsubhas 8y ago
I work at a large European startup. We are also heavily invested in and love kubernetes, but this is a totally apples to oranges comparison. Heroku is a PaaS and Kubernetes is a CaaS. K8s is great at what does, but to make it act like Heroku is a huge amount of effort and needs a team to manage the tooling around it. Assuming that such team is not needed, and k8s can simply be used like heroku with a bunch of extra CLI tools, usually leads to "Wild Wild West" clusters.
auslander 8y ago
> ..agile without hiring a large Ops team, .. we were beginning to outgrow Heroku. We ended up .. running on Google Kubernetes Engine Or you could hire few cloud infra devs and stick to autoscaled VMs (EC2), not spending time on k8s? I bet k8s took you a while to set up and maintain.

225 points by shosti 8y ago

neya 8y ago

I'm interested to know why they didn't move to Google AppEngine instead, which offers a better experience and more advanced features overall. Especially considering that AppEngine is a direct competitor to Heroku than Kubernetes engine.

tlrobinson 8y ago
The post has a whole section called "Why Kubernetes?"
- neya 8y ago
  It doesn't explain "Why not AppEngine"? Which was my question..
  - tlrobinson 8y ago
    Presumably because of all the reasons they chose Kubernetes? FTA: * Kubernetes has a huge amount of traction in the DevOps landscape, with managed implementations from all the major cloud vendors and virtually endless training materials and complementary technologies. * Kubernetes is open source, which was a major plus: it meant that we could avoid vendor lock-in and implement local development environments that mimic production. * Kubernetes has a large feature set that fit well with our requirements, including our more exotic necessities like autoscaling based on custom metrics. AFAIK none of those are true of AppEngine.
    - riku_iki 8y ago
      > Kubernetes has a huge amount of traction in the DevOps landscape the appeal of AppEngine is that you don't need DevOps, google manages and monitors services for you.
      - tlrobinson 8y ago
        That's also true of "managed" Kubernetes platforms like Google Kubernetes Engine and Amazon EKS, the difference being no (or at least less) vendor lock-in.
        ukd1 8y ago
        ^ this is why it won
        riku_iki 8y ago
        But you still need to write your app deployment scripts, integrate logs and monitoring.
    - spyspy 8y ago
      GAE comes with a full local dev setup out of the box. You can even run multiple apps with one command.
    - auslander 8y ago
      > Kubernetes has a huge amount of traction in the DevOps landscape Definition of 'hype' here :)
alasdair_ 8y ago
>I'm interested to know why they didn't move to Google AppEngine instead, which offers a better experience and more advanced features overall. Especially considering that AppEngine is a direct competitor to Heroku than Kubernetes engine. AppEngine has an enormous number of limitations that you only hit once you scale up and gets very expensive very quickly.
- rlancer 8y ago
  Is standard more expensive in reality? Any data on this, flex is certainly very expensive
  - haimez 8y ago
    Flex is not at all very expensive. Per instance cost is about 1/2 the cost compared to heroku, putting it on par with the likes of ECS in terms of cost but ships with usable monitoring, logging, and metrics out of the box with extremely generous free tiers that stay free way longer and that are still marginally cheaper than what you’d pay in AWS land if you exceed them.
- neya 8y ago
  > AppEngine has an enormous number of limitations I have used AppEngine in production for about 50+ clients and I am genuinely really curious to know what these limitations are. Maybe it is dependent on the programming language/framework? I run Phoenix/Elixir with Vue.JS + PostgreSQL as standard for most of my clients, it's really a breeze to work with. In addition, these are the advantages of working with AppEngine: https://news.ycombinator.com/item?id=17516530 https://news.ycombinator.com/item?id=17516530
  - clhodapp 8y ago
    The AppEngine Standard Java 8 API is severely limited because it is coupled to the Servlet API, which severely screwed up on the design of its async API. As a Scala developer, this limitation pretty much sinks the platform for me, since the flexible environment is really expensive and has a worse value proposition vs managed Kubernetes.

mahgnous 8y ago

That's not much of a stretch there.

kerng 8y ago

Crossing my fingers for you that GCP won't shut down your account because some mysterious Google AI decides so.

ec109685 8y ago
One of the reasons for choosing GKE was reducing vendor lock in.
jasonvorhe 8y ago
That should no longer happen since they have added manual oversight to account closures.

wrs 8y ago

We are happy with GKE, but have gone from Cloud SQL PostgreSQL back to self-managed PostgreSQL VMs. Cloud SQL is still stuck on version 9.6, and still has no point-in-time recovery ability. It's disappointing because the rest of the GCP offering is pretty well thought out and making rapid progress but Cloud SQL seems not to be getting much love.

philliphaydon 8y ago
AWS is definitely better for managed PostgreSQL compared to GCE/Azure.
- wapoamspomw 8y ago
  Aurora Postgres is pretty fantastic.
  - jdreaver 8y ago
    We just switched to Aurora Postgres, and I agree wholeheartedly. The main DB client is a web app with ~4000 txns/second, and we instantly got a ~30% performance boost. We also got much more consistent performance for our slower queries thanks to Aurora's fast, custom storage engine.
    - riku_iki 8y ago
      > The main DB client is a web app with ~4000 txns/second curious what is your bill for this
      - jdreaver 8y ago
        For Aurora we are on a db.r4.8xlarge, which is ~$3340 per month for the instance alone. Aurora also charges for IOPS against storage, and we pay about $350 per month for those IOPS. Our DB is ~400 GB, so ~$40 per month for that as well. I am pretty sure we could live comfortably on a db.r4.4xlarge, but we have some bursty analytics applications that sometimes bring our load up for about a minute, and we like the peace of mind of headroom during peak hours (we are in edtech with super predictable traffic throughout US school hours). Total side note: I just went to the pricing page and it looks like they just released db.r5 instances for Aurora (https://aws.amazon.com/rds/aurora/pricing/ https://aws.amazon.com/rds/aurora/pricing/)! It might be time to try those out. I don't see an announcement just yet though...

deleted 8y ago

[deleted]

zenlot 8y ago

For some reason those blogs become boring. Whoever moves to Kubernetes, or chooses to use some service of major cloud provider feels a need to write a blog about it having a same theme as everyone else. Bonus points if it explains what containers are and differences between EKS, ECS and others.

techslave 8y ago
well you are missing the point. the tech details are irrelevant in posts like this. no one, but no one, is throwing down knowledge and insight that anyone and their mama doesn’t know or can’t easily acquire. people just don’t give away their secret sauce that easily. the 2 points are a. a recruiting signal. it lets candidates know “we are like you”. we care about the tech stack and what we do. we care about elegant and “good” solutions. b. it gives company devs a public sounding board. a chance to have a bigger voice than in house obscurity. if you think these blogs are really about the tech, well then yes they are quite boring.
- nitinreddy88 8y ago
  I guess it's becoming a trend "we moved from here to here" and let's make it to top of HN. Unless current provider has horrible support/issues which you want to specify to help others, these articles are useless. HN community making these articles to front page which neither describes architect challenges/scalability challenges is pointless
  - staticassertion 8y ago
    I found the section on their evaluation of other potential systems like ECS to be interesting - it's something I'm currently considering myself.
    - ukd1 8y ago
      glad it was useful!

mcguireio 8y ago

Now if only they could quit spamming every inbox of every company I've worked at, id be impressed. Their sales automation is out of control. Honestly...I write them off before ever looking at their services.

markbnj 8y ago

We've been running in production on GKE for a little over two years and it's been a solid platform since day one. It's nice to read articles like this and see others coming to the same conclusions we've come to. If your practices and workflow are oriented to containers and you outgrow your PAAS then k8s is the logical place to land. With respect to the choice of helm: we started out rolling our own pipeline with sed and awk and the usual suspects. When that became too complex helm was just taking off and we moved to that. We still use it to install various charts from stable that implement infrastructure things. For our own applications we found that there was just too much cognitive dissonance between the helm charts and the resulting resources. Essentially the charts and the values we plugged into them became a second API to kubernetes, obfuscating the actual API below. The conventions around the "package manager" role that the tool has taken for itself also contribute to lessen readability due to scads of boilerplate and name mangling. We recently started deploying things in a new pipeline based on kustomize. We keep base yaml resources in the application repo and apply patches from a config repo to finalize them for a given environment. So far it's working out quite well and the applications engineers like it much better. Now with kubectl 1.14 kustomize's features have been pulled in to that tool, something I have mixed feelings about, but at least the more declarative approach does seem to be the way the wind is blowing.

zapita 8y ago
Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14?
- markbnj 8y ago
  > Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14? It's nothing too surprising :). Simply a preference for simple, compose-able tools. I personally feel like patching yaml client-side is outside the scope of a "ctl" tool.
physicles 8y ago
Perhaps I don’t understand the design decisions behind Helm, but it’s always struck me as having a severe impedance mismatch with k8s itself. It defines another entirely different schema, and relies on an agent running in your cluster that’s also trying to reconcile desired state with actual state (which k8s itself is also doing). I’m skeptical that you could use it extensively without also understanding the k8s stuff underneath it. kustomize came along just as it’s become untenable for us to copy/paste config to multiple environments. I like that it’s pretty much the simplest possible way to customize yaml, and plan to dive in soon.

autotune 8y ago

Basically moved to my dream tech stack, jealous of the SRE’s over there.

maciejgryka 8y ago
We are hiring :) https://rainforestqa.com/careers https://rainforestqa.com/careers
- autotune 8y ago
  Ha, would love to as a remote candidate out of NYC, don't see anything for SRE though.

shay_ker 8y ago

> At one point we attempted to migrate to Heroku Shield to address some of these issues, but we found that it wasn’t a good fit for our application. This part seems very hand wavy, given that Heroku Shield would've solved many (all?) of their problems. > We were also running into limitations with Heroku on the compute side: some of our newer automation-based features involve running a large number of short-lived batch jobs, which doesn’t work well on Heroku (due to the relatively high cost of computing resources). How much memory did their batch jobs actually need? If they're using Rails, then I'm assuming they're just running a bunch of Sidekiq jobs that are querying PG. I'm surprised that they'd need that much in terms of compute resources. They should be able to get very, very far by making PG do a lot of the work, or by streaming data from PG and not holding a lot of data in memory. Even if they did need all this, the following two options seem WAY easier to manage: 1) Use dokku to run your super-intense Sidekiq batch jobs on beefy EC2 instances. You can still schedule them in your Rails app in Heroku, no big deal. Many engineering teams have to do this type of split-up anyway when it comes to Application Engineers and Data Engineers, this is just a simpler way to do it. 2) Similar to 1), use a different language runtime for the batch jobs. If you really need to run CPU intensive jobs, why are you using Ruby? If the jobs aren't so intense to mandate maintaining two languages (fwiw, not that hard), why will moving to k8s solve the issue? Personally, I'm not sold on their decision to move to Kubernetes, and I use Kubernetes for my job.

shosti 8y ago
> This part seems very hand wavy, given that Heroku Shield would've solved many (all?) of their problems. Author here; I don’t want to go into too much detail, but we tried Shield early on and had a negative experience that made us wary about using the platform (it seems to use a different tech stack under the hood from “normal” Heroku and lacks a lot of the things that make Heroku great). Also it’s very expensive compared to VPC-based solutions on AWS and GCP. W.R.T. the batch jobs, I think I didn’t explain super well—we are using a different language and runtime from our “normal” background processing jobs (which use worker queues in Rails), it’s just that Heroku isn’t very well suited for the use case (which is basically FaaS-like but with long-lived jobs). The “split” workflow you described is basically what we were doing (but with AWS Batch instead of Dokku); it’s just that it’s more cost-efficient to consolidate everything into one cluster (especially with preemptible gke nodes) and also better to have a common set of tooling for the Ops team. To be fair, we haven’t yet completed the move from Batch to k8s so it’s possible that part of the plan won’t pan out as expected.

_0nac 8y ago

If you can't wait for their teaser of "In a future post, I’ll cover the migration process itself", the GCP site has a hands-on tutorial of migrating an app which may prove interesting: https://cloud.google.com/solutions/migrating-ruby-on-rails-apps-on-heroku-to-gke https://cloud.google.com/solutions/migrating-ruby-on-rails-a... Disclaimer: I work for GCP and wrote most of that :D At the end of the day, Heroku and GKE are rather different beasts with different philosophies, so migrations are never going to be 1:1. I expect this to become simpler over time as tooling matures though, eg. using https://buildpacks.io https://buildpacks.io to build Docker images instead of having to craft them by hand seems promising.

ZeroCool2u 8y ago
I've been playing around with GCP, mostly app engine, the past couple weeks making a toy application to get a little more comfortable with building and designing serverless apps. Just wanted to say you, and whoever else, is writing those docs does an amazing job. Seriously, I've tried all 3 platforms, because of work and GCP is by far the easiest to become productive with and the main reason is the glorious documentation. Keep up the great work!
- devlance 7y ago
  Great to hear about your experience with the docs and appreciate the call out. You can use the "Send Feedback" button on any of the doc pages to let us know about any feedback you have. Real people read it.
  - ZeroCool2u 7y ago
    I'll keep that in mind!

NightlyDev 8y ago

I've always wondered: Is kubernetes hard to host on your own on a couple of servers for production? I've never tried, but I've heard a lot of people saying it's very hard, but people are often complaining about the most basic stuff as being hard.. Sooo..?

bauerd 8y ago
Depends on what you mean by hard. Bootstrapping a K8s control plane? Not so much, kubeadm does all the heavylifting for you. Keeping your install recent, etcd performant and backed up, maintaining and scaling underlying filesystems etc. is still a full-time job for production usage imho. Essentially you'll find yourself operating a number of distributed systems below your application stack.
physicles 8y ago
More important than whether it’s easy or hard, is the fact that it’s unnecessary. GKE is great and you only pay for your compute (as opposed to EKS, which is over $140/mo for the control plane, a move which always struck me as a terrible business decision). I maintain two prod and two test environments: AWS+kops (since before EKS existed), Alibaba+kubeadm (because China), and two local ones that used to be minikube but are now just kubeadm. I spent a total of about two months getting the knowledge to do that, and these days I spend about 4 hours a week on maintenance. I do minor version upgrades but put off major ones because there’s not much business value and upgrades can break stuff. It’s my least favorite part of my job. I’d rather be coding or doing architecture, which is what I spend most of my time doing. We’ll be moving the AWS stuff to GKE in a couple months. Still haven’t heard much on the quality of Alibaba’s offering. Their IaaS is solid but some of their other services will sometimes throw errors that return 0 hits on google, no documentation, which scares me.
flurdy 8y ago
You would have to follow this the whole way: https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
- 8y ago

rdsubhas 8y ago

I work at a large European startup. We are also heavily invested in and love kubernetes, but this is a totally apples to oranges comparison. Heroku is a PaaS and Kubernetes is a CaaS. K8s is great at what does, but to make it act like Heroku is a huge amount of effort and needs a team to manage the tooling around it. Assuming that such team is not needed, and k8s can simply be used like heroku with a bunch of extra CLI tools, usually leads to "Wild Wild West" clusters.

auslander 8y ago

> ..agile without hiring a large Ops team, .. we were beginning to outgrow Heroku. We ended up .. running on Google Kubernetes Engine Or you could hire few cloud infra devs and stick to autoscaled VMs (EC2), not spending time on k8s? I bet k8s took you a while to set up and maintain.