3 ms·

I moved us from Heroku to GKE a few years ago when we had ~4 engineers - Took a bit to get things figured out and get CI deploys etc working well, but it really

I moved us from Heroku to GKE a few years ago when we had ~4 engineers - Took a bit to get things figured out and get CI deploys etc working well, but it really wasn't all that complicated. Honestly baffled at how many people are convinced you need hundreds of engineers before k8s can work well - they must be doing something very different from what we are. We did keep databases out of k8s (until recently when we added CockroachDB running inside the cluster) and only had ~6 services to move which may have kept things simpler. We're now scaling to run thousands of vcpu at peak times and a dozen different services, and it's still not all that hard to manage. Can somebody who has had the opposite experience comment on what actually made it so difficult to implement? I imagine that if you are managing the cluster control plane yourself that will make things much more difficult - but unless you have some very specific requirement you can use a hosted k8s to reduce that.

chillfox 6y ago
Not everyone can run in the cloud, and of the ones who can there are a number of organizations who can't put their data in clouds that have to comply with US court orders. Now if you are in Europe then Scaleway got a K8s offering not that long ago, but the rest of the world have to run that stuff local and running K8s securely on a locked down corporate network can be very complicated.
- jordanthoms 6y ago
  Sure, but the advice people generally give is that small teams and startups shouldn't use k8s - and those are the organizations who can most likely use one of the hosted k8s implementations (which are available anywhere AWS/GCP/Azure have datacenters). Larger corporates will have more complexity to work in with but also will have larger teams which can handle that.
mbyio 6y ago
If you use GKE, you have been spoiled. Other Kubernetes platforms - in particular EKS, which is what most people are going to use since most people use AWS - are way more work to setup and maintain. (Emphasis on the "maintain".) Also, I think a lot of companies have really terrible devops practices that don't work in Kubernetes. So their move to Kubernetes includes a lot of extra work that isn't really caused by the move to Kubernetes.
- jordanthoms 6y ago
  Yeah, that's probably a major factor here - we moved to google cloud specifically for GKE and it's been pretty great. Have had a couple of issues (mostly with kube-dns autoscaling not keeping up, node-local dns helped a lot), but ultimately it's saved far more work than it's created.
subsection1h 6y ago
Roughly how many person hours were needed to implement review apps on GKE? Do your GKE review apps auto-hibernate like Heroku review apps do when using AutoIdle (https://autoidle.com https://autoidle.com)?
- jordanthoms 6y ago
  We don't have review apps yet, we just have a few testing environments which devs can push changes they want to have available to. It is a really nice feature of heroku though so something it'd be nice to replicate. I think it'd probably take a week or so of dev time to do, the hardest part would be automating spinning up copies of all the dependencies we need.

chillfox 6y ago

Not everyone can run in the cloud, and of the ones who can there are a number of organizations who can't put their data in clouds that have to comply with US court orders. Now if you are in Europe then Scaleway got a K8s offering not that long ago, but the rest of the world have to run that stuff local and running K8s securely on a locked down corporate network can be very complicated.

jordanthoms 6y ago
Sure, but the advice people generally give is that small teams and startups shouldn't use k8s - and those are the organizations who can most likely use one of the hosted k8s implementations (which are available anywhere AWS/GCP/Azure have datacenters). Larger corporates will have more complexity to work in with but also will have larger teams which can handle that.

mbyio 6y ago

If you use GKE, you have been spoiled. Other Kubernetes platforms - in particular EKS, which is what most people are going to use since most people use AWS - are way more work to setup and maintain. (Emphasis on the "maintain".) Also, I think a lot of companies have really terrible devops practices that don't work in Kubernetes. So their move to Kubernetes includes a lot of extra work that isn't really caused by the move to Kubernetes.

jordanthoms 6y ago
Yeah, that's probably a major factor here - we moved to google cloud specifically for GKE and it's been pretty great. Have had a couple of issues (mostly with kube-dns autoscaling not keeping up, node-local dns helped a lot), but ultimately it's saved far more work than it's created.

subsection1h 6y ago

Roughly how many person hours were needed to implement review apps on GKE? Do your GKE review apps auto-hibernate like Heroku review apps do when using AutoIdle (https://autoidle.com https://autoidle.com)?

jordanthoms 6y ago
We don't have review apps yet, we just have a few testing environments which devs can push changes they want to have available to. It is a really nice feature of heroku though so something it'd be nice to replicate. I think it'd probably take a week or so of dev time to do, the hardest part would be automating spinning up copies of all the dependencies we need.