3 ms·

I can understand the use of kubernetes in very large orgs to manage clusters of hundreds of nodes, but it seems to me the complexity isn’t justified if you only

I can understand the use of kubernetes in very large orgs to manage clusters of hundreds of nodes, but it seems to me the complexity isn’t justified if you only have 1-100 say. There are lots of possibilities between 1 server and 100, and lots of ways to have simple replicable deploys if your needs are simple (probably 95% of businesses). Simple load balancers without auto-scaling work fine! For smaller non-critical services sometimes even one reliable server is an acceptable trade-off. Do most companies really need kubernetes given the complexity and difficulty it introduces?

te_chris 6y ago
No, but it’s not just complexity. We’re a small company, but after spending about a month to setup our GKE a couple of years ago (incl learning terraform in the process), it’s been rock solid, low involvement and reliable. Declarative resource specification is a legitimate game changer and you couldn’t pay me to go back.
- warent 6y ago
  I'm not sure why you're getting downvoted unless people are afraid that you're sponsored by Google. Your experience aligns with mine. Running a SaaS app by myself that serves heavy loads around 1 rps with burst of 400+ rps, k8s works really great for my use case. But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments. Plus, often times it's really unclear if I should scale vertically or horizontally.
  - gizzlon 6y ago
    "k8s works really great for my use case. [..] But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments." TBO, it does not sound like it's working great for your use-case =) Wouldn't something like Cloud Run, AppEgine Standard or Heroku be much simpler and cheaper? Or just a single $5-15 per month* VM? * depending on how much oomph you need for those 400 rps
    - warent 6y ago
      Sadly no because I need stateful RAM. A single VM might work but I'm scared
- grey-area 6y ago
  Standardised declarative config/setup I've found really useful, declarative resource specification (as in n servers for service y) I don't really have a use for though I could see how in larger companies with hundreds of servers and lots of employees it would be attractive. If it is rock solid and reliable and impossible to mess up, I could see it being really nice (as your experiences sounds), though it feels to me like end users of services like GKE should not have to know about kubernetes or its complexity, nor have a chance to mess it up by editing configs directly. I guess the ultimate goal of services like GKE autopilot is for you not to have to worry about kubernetes at all, just give them your workload and have it run on whatever resources they think is appropriate? I do think it's important to recognise though that there are lots of ways to host services, and simpler with less abstractions is often better and more reliable and certainly easier to debug when things go wrong.
  - te_chris 6y ago
    I mean yeah, I’m not arguing against autopilot and would probably have used it if it existed when we kicked off. Cloud Run had just come out and I really wanted to use that but the networking wouldn’t do what we needed, so had to go all in. But in the end, we learned enough Kube, we specified it all with Terraform, we haven’t tried to do too many fancy or complicated things, and we have reliable infrastructure fed with a simple deployment/build pipeline (just using cloud build, it’s clunky but it works). I mean maybe I’m conflicted as we’re part of GCS, but the system works well for us. If it didn’t we could’ve jumped to Azure or AWS and got credits there.
    - grey-area 6y ago
      Sounds like a nice reliable experience to me, thanks for sharing your experience on it. If I hear more reports like this I might just have to try out GKE :)
trumpeta 6y ago
Maybe I can offer an answer to your question, I have worked at a couple of companies where we ran "small" scale k8s clusters (1-100 nodes as you say). We have chosen k8s and I would again, because its nice to use. Its not necessarily easier, as you point out, the complexity of managing the cluster is considerable. But if you use a managed cluster like EKS or DO's k8s offering, you don't have to worry too much about the nodes and the unit of worry is the k8s config and then for deployment you can use Docker. I like Docker, because its nice. Its nice to have the same setup locally as you have remotely. In my experience the tooling around k8s is nice to manage declaratively, I never liked working with machines directly because even tools like Chef or Ansible feel very flimsy. The other thing you can do is run on ECS or similar, but there the flexibility is a lot lower. So k8s for me offers the sweet spot of being able to do a lot quickly with a nice declarative interface. I'd be interested to hear your take on how to best run a small cluster though.
- grey-area 6y ago
  Thanks, that's really interesting. Everyone has different challenges and requirements, and of course different experiences. For smaller setups (say 1-10 services) I'm quite happy with cloud config and one VM per process behind one load balancer per service. It's simple to set up, scale and reproduce. This setup doesn't autoscale, but I've never really felt the need. We use Go and deploy one static binary per service at work with minimal dependencies so docker has never been very interesting. We could redeploy almost all the services we run within minutes if required with no data loss, so that bit feels similar to K8s I imagine. For even smaller companies (many services at many companies) a single reliable server per service is often fine - it depends of course on things like uptime requirements for that service but not everything is of critical importance and sometimes uptime can be higher with a single untouched service. I think what I'd worry about with a k8s config which affects live deployments is that I could make a tweak which seemed reasonable in isolation but broke things in inscrutable ways - many outages at big companies seem to be related to config changes nowadays. With a simpler setup there is less chance of bringing everything down with a config change, because things are relatively static after deploy.
  - 6y ago

te_chris 6y ago

No, but it’s not just complexity. We’re a small company, but after spending about a month to setup our GKE a couple of years ago (incl learning terraform in the process), it’s been rock solid, low involvement and reliable. Declarative resource specification is a legitimate game changer and you couldn’t pay me to go back.

warent 6y ago
I'm not sure why you're getting downvoted unless people are afraid that you're sponsored by Google. Your experience aligns with mine. Running a SaaS app by myself that serves heavy loads around 1 rps with burst of 400+ rps, k8s works really great for my use case. But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments. Plus, often times it's really unclear if I should scale vertically or horizontally.
- gizzlon 6y ago
  "k8s works really great for my use case. [..] But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments." TBO, it does not sound like it's working great for your use-case =) Wouldn't something like Cloud Run, AppEgine Standard or Heroku be much simpler and cheaper? Or just a single $5-15 per month* VM? * depending on how much oomph you need for those 400 rps
  - warent 6y ago
    Sadly no because I need stateful RAM. A single VM might work but I'm scared
grey-area 6y ago
Standardised declarative config/setup I've found really useful, declarative resource specification (as in n servers for service y) I don't really have a use for though I could see how in larger companies with hundreds of servers and lots of employees it would be attractive. If it is rock solid and reliable and impossible to mess up, I could see it being really nice (as your experiences sounds), though it feels to me like end users of services like GKE should not have to know about kubernetes or its complexity, nor have a chance to mess it up by editing configs directly. I guess the ultimate goal of services like GKE autopilot is for you not to have to worry about kubernetes at all, just give them your workload and have it run on whatever resources they think is appropriate? I do think it's important to recognise though that there are lots of ways to host services, and simpler with less abstractions is often better and more reliable and certainly easier to debug when things go wrong.
- te_chris 6y ago
  I mean yeah, I’m not arguing against autopilot and would probably have used it if it existed when we kicked off. Cloud Run had just come out and I really wanted to use that but the networking wouldn’t do what we needed, so had to go all in. But in the end, we learned enough Kube, we specified it all with Terraform, we haven’t tried to do too many fancy or complicated things, and we have reliable infrastructure fed with a simple deployment/build pipeline (just using cloud build, it’s clunky but it works). I mean maybe I’m conflicted as we’re part of GCS, but the system works well for us. If it didn’t we could’ve jumped to Azure or AWS and got credits there.
  - grey-area 6y ago
    Sounds like a nice reliable experience to me, thanks for sharing your experience on it. If I hear more reports like this I might just have to try out GKE :)

trumpeta 6y ago

Maybe I can offer an answer to your question, I have worked at a couple of companies where we ran "small" scale k8s clusters (1-100 nodes as you say). We have chosen k8s and I would again, because its nice to use. Its not necessarily easier, as you point out, the complexity of managing the cluster is considerable. But if you use a managed cluster like EKS or DO's k8s offering, you don't have to worry too much about the nodes and the unit of worry is the k8s config and then for deployment you can use Docker. I like Docker, because its nice. Its nice to have the same setup locally as you have remotely. In my experience the tooling around k8s is nice to manage declaratively, I never liked working with machines directly because even tools like Chef or Ansible feel very flimsy. The other thing you can do is run on ECS or similar, but there the flexibility is a lot lower. So k8s for me offers the sweet spot of being able to do a lot quickly with a nice declarative interface. I'd be interested to hear your take on how to best run a small cluster though.

grey-area 6y ago
Thanks, that's really interesting. Everyone has different challenges and requirements, and of course different experiences. For smaller setups (say 1-10 services) I'm quite happy with cloud config and one VM per process behind one load balancer per service. It's simple to set up, scale and reproduce. This setup doesn't autoscale, but I've never really felt the need. We use Go and deploy one static binary per service at work with minimal dependencies so docker has never been very interesting. We could redeploy almost all the services we run within minutes if required with no data loss, so that bit feels similar to K8s I imagine. For even smaller companies (many services at many companies) a single reliable server per service is often fine - it depends of course on things like uptime requirements for that service but not everything is of critical importance and sometimes uptime can be higher with a single untouched service. I think what I'd worry about with a k8s config which affects live deployments is that I could make a tweak which seemed reasonable in isolation but broke things in inscrutable ways - many outages at big companies seem to be related to config changes nowadays. With a simpler setup there is less chance of bringing everything down with a config change, because things are relatively static after deploy.
- 6y ago