4 ms·

If you think Kubernetes is unnecessary overhead, you've never operated at significant scale.

gtirloni 4y ago
Exactly. And probably not at significant complexity either.
throwaway0129u 4y ago
Not at all, k8s is not designed for very large scale. Unsurprisingly, FAANGS don't use it to manage their own platforms. Edit: Google's Borg is a very different beast. Edit: no need to patronize me. I worked on massive scale deployments otherwise I would not be commenting.
- halfstar91 4y ago
  Many large organisations use Kubernetes (Google, Spotify, IBM, etc). Regardless, large scale and very large scale are different. Kubernetes is well suited for controlling fleets of compute resource in the order of 10,000s CPU cores, and terabytes of memory. The compute overhead to orchestrate these clusters is well worth the simplicity/standardisation/auto-scaling that comes with Kubernetes. Many people have never had to operate VMs in the hundreds or thousands and do not understand the challenges that come with maintaining varied workloads, alongside capacity planning at that scale.
- chr15p 4y ago
  a million nodes running a single application is scale, but a thousand nodes running a thousand applications is also scale, and they are very different beasts. The FAANGs operate the first kind, k8s is mostly aimed at the second kind scale, so its designed "for scale", for some definitions of scale.
- yuppie_scum 4y ago
  K8s spun off of Google’s Borg operator software specifically designed for high availability at FAANG scale. So essentially K8s is the “community edition.” Go read the Google SRE Book for context. We use it to serve ruby with 50 million requests per minute just fine. And the best part is the Horizontal Pod Autoscaler which saves our ass during seasonal spikes. While serverless/lambda are great I do think K8s is the most flexible way to serve rapidly changing containerized workload at scale.
aristofun 4y ago
Actually i have experience working un hi loaded env operated by k8s. But for 99% of projects i see, it’s a waste of time and resources (mostly people resources, not just cpu). HN is a perfect example of a project that doesn’t need it, no matter the traffic. If you need some additional flexibility and scalability over “bare metal” setup, you can go far with just docker compose or swarm until you have no choice but use k8s Again, if you know what you are doing.
- mdm_ 4y ago
  I worked in an academic library where the scale was on the order of hundreds of users per day, but many of those users were faculty/researchers spread out across the globe who got very grumpy when whatever online archive/exhibit was not available. I migrated our services from a very “pet” oriented architecture to a 4-node Proxmox cluster with Docker Swarm deploying containers across four VMs and it worked great. Services we brought up on this infra still to this date have 100% uptime, through updates and server reboots and other events that formerly would have taken sites offline temporarily. I looked at k8s briefly and it seemed like total overkill. I probably would have spent weeks or months learning k8s for no appreciable advantage over swarm.
petercooper 4y ago
Kubernetes is fantastic, though I think of it more as a tool for managing organizational complexity than ops. You can vertically scale beyond the level needed by most commercial applications nowadays using ad-hoc approaches on a machine or two with 16+ cores, 256GB+ RAM, terabytes of NVM drives, etc. but many (even small) companies have policies, teams, and organizational challenges that demand a lot of the structure and standardization tools like Kubernetes bring to the table. So I'm not disagreeing with your assertion, but I'd perhaps scope it to saying it's useful overhead at significant organizational scale, but you can certainly operate at a significant technical scale without such things.. and that can be quite fun if you have the team or app for it :)
Denvercoder9 4y ago
Kubernetes is great at significant scale, that's what it's designed for. It has significant overhead if you don't need that scale.
Sharlin 4y ago
And how many k8s believers ever reach significant scale? It's just like back when NoSQL was the trendy thing and people thought a couple gigabytes meant "big data". Mostly it’s simply cargo culting.
- aristofun 4y ago
  Brilliant rephrasing of my point.

Exactly. And probably not at significant complexity either.

Not at all, k8s is not designed for very large scale. Unsurprisingly, FAANGS don't use it to manage their own platforms. Edit: Google's Borg is a very different beast. Edit: no need to patronize me. I worked on massive scale deployments otherwise I would not be commenting.

halfstar91 4y ago
Many large organisations use Kubernetes (Google, Spotify, IBM, etc). Regardless, large scale and very large scale are different. Kubernetes is well suited for controlling fleets of compute resource in the order of 10,000s CPU cores, and terabytes of memory. The compute overhead to orchestrate these clusters is well worth the simplicity/standardisation/auto-scaling that comes with Kubernetes. Many people have never had to operate VMs in the hundreds or thousands and do not understand the challenges that come with maintaining varied workloads, alongside capacity planning at that scale.
chr15p 4y ago
a million nodes running a single application is scale, but a thousand nodes running a thousand applications is also scale, and they are very different beasts. The FAANGs operate the first kind, k8s is mostly aimed at the second kind scale, so its designed "for scale", for some definitions of scale.
yuppie_scum 4y ago
K8s spun off of Google’s Borg operator software specifically designed for high availability at FAANG scale. So essentially K8s is the “community edition.” Go read the Google SRE Book for context. We use it to serve ruby with 50 million requests per minute just fine. And the best part is the Horizontal Pod Autoscaler which saves our ass during seasonal spikes. While serverless/lambda are great I do think K8s is the most flexible way to serve rapidly changing containerized workload at scale.

aristofun 4y ago

Actually i have experience working un hi loaded env operated by k8s. But for 99% of projects i see, it’s a waste of time and resources (mostly people resources, not just cpu). HN is a perfect example of a project that doesn’t need it, no matter the traffic. If you need some additional flexibility and scalability over “bare metal” setup, you can go far with just docker compose or swarm until you have no choice but use k8s Again, if you know what you are doing.

mdm_ 4y ago
I worked in an academic library where the scale was on the order of hundreds of users per day, but many of those users were faculty/researchers spread out across the globe who got very grumpy when whatever online archive/exhibit was not available. I migrated our services from a very “pet” oriented architecture to a 4-node Proxmox cluster with Docker Swarm deploying containers across four VMs and it worked great. Services we brought up on this infra still to this date have 100% uptime, through updates and server reboots and other events that formerly would have taken sites offline temporarily. I looked at k8s briefly and it seemed like total overkill. I probably would have spent weeks or months learning k8s for no appreciable advantage over swarm.

petercooper 4y ago

Kubernetes is fantastic, though I think of it more as a tool for managing organizational complexity than ops. You can vertically scale beyond the level needed by most commercial applications nowadays using ad-hoc approaches on a machine or two with 16+ cores, 256GB+ RAM, terabytes of NVM drives, etc. but many (even small) companies have policies, teams, and organizational challenges that demand a lot of the structure and standardization tools like Kubernetes bring to the table. So I'm not disagreeing with your assertion, but I'd perhaps scope it to saying it's useful overhead at significant organizational scale, but you can certainly operate at a significant technical scale without such things.. and that can be quite fun if you have the team or app for it :)

Denvercoder9 4y ago

Kubernetes is great at significant scale, that's what it's designed for. It has significant overhead if you don't need that scale.

Sharlin 4y ago

And how many k8s believers ever reach significant scale? It's just like back when NoSQL was the trendy thing and people thought a couple gigabytes meant "big data". Mostly it’s simply cargo culting.

aristofun 4y ago
Brilliant rephrasing of my point.