9 ms·

K8s with 1M nodes

284 points by denysvitali 1y ago

ktpsns 1y ago
Typical large scale high performance computing clusters are at a size of 10k nodes (for instance Jupiter and SuperMUC in Germany) [1]. These centers are quite remarkably big buildings. I wonder how much 1M node single k8s clusters there are in the world right now. Most likely at the hyperscalers. [1] what is a node? Typically it is a synonym for "server". In some configurations HPC schedulers allow node sharing. Then we talk about order of 100k cores to be scheduled.
- stackskipton 1y ago
  I doubt any Hyperscalers are running 1M Node clusters either. They probably just have groups of clusters at each datacenter and some overall scheduler that determines which cluster is best suited for workload during deployment then connects to that cluster and schedules the workload.
  - deleted 1y ago
    [deleted]
  - merb 1y ago
    Some hyperscalers even have services for that. Which even makes it possible to have cross cluster ingress. And other things. And it makes it possible to have multiple cluster ingress different regions that somewhat work together.
- osigurdson 1y ago
  >> [1] what is a node? Typically it is a synonym for "server". In some configurations HPC schedulers allow node sharing I'm sure they mean actual servers / not just cores. Even in traditional HPC it isn't abstracted to the level of individual cores usually since most HPC jobs care about memory bandwidth - even with Infiniband or other techniques throughput / latency is much worse than on a single machine. Of course, multiple machines are connected (usually using MPI / Infiniband) but important to try to minimize communication between nodes where possible. For AI workloads, they are running GPUs - so 10K+ cores on a single device so even less likely to be talking about cores here.
- femiagbabiaka 1y ago
  This post is just about a reference architecture, but last I knew OpenAI ran VMs on Azure. https://openai.com/index/scaling-kubernetes-to-7500-nodes/ https://openai.com/index/scaling-kubernetes-to-7500-nodes/
up2isomorphism 1y ago
“Perhaps my spiciest take from this entire project: most clusters don’t actually need the level of reliability and durability that etcd provides.” This assumption is completely out of touch, and is especially funny when the goal is to build an extra large cluster.
- itsnowandnever 1y ago
  etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.
  - jauntywundrkind 1y ago
    The API server is the thing. It so happens that the API server can mostly be a thin shell over etcd. But etcd itself while so common is not sacrosanct. https://github.com/k3s-io/kine https://github.com/k3s-io/kine is a reasonably adequate substitute for etcd. sqlite, MySQL, PostgreSQL can also be substituted in. Etcd is from the ground up built to be more scale-out reliable, and that rocks to have baked in. But given how easy it is to substitute etcd out, I feel like we are at least a little off if we're trying to say "etcd is also the entire point of k8s" (the APIserver is)
    - itsnowandnever 1y ago
      that's fair but that 99% of all apiserver deployments in the world have the same standard boilerplate footprint is a large part of why it became so ubiquitous. that people running it locally don't have to make any decisions about how to deploy which database or why to use this one over that one... and that's also the same situation in production so people doing stuff in dev aren't punched in the face by an exponentially more complex system in production is huge.
    - dmlittle 1y ago
      It's been a while since I've checked this but a few years ago we tried to limit test kine on a large-ish cluster and it performed pretty poorly. It's fine for small clusters but the way they have to implement the watch semantics makes it perform poorly (at least this was the case a few years ago).
rixed 1y ago
I don't get the point of benchmarking k8s without the guarantees of etcd. At some point, you are just competing with clusterssh.
- cyberax 1y ago
  How often do you have sudden host failures? Especially if you use a half-decent server with redundant components for the DB node? Once in maybe 10 years?
  - dmlittle 1y ago
    The node failure rate is much higher than that. On a 1M node cluster of cloud-managed instances (AWS, GCP, Azure, etc.) you'd likely see failures a few times a month, if not more.
    - cyberax 1y ago
      Yep. And the chances that the DB node with the control plane fails are therefore less than one in ten thousand.
  - rixed 1y ago
    It's not about failure, it's about servicing the host running etcd or the network around it.
wppick 1y ago
If you don't need the isolation of of k8s then don't forget about erlang, which is another option to scale up to 1 million functions. Obviously k8s containers (which are fundamentally just isolated processes) and erlang processes are not interchangeable things, but when thinking about needing in the order of millions of processes erlang is pretty good prior art
- theptip 1y ago
  This is 1m nodes, you typically run tens or hundreds of pods per node, each with one or more containers. So more like 100m+ functions if I follow the Erlang analogy correctly?
  - reactordev 1y ago
    This is not analogous. It’s just someone beating the Erlang drum. You can’t PyTorch in Erlang.
    - wppick 1y ago
      Kubernetes is way heavier than Erlang’s lightweight processes, so for millions of tasks at scale, a middle-ground solution could blend Erlang’s concurrency efficiency with k8s’ orchestration power, dodging containers’ overhead while keeping flexibility for diverse workloads. That's if you don't actually need the strict isolation of pods/containers and you're just trying to run something at massive scale. I don't get why so many people want to run everything as heavy container processes or pods vs coming up with a better solution. The point is we don't have to fit every problem into the shoe called kubernetes if it doesn't seem to fit, and we should look at other ways to spin up millions of processes
    - victorbjorklund 1y ago
      There are similar libraies in Elixir. Is the ecosystem for ML as developed as for python? Nope, but not every ML project needs the most obscure libraries etc. (For the record I don't really see Erlang clusters as a replacement for k8s)
jeffinhat 1y ago
This is an awesome experiment and write up. I really appreciate the reproducibility. I would like to see how moving to database that scales write throughput with replicas would behave, namely FoundationDB. I think this will require more than an intermediary like kine to be efficient, as the author illustrates the apisever does a fair bit of its own watching and keeping state. I also think there's benefit, at least for blast radius, to shard the server by api group or namespace. I think years ago this would have been a non starter with the community, but given AWS has replaced etcd (or at least aspects) with their internal log service for their large cluster offering, I bet there's some appetite for making this interchangable and bringing and open source solution to market. I share the authors viewpoint that for modern cloud based deployments, you're probably best avoiding it and relying on VMs being stable and recoverable. I think reliability does matter if you want to actually realize the "borg" value and run it on bare metal across a serious fleet. I haven't found the business justification to work on that though!
- ymelnyk 1y ago
  Here you go, Kine FoundationDB backend https://github.com/melgenek/f8n https://github.com/melgenek/f8n To be honest, I was building it with the purpose of matching the Etcd scale, but making foundationdb a multitenant data store. But with the recent craze of scalability , I'll be investing time into understanding how far foundationdb can be pushed as a K8s data store. Stay tuned.
  - jeffinhat 1y ago
    Awesome, I will! It would be great to see where the limits are with this approach. I think at some point, you need to go deeper into the apiserver for scale than an API compatible shim, but this is just conjecture and not real data.
halayli 1y ago
without publishing mem_etcd code, and without telling us what happens when one of the etcd/mem_etcd node dies to compare, this write up doesn't provide much information.
- Brybry 1y ago
  I believe the code is here: https://github.com/bchess/k8s-1m/tree/main/mem_etcd https://github.com/bchess/k8s-1m/tree/main/mem_etcd https://github.com/bchess/k8s-1m/blob/main/RUNNING.adoc#mem_etcd https://github.com/bchess/k8s-1m/blob/main/RUNNING.adoc#mem_...
__turbobrew__ 1y ago
I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a way to get around the scalability issues, but unfortunately for the rest of us who do not have spanner there aren’t any great options. I feel like at a fundamental level that pod affinity, antiaffinity, and topology spreads are not compatible with very large clusters due to the complexity explosion in large clusters. Another thing to consider is that the larger a cluster becomes, the larger the blast radius is. I have had clusters of 10k nodes spectacularly fail due to code bugs within k8s. Sharding total compute capacity compute capacity into multiple isolated k8s clusters reduces the likelihood that a software bug is going to take down everything as you can carefully upgrade only a single cell at a time with bake periods between each cell.
- femiagbabiaka 1y ago
  Yep, every cluster approaching 10k I know of has either pared back etcd's durability guarantees or rewritten and replaced it in some manner. Actually the post goes into detail about doing this exactly, the Alibaba paper they reference says about the same. > Sharding total compute capacity compute capacity into multiple isolated k8s clusters reduces the likelihood that a software bug is going to take down everything as you can carefully upgrade only a single cell at a time with bake periods between each cell. Yeah, I've been meaning to try out something like Armada to simplify things on the cluster-user side. Cluster-providers have lots of tools to make managing multiple clusters easier but if it means having to rewrite every batch job..
- jen20 1y ago
  AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently. [1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazon-eks-ultra-scale-clusters/ https://aws.amazon.com/blogs/containers/under-the-hood-amazo... [2]: https://cloud.google.com/blog/products/containers-kubernetes/gke-65k-nodes-and-counting https://cloud.google.com/blog/products/containers-kubernetes... [3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-to-the-latest-azure-cosmos-db-announcements/#:~:text=Powering%20Kubernetes%20with%20etcd%20API,complexity%20of%20managing%20etcd%20clusters. https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...
ai-christianson 1y ago
Love this. There's no reason k8s shouldn't scale much further.
wb14123 1y ago
Instead of giving up the good guarantee of etcd, a better approach maybe grouping some nodes together to create a tree like structure with sub clusters.
- AlphaSite 1y ago
  That was the whole concept behind KCP iirc. It was designed to provide tenancy atop 1 or more clusters.
skeptrune 1y ago
I read this as napkin math[1] for Kube and thoroughly enjoyed. You can only find the important numbers relative to performance and scaling by trying to accomplish some kind of goal. Benchmarks are mostly bikeshedding. [1]: https://sirupsen.com/napkin https://sirupsen.com/napkin
vebgen 1y ago
This is an absolutely incredible technical deep-dive. The section on replacing etcd with mem_etcd resonates with challenges we've been tackling at a much smaller scale building an AI agent system. A few thoughts: *On watch streams and caching*: Your observation about the B-Tree vs hashmap cache tradeoff is fascinating. We hit similar contention issues with our agent's context manager - switched from a simple dict to a more complex indexed structure for faster "list all relevant context" queries, but update performance suffered. The lesson about O(1) writes vs O(log n) reads being the wrong tradeoff for high-write workloads is universal. *On optimistic concurrency for scheduling*: The scatter-gather scheduler design is elegant. We use a similar pattern for our dual-agent system (TARS planner + CASE executor) where both agents operate semi-independently but need coordination. Your point about "presuming no conflicts, but handling them when they occur" is exactly what we learned - pessimistic locking kills throughput far worse than occasional retries. *The spicy take on durability*: "Most clusters don't need etcd's reliability" is provocative but I suspect correct for many use cases. For our Django development agent, we keep execution history in SQLite with WAL mode (no fsync), betting that if the host crashes, we'd rather rebuild from Git than wait on every write. Similar philosophy. The mem_etcd implementation in Rust is particularly interesting - curious if you considered using FoundationDB's storage engine or something similar vs rolling your own? The per-prefix file approach is clever for reducing write amplification. Fantastic work - this kind of empirical systems research is exactly what the community needs more of. The "what are the REAL limits" approach vs "conventional wisdom says X" is refreshing.
rememberlenny 1y ago
People don’t realize how crucial Ben was in the forming of OpenAI as it is known today. This is an extremely underrated post.
kawsper 1y ago
That's really impressive and an interesting experiment. I was about to say that Nomad did something similar, but that was 2 million Docker containers across 6100 nodes, https://www.hashicorp.com/en/c2m https://www.hashicorp.com/en/c2m
throwdbaaway 1y ago
This looks impressive. As someone who is not familiar with ML, I do have a question -- surely in 2025 there must be a way to schedule a large pytorch job across multiple k8s clusters? EKS and GKE already provide VPC native flat network by default .
- sailingparrot 1y ago
  The issue isn’t so much scheduling as it is stability. More clusters means one more layer of things that can crash your (very expensive) training. You also then still need to write tooling to manage cross cluster trainings correctly just as starting/stopping roughly at the same time, resuming from checkpoints, node health monitoring etc. Nothing dealbreaking, but if it could just work in a single cluster that would be nicer.
czhu12 1y ago
If anyone is looking for a gentler, Heroku like onramp to Kubernetes, its exactly why I built Canine [1]. In retrospect, at my previous company, what we really needed in the early days was something that was Heroku-like (don't make me think about infra (!)) but could be easily added to and scaled up over time, as our service grew. We eventually grew to about 10M users, using the site monthly, and had to do a huge effort to migrate to Kubernetes. Canine's philosophy is: full Kubernetes, with a deployment layer on top. If you ever out grow it, just dump Canine entirely, and work directly with the Kubernetes system it's operating. It even gives you all the K8s YAML config needed to offboard. It's also similar to how the dev infra works at Airbnb (where I worked before that) -- Kubernetes underneath, a user friendly interface on top.
- rootlocus 1y ago
  You're not the first one to think that k9 is a good name for a kubernetes related technology https://k9scli.io https://k9scli.io
brimstedt 1y ago
It's an interesting and fun experiment, but what are real usecases for such a cluster?
- dbattaglia 1y ago
  At my last employer Elastic we definitely ran into these limits on the cloud SaaS team moving Elastocsearch node containers from our proprietary orchestration to k8s. I’m not sure how they eventually solved it but I believe the plan was essentially sharding ES clusters to different regional k8s clusters.
hardwaresofton 1y ago
Lasto ask: Does anyone have benchmark results for kine?
rvz 1y ago
> Many limitations are imposed by software. Or the amount of funding a startup has. The bottom line is, you are not OpenAI or Google.
pluc 1y ago
> Early on in this project, I asked ChatGPT “I want to scale Kubernetes to 1 million nodes. What types of problems would I need to overcome?” click
- breakingcups 1y ago
  It's a shame the author led with something that carries about the same authority as a horoscope, since the rest of the article is actually quite interesting.
  - tbrockman 1y ago
    If you have sufficient knowledge in the subject matter you're questioning ChatGPT about, you can fairly reliably discern complete bullshit from something plausibly true that warrants additional investigation (which I'd say is more useful than your typical horoscope). In isolation it seems worth the gamble to me, so long as you don't view it as much more than consulting the tea leaves.
    - pluc 1y ago
      You could also spend 5 minutes thinking instead of using a 10th of the Texas power grid to do it for you. Remember when we used to not know shit, and it would stay with you for days - before Google happened? The same is about to happen with mental acuity and AI. Use it or lose it - and I hope every fanboys' brain turns to mush before they witness irreversible AI-caused climate catastrophes solely to be able to speak natural english to a search engine. If you outsource your thinking process, don't come bitching when it's gone.
      - tbrockman 1y ago
        Yeah, I’m actually fairly sympathetic to your perspective, that’s why I said “in isolation”. I might think it’s possible to balance use in a way that’s not as detrimental to myself personally or others (self-hosted models, judicious use, etc.), but I definitely don’t disagree that it seems likely to do more harm than good currently.
adsharma 1y ago
I saw one fleeting reference to a CNI in the article. Anyone familiar with the space will tell you this is the biggest blocker in production. You will have to pay for an "enterprise" CNI to make it work.

284 points by denysvitali 1y ago

Typical large scale high performance computing clusters are at a size of 10k nodes (for instance Jupiter and SuperMUC in Germany) [1]. These centers are quite remarkably big buildings. I wonder how much 1M node single k8s clusters there are in the world right now. Most likely at the hyperscalers. [1] what is a node? Typically it is a synonym for "server". In some configurations HPC schedulers allow node sharing. Then we talk about order of 100k cores to be scheduled.

stackskipton 1y ago
I doubt any Hyperscalers are running 1M Node clusters either. They probably just have groups of clusters at each datacenter and some overall scheduler that determines which cluster is best suited for workload during deployment then connects to that cluster and schedules the workload.
- deleted 1y ago
  [deleted]
- merb 1y ago
  Some hyperscalers even have services for that. Which even makes it possible to have cross cluster ingress. And other things. And it makes it possible to have multiple cluster ingress different regions that somewhat work together.
osigurdson 1y ago
>> [1] what is a node? Typically it is a synonym for "server". In some configurations HPC schedulers allow node sharing I'm sure they mean actual servers / not just cores. Even in traditional HPC it isn't abstracted to the level of individual cores usually since most HPC jobs care about memory bandwidth - even with Infiniband or other techniques throughput / latency is much worse than on a single machine. Of course, multiple machines are connected (usually using MPI / Infiniband) but important to try to minimize communication between nodes where possible. For AI workloads, they are running GPUs - so 10K+ cores on a single device so even less likely to be talking about cores here.
femiagbabiaka 1y ago
This post is just about a reference architecture, but last I knew OpenAI ran VMs on Azure. https://openai.com/index/scaling-kubernetes-to-7500-nodes/ https://openai.com/index/scaling-kubernetes-to-7500-nodes/

up2isomorphism 1y ago

“Perhaps my spiciest take from this entire project: most clusters don’t actually need the level of reliability and durability that etcd provides.” This assumption is completely out of touch, and is especially funny when the goal is to build an extra large cluster.

itsnowandnever 1y ago
etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.
- jauntywundrkind 1y ago
  The API server is the thing. It so happens that the API server can mostly be a thin shell over etcd. But etcd itself while so common is not sacrosanct. https://github.com/k3s-io/kine https://github.com/k3s-io/kine is a reasonably adequate substitute for etcd. sqlite, MySQL, PostgreSQL can also be substituted in. Etcd is from the ground up built to be more scale-out reliable, and that rocks to have baked in. But given how easy it is to substitute etcd out, I feel like we are at least a little off if we're trying to say "etcd is also the entire point of k8s" (the APIserver is)
  - itsnowandnever 1y ago
    that's fair but that 99% of all apiserver deployments in the world have the same standard boilerplate footprint is a large part of why it became so ubiquitous. that people running it locally don't have to make any decisions about how to deploy which database or why to use this one over that one... and that's also the same situation in production so people doing stuff in dev aren't punched in the face by an exponentially more complex system in production is huge.
  - dmlittle 1y ago
    It's been a while since I've checked this but a few years ago we tried to limit test kine on a large-ish cluster and it performed pretty poorly. It's fine for small clusters but the way they have to implement the watch semantics makes it perform poorly (at least this was the case a few years ago).

rixed 1y ago

I don't get the point of benchmarking k8s without the guarantees of etcd. At some point, you are just competing with clusterssh.

cyberax 1y ago
How often do you have sudden host failures? Especially if you use a half-decent server with redundant components for the DB node? Once in maybe 10 years?
- dmlittle 1y ago
  The node failure rate is much higher than that. On a 1M node cluster of cloud-managed instances (AWS, GCP, Azure, etc.) you'd likely see failures a few times a month, if not more.
  - cyberax 1y ago
    Yep. And the chances that the DB node with the control plane fails are therefore less than one in ten thousand.
- rixed 1y ago
  It's not about failure, it's about servicing the host running etcd or the network around it.

wppick 1y ago

If you don't need the isolation of of k8s then don't forget about erlang, which is another option to scale up to 1 million functions. Obviously k8s containers (which are fundamentally just isolated processes) and erlang processes are not interchangeable things, but when thinking about needing in the order of millions of processes erlang is pretty good prior art

theptip 1y ago
This is 1m nodes, you typically run tens or hundreds of pods per node, each with one or more containers. So more like 100m+ functions if I follow the Erlang analogy correctly?
- reactordev 1y ago
  This is not analogous. It’s just someone beating the Erlang drum. You can’t PyTorch in Erlang.
  - wppick 1y ago
    Kubernetes is way heavier than Erlang’s lightweight processes, so for millions of tasks at scale, a middle-ground solution could blend Erlang’s concurrency efficiency with k8s’ orchestration power, dodging containers’ overhead while keeping flexibility for diverse workloads. That's if you don't actually need the strict isolation of pods/containers and you're just trying to run something at massive scale. I don't get why so many people want to run everything as heavy container processes or pods vs coming up with a better solution. The point is we don't have to fit every problem into the shoe called kubernetes if it doesn't seem to fit, and we should look at other ways to spin up millions of processes
  - victorbjorklund 1y ago
    There are similar libraies in Elixir. Is the ecosystem for ML as developed as for python? Nope, but not every ML project needs the most obscure libraries etc. (For the record I don't really see Erlang clusters as a replacement for k8s)

jeffinhat 1y ago

This is an awesome experiment and write up. I really appreciate the reproducibility. I would like to see how moving to database that scales write throughput with replicas would behave, namely FoundationDB. I think this will require more than an intermediary like kine to be efficient, as the author illustrates the apisever does a fair bit of its own watching and keeping state. I also think there's benefit, at least for blast radius, to shard the server by api group or namespace. I think years ago this would have been a non starter with the community, but given AWS has replaced etcd (or at least aspects) with their internal log service for their large cluster offering, I bet there's some appetite for making this interchangable and bringing and open source solution to market. I share the authors viewpoint that for modern cloud based deployments, you're probably best avoiding it and relying on VMs being stable and recoverable. I think reliability does matter if you want to actually realize the "borg" value and run it on bare metal across a serious fleet. I haven't found the business justification to work on that though!

ymelnyk 1y ago
Here you go, Kine FoundationDB backend https://github.com/melgenek/f8n https://github.com/melgenek/f8n To be honest, I was building it with the purpose of matching the Etcd scale, but making foundationdb a multitenant data store. But with the recent craze of scalability , I'll be investing time into understanding how far foundationdb can be pushed as a K8s data store. Stay tuned.
- jeffinhat 1y ago
  Awesome, I will! It would be great to see where the limits are with this approach. I think at some point, you need to go deeper into the apiserver for scale than an API compatible shim, but this is just conjecture and not real data.

halayli 1y ago

without publishing mem_etcd code, and without telling us what happens when one of the etcd/mem_etcd node dies to compare, this write up doesn't provide much information.

Brybry 1y ago
I believe the code is here: https://github.com/bchess/k8s-1m/tree/main/mem_etcd https://github.com/bchess/k8s-1m/tree/main/mem_etcd https://github.com/bchess/k8s-1m/blob/main/RUNNING.adoc#mem_etcd https://github.com/bchess/k8s-1m/blob/main/RUNNING.adoc#mem_...

__turbobrew__ 1y ago

I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a way to get around the scalability issues, but unfortunately for the rest of us who do not have spanner there aren’t any great options. I feel like at a fundamental level that pod affinity, antiaffinity, and topology spreads are not compatible with very large clusters due to the complexity explosion in large clusters. Another thing to consider is that the larger a cluster becomes, the larger the blast radius is. I have had clusters of 10k nodes spectacularly fail due to code bugs within k8s. Sharding total compute capacity compute capacity into multiple isolated k8s clusters reduces the likelihood that a software bug is going to take down everything as you can carefully upgrade only a single cell at a time with bake periods between each cell.

femiagbabiaka 1y ago
Yep, every cluster approaching 10k I know of has either pared back etcd's durability guarantees or rewritten and replaced it in some manner. Actually the post goes into detail about doing this exactly, the Alibaba paper they reference says about the same. > Sharding total compute capacity compute capacity into multiple isolated k8s clusters reduces the likelihood that a software bug is going to take down everything as you can carefully upgrade only a single cell at a time with bake periods between each cell. Yeah, I've been meaning to try out something like Armada to simplify things on the cluster-user side. Cluster-providers have lots of tools to make managing multiple clusters easier but if it means having to rewrite every batch job..
jen20 1y ago
AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently. [1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazon-eks-ultra-scale-clusters/ https://aws.amazon.com/blogs/containers/under-the-hood-amazo... [2]: https://cloud.google.com/blog/products/containers-kubernetes/gke-65k-nodes-and-counting https://cloud.google.com/blog/products/containers-kubernetes... [3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-to-the-latest-azure-cosmos-db-announcements/#:~:text=Powering%20Kubernetes%20with%20etcd%20API,complexity%20of%20managing%20etcd%20clusters. https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...

ai-christianson 1y ago

Love this. There's no reason k8s shouldn't scale much further.

wb14123 1y ago

Instead of giving up the good guarantee of etcd, a better approach maybe grouping some nodes together to create a tree like structure with sub clusters.

AlphaSite 1y ago
That was the whole concept behind KCP iirc. It was designed to provide tenancy atop 1 or more clusters.

skeptrune 1y ago

I read this as napkin math[1] for Kube and thoroughly enjoyed. You can only find the important numbers relative to performance and scaling by trying to accomplish some kind of goal. Benchmarks are mostly bikeshedding. [1]: https://sirupsen.com/napkin https://sirupsen.com/napkin

vebgen 1y ago

This is an absolutely incredible technical deep-dive. The section on replacing etcd with mem_etcd resonates with challenges we've been tackling at a much smaller scale building an AI agent system. A few thoughts: *On watch streams and caching*: Your observation about the B-Tree vs hashmap cache tradeoff is fascinating. We hit similar contention issues with our agent's context manager - switched from a simple dict to a more complex indexed structure for faster "list all relevant context" queries, but update performance suffered. The lesson about O(1) writes vs O(log n) reads being the wrong tradeoff for high-write workloads is universal. *On optimistic concurrency for scheduling*: The scatter-gather scheduler design is elegant. We use a similar pattern for our dual-agent system (TARS planner + CASE executor) where both agents operate semi-independently but need coordination. Your point about "presuming no conflicts, but handling them when they occur" is exactly what we learned - pessimistic locking kills throughput far worse than occasional retries. *The spicy take on durability*: "Most clusters don't need etcd's reliability" is provocative but I suspect correct for many use cases. For our Django development agent, we keep execution history in SQLite with WAL mode (no fsync), betting that if the host crashes, we'd rather rebuild from Git than wait on every write. Similar philosophy. The mem_etcd implementation in Rust is particularly interesting - curious if you considered using FoundationDB's storage engine or something similar vs rolling your own? The per-prefix file approach is clever for reducing write amplification. Fantastic work - this kind of empirical systems research is exactly what the community needs more of. The "what are the REAL limits" approach vs "conventional wisdom says X" is refreshing.

rememberlenny 1y ago

People don’t realize how crucial Ben was in the forming of OpenAI as it is known today. This is an extremely underrated post.

kawsper 1y ago

That's really impressive and an interesting experiment. I was about to say that Nomad did something similar, but that was 2 million Docker containers across 6100 nodes, https://www.hashicorp.com/en/c2m https://www.hashicorp.com/en/c2m

throwdbaaway 1y ago

This looks impressive. As someone who is not familiar with ML, I do have a question -- surely in 2025 there must be a way to schedule a large pytorch job across multiple k8s clusters? EKS and GKE already provide VPC native flat network by default .

sailingparrot 1y ago
The issue isn’t so much scheduling as it is stability. More clusters means one more layer of things that can crash your (very expensive) training. You also then still need to write tooling to manage cross cluster trainings correctly just as starting/stopping roughly at the same time, resuming from checkpoints, node health monitoring etc. Nothing dealbreaking, but if it could just work in a single cluster that would be nicer.

czhu12 1y ago

If anyone is looking for a gentler, Heroku like onramp to Kubernetes, its exactly why I built Canine [1]. In retrospect, at my previous company, what we really needed in the early days was something that was Heroku-like (don't make me think about infra (!)) but could be easily added to and scaled up over time, as our service grew. We eventually grew to about 10M users, using the site monthly, and had to do a huge effort to migrate to Kubernetes. Canine's philosophy is: full Kubernetes, with a deployment layer on top. If you ever out grow it, just dump Canine entirely, and work directly with the Kubernetes system it's operating. It even gives you all the K8s YAML config needed to offboard. It's also similar to how the dev infra works at Airbnb (where I worked before that) -- Kubernetes underneath, a user friendly interface on top.

rootlocus 1y ago
You're not the first one to think that k9 is a good name for a kubernetes related technology https://k9scli.io https://k9scli.io

brimstedt 1y ago

It's an interesting and fun experiment, but what are real usecases for such a cluster?

dbattaglia 1y ago
At my last employer Elastic we definitely ran into these limits on the cloud SaaS team moving Elastocsearch node containers from our proprietary orchestration to k8s. I’m not sure how they eventually solved it but I believe the plan was essentially sharding ES clusters to different regional k8s clusters.

hardwaresofton 1y ago

Lasto ask: Does anyone have benchmark results for kine?

rvz 1y ago

> Many limitations are imposed by software. Or the amount of funding a startup has. The bottom line is, you are not OpenAI or Google.

pluc 1y ago

> Early on in this project, I asked ChatGPT “I want to scale Kubernetes to 1 million nodes. What types of problems would I need to overcome?” click

breakingcups 1y ago
It's a shame the author led with something that carries about the same authority as a horoscope, since the rest of the article is actually quite interesting.
- tbrockman 1y ago
  If you have sufficient knowledge in the subject matter you're questioning ChatGPT about, you can fairly reliably discern complete bullshit from something plausibly true that warrants additional investigation (which I'd say is more useful than your typical horoscope). In isolation it seems worth the gamble to me, so long as you don't view it as much more than consulting the tea leaves.
  - pluc 1y ago
    You could also spend 5 minutes thinking instead of using a 10th of the Texas power grid to do it for you. Remember when we used to not know shit, and it would stay with you for days - before Google happened? The same is about to happen with mental acuity and AI. Use it or lose it - and I hope every fanboys' brain turns to mush before they witness irreversible AI-caused climate catastrophes solely to be able to speak natural english to a search engine. If you outsource your thinking process, don't come bitching when it's gone.
    - tbrockman 1y ago
      Yeah, I’m actually fairly sympathetic to your perspective, that’s why I said “in isolation”. I might think it’s possible to balance use in a way that’s not as detrimental to myself personally or others (self-hosted models, judicious use, etc.), but I definitely don’t disagree that it seems likely to do more harm than good currently.

adsharma 1y ago

I saw one fleeting reference to a CNI in the article. Anyone familiar with the space will tell you this is the biggest blocker in production. You will have to pay for an "enterprise" CNI to make it work.