3 ms·

I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and e

I feel like etcd is one of the few use cases where Intel Optane would actually make sense. I build and run several bare metal clusters with over 10k nodes and etcd is by and large the biggest pain for us. Sometimes an etcd node just randomly stops accepting any proposals which halts the entire cluster until you can remove the bad etcd node. From what I remember, GKE has implemented an etcd shim on top of spanner as a way to get around the scalability issues, but unfortunately for the rest of us who do not have spanner there aren’t any great options. I feel like at a fundamental level that pod affinity, antiaffinity, and topology spreads are not compatible with very large clusters due to the complexity explosion in large clusters. Another thing to consider is that the larger a cluster becomes, the larger the blast radius is. I have had clusters of 10k nodes spectacularly fail due to code bugs within k8s. Sharding total compute capacity compute capacity into multiple isolated k8s clusters reduces the likelihood that a software bug is going to take down everything as you can carefully upgrade only a single cell at a time with bake periods between each cell.

femiagbabiaka 1y ago
Yep, every cluster approaching 10k I know of has either pared back etcd's durability guarantees or rewritten and replaced it in some manner. Actually the post goes into detail about doing this exactly, the Alibaba paper they reference says about the same. > Sharding total compute capacity compute capacity into multiple isolated k8s clusters reduces the likelihood that a software bug is going to take down everything as you can carefully upgrade only a single cell at a time with bake periods between each cell. Yeah, I've been meaning to try out something like Armada to simplify things on the cluster-user side. Cluster-providers have lots of tools to make managing multiple clusters easier but if it means having to rewrite every batch job..
jen20 1y ago
AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently. [1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazon-eks-ultra-scale-clusters/ https://aws.amazon.com/blogs/containers/under-the-hood-amazo... [2]: https://cloud.google.com/blog/products/containers-kubernetes/gke-65k-nodes-and-counting https://cloud.google.com/blog/products/containers-kubernetes... [3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-to-the-latest-azure-cosmos-db-announcements/#:~:text=Powering%20Kubernetes%20with%20etcd%20API,complexity%20of%20managing%20etcd%20clusters. https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...
- fowl2 1y ago
  Interestingly the public of Azure’s etcd-compatible service was withdrawn before exiting preview. [1] https://learn.microsoft.com/en-us/answers/questions/154061/about-etcd-support-in-azure-cosmos-db https://learn.microsoft.com/en-us/answers/questions/154061/a...
  - jen20 1y ago
    It's interesting they ever exposed it at all really! I don't think you can use Google's Spanner-based etcd replacement for a self-managed Kubernetes cluster, for example.
- jbnorth 1y ago
  While I can’t speak for the others, AWS doesn’t replace all of etcd. Only the raft consensus layer which is replaced with Journal which is an internal AWS service.
torginus 1y ago
It,s nice to know that the upper bound of the resiliency of a k8s cluster is the amount of redundancy etcd has - which is in essence a horizontally scaled monolith.
jamesblonde 1y ago
Is it throughput and latency that are the etcd bottlenecks? Our database, RonDB, is an in-memory open-source database (a fork of MySQL Cluster). We have scaled it to 100m reads/sec on AWS hardware (not even top of the line). Might be an interesting project to implement an open-source etcd shim on top of it? Reference: https://www.rondb.com/post/100m-key-lookups-sec-with-rest-api-and-python-short https://www.rondb.com/post/100m-key-lookups-sec-with-rest-ap...
- davidgl 1y ago
  See https://github.com/k3s-io/kine https://github.com/k3s-io/kine, k3s uses this to shim etcd to MySQL, Postgres and sqlite
- nonameiguess 1y ago
  The setting is configurable, but by default, etcd's Raft implementation requires a voting node to write to disk before it makes a vote, as in actually flushing to disk, not just writing to the file cache. Since you need a majority vote before a client can get a response, this is why it's strongly recommended you use the fastest possible disks, keep the nodes geographically close to each other, and etcd's default storage is only 2GB per node. All in all, it was a poor choice for Kubernetes to use this as its backend in the first place. Apparently, Google uses its own shim, but there is also kine, which was created a long time ago for k3s and allows you to use a RDBMS. k3s used sqlite as its default originally, but any API equivalent database would work. We should keep in mind etcd was meant to literally be the distributed /etc directory for CoreOS, something you would read from often but perform very few writes to. It's a configuration store. Kubernetes deciding to also use it for /var was never a great idea.
  - jamesblonde 1y ago
    RonDB uses a non-blocking 2PC algorithm - commits in memory, and then does a group commit of transactions to disk every 500ms. This means it can handle insane write throughput, as well as read throughput. However, if both your DB nodes fail, you could lose 500ms of data - which is not the end of the world for k8s. Normally, you would locate DB nodes in different AZes, reducing the probabilty of correlated failures.

femiagbabiaka 1y ago

Yep, every cluster approaching 10k I know of has either pared back etcd's durability guarantees or rewritten and replaced it in some manner. Actually the post goes into detail about doing this exactly, the Alibaba paper they reference says about the same. > Sharding total compute capacity compute capacity into multiple isolated k8s clusters reduces the likelihood that a software bug is going to take down everything as you can carefully upgrade only a single cell at a time with bake periods between each cell. Yeah, I've been meaning to try out something like Armada to simplify things on the cluster-user side. Cluster-providers have lots of tools to make managing multiple clusters easier but if it means having to rewrite every batch job..

jen20 1y ago

AFAIK all the hyperscalers have replaced etcd for their managed Kubernetes services [1], [2], [3] - though Azure is the least clear about what they actually use currently. [1]: https://aws.amazon.com/blogs/containers/under-the-hood-amazon-eks-ultra-scale-clusters/ https://aws.amazon.com/blogs/containers/under-the-hood-amazo... [2]: https://cloud.google.com/blog/products/containers-kubernetes/gke-65k-nodes-and-counting https://cloud.google.com/blog/products/containers-kubernetes... [3]: https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-to-the-latest-azure-cosmos-db-announcements/#:~:text=Powering%20Kubernetes%20with%20etcd%20API,complexity%20of%20managing%20etcd%20clusters. https://azure.microsoft.com/en-us/blog/a-cosmonaut-s-guide-t...

fowl2 1y ago
Interestingly the public of Azure’s etcd-compatible service was withdrawn before exiting preview. [1] https://learn.microsoft.com/en-us/answers/questions/154061/about-etcd-support-in-azure-cosmos-db https://learn.microsoft.com/en-us/answers/questions/154061/a...
- jen20 1y ago
  It's interesting they ever exposed it at all really! I don't think you can use Google's Spanner-based etcd replacement for a self-managed Kubernetes cluster, for example.
jbnorth 1y ago
While I can’t speak for the others, AWS doesn’t replace all of etcd. Only the raft consensus layer which is replaced with Journal which is an internal AWS service.

torginus 1y ago

It,s nice to know that the upper bound of the resiliency of a k8s cluster is the amount of redundancy etcd has - which is in essence a horizontally scaled monolith.

jamesblonde 1y ago

Is it throughput and latency that are the etcd bottlenecks? Our database, RonDB, is an in-memory open-source database (a fork of MySQL Cluster). We have scaled it to 100m reads/sec on AWS hardware (not even top of the line). Might be an interesting project to implement an open-source etcd shim on top of it? Reference: https://www.rondb.com/post/100m-key-lookups-sec-with-rest-api-and-python-short https://www.rondb.com/post/100m-key-lookups-sec-with-rest-ap...

davidgl 1y ago
See https://github.com/k3s-io/kine https://github.com/k3s-io/kine, k3s uses this to shim etcd to MySQL, Postgres and sqlite
nonameiguess 1y ago
The setting is configurable, but by default, etcd's Raft implementation requires a voting node to write to disk before it makes a vote, as in actually flushing to disk, not just writing to the file cache. Since you need a majority vote before a client can get a response, this is why it's strongly recommended you use the fastest possible disks, keep the nodes geographically close to each other, and etcd's default storage is only 2GB per node. All in all, it was a poor choice for Kubernetes to use this as its backend in the first place. Apparently, Google uses its own shim, but there is also kine, which was created a long time ago for k3s and allows you to use a RDBMS. k3s used sqlite as its default originally, but any API equivalent database would work. We should keep in mind etcd was meant to literally be the distributed /etc directory for CoreOS, something you would read from often but perform very few writes to. It's a configuration store. Kubernetes deciding to also use it for /var was never a great idea.
- jamesblonde 1y ago
  RonDB uses a non-blocking 2PC algorithm - commits in memory, and then does a group commit of transactions to disk every 500ms. This means it can handle insane write throughput, as well as read throughput. However, if both your DB nodes fail, you could lose 500ms of data - which is not the end of the world for k8s. Normally, you would locate DB nodes in different AZes, reducing the probabilty of correlated failures.