4 ms·

Anyone know if Netflix does anything for the k8s storage layer? I imagine they are at the scale where etcd starts to go kaboom? Or maybe they have enough cells

Anyone know if Netflix does anything for the k8s storage layer? I imagine they are at the scale where etcd starts to go kaboom? Or maybe they have enough cells where that isn’t a problem? Given Amazon and Google have their own secret sauce for replacing etcd, I am wondering if Netflix does anything special?

stackskipton 3mo ago
It's possible they are using kine: https://github.com/k3s-io/kine https://github.com/k3s-io/kine
scripni 3mo ago
This runs on AWS managed EKS these days, this talk goes into more detail about Netflix's special sauce around the k8s control plane: https://www.youtube.com/watch?v=vaTOiXR2KSM https://www.youtube.com/watch?v=vaTOiXR2KSM Netflix actually has much fewer cells than you'd expect btw, their special sauce IMO is federation and using a small subset of k8s APIs.
- __turbobrew__ 3mo ago
  I am surprised a company at that scale is running on managed EKS, maybe I underestimate how large the clusters are.
  - zbentley 3mo ago
    EKS can get pretty damn big, well into the thousands of nodes without much special tuning, and beyond that with some care and control plane monitoring. Expensive, though.
    - __turbobrew__ 3mo ago
      > Expensive, though. That is my point. I work at a large multinational and we run tens of thousands of kubernetes nodes on-prem and Im pretty sure that would be in the hundreds of millions of dollars per year to run in EKS. We run on-prem nodes about equivalent to c6a.32xlarge and even with 2 year reserved pricing you are looking at $17k/year/node. At 20000 nodes you are looking at $340 million/year, not including egress fees or any other AWS service charges (such as EBS). I can tell you with certainty that the all-in costs to run kubernetes on-prem (including staffing costs) is a lot less than $340 million/year AND we don’t have vendor lock in. In total we have 7 full time engineers building and running on-prem kubernetes. The more nodes you have, the more it makes sense as the team size is mostly independent of the number of nodes, so that team of 7 could also run 40000 nodes without issues. The cost becomes dominated by the capex to purchase hardware. I would say team size is log(nodes). For a company the scale of Netflix, I would assume the math is similar — especially since they already have in house expertise to run their own hardware — but maybe they get a very steep discount from AWS.

stackskipton 3mo ago

It's possible they are using kine: https://github.com/k3s-io/kine https://github.com/k3s-io/kine

scripni 3mo ago

This runs on AWS managed EKS these days, this talk goes into more detail about Netflix's special sauce around the k8s control plane: https://www.youtube.com/watch?v=vaTOiXR2KSM https://www.youtube.com/watch?v=vaTOiXR2KSM Netflix actually has much fewer cells than you'd expect btw, their special sauce IMO is federation and using a small subset of k8s APIs.

__turbobrew__ 3mo ago
I am surprised a company at that scale is running on managed EKS, maybe I underestimate how large the clusters are.
- zbentley 3mo ago
  EKS can get pretty damn big, well into the thousands of nodes without much special tuning, and beyond that with some care and control plane monitoring. Expensive, though.
  - __turbobrew__ 3mo ago
    > Expensive, though. That is my point. I work at a large multinational and we run tens of thousands of kubernetes nodes on-prem and Im pretty sure that would be in the hundreds of millions of dollars per year to run in EKS. We run on-prem nodes about equivalent to c6a.32xlarge and even with 2 year reserved pricing you are looking at $17k/year/node. At 20000 nodes you are looking at $340 million/year, not including egress fees or any other AWS service charges (such as EBS). I can tell you with certainty that the all-in costs to run kubernetes on-prem (including staffing costs) is a lot less than $340 million/year AND we don’t have vendor lock in. In total we have 7 full time engineers building and running on-prem kubernetes. The more nodes you have, the more it makes sense as the team size is mostly independent of the number of nodes, so that team of 7 could also run 40000 nodes without issues. The cost becomes dominated by the capex to purchase hardware. I would say team size is log(nodes). For a company the scale of Netflix, I would assume the math is similar — especially since they already have in house expertise to run their own hardware — but maybe they get a very steep discount from AWS.