3 ms·

Most famously, Siri (used to?) run on a very large scale Mesos deployment (10000s of nodes, much higher than Kubernetes can scale to). Unfortunately the origin

Most famously, Siri (used to?) run on a very large scale Mesos deployment (10000s of nodes, much higher than Kubernetes can scale to). Unfortunately the original article is lost, but here's a summary: https://daringfireball.net/linked/2015/04/29/siri-apache-mesos https://daringfireball.net/linked/2015/04/29/siri-apache-mes...

jeffbee 6y ago
OK but what was the utilization? I'm not really sure K8s is state-of-the-art either. There are published research papers about very-large-scale clusters with 80%+ resource utilization.
- dharmab 6y ago
  In our production experience, utilization had far more to do with the service owners (or autoscalers/auto-tuners) correctly choosing the cgroups and CPU scheduler allocations, as well as the kernel settings for cgroup slicing and CPU scheduler. We had Mesos clusters with 3% utilization and have Kubernetes clusters with 95%+ utilization. But we also have Kubernetes clusters with <10% utilization.
madhadron 6y ago
To be fair, Kubernetes right now only schedules relatively small clusters. But it turns out that the majority of the world is not Facebook or Google and only needs relatively small clusters.
- davidopp__ 6y ago
  > To be fair, Kubernetes right now only schedules relatively small clusters. This is not really true: https://news.ycombinator.com/item?id=25907312 https://news.ycombinator.com/item?id=25907312 https://cloud.google.com/blog/products/containers-kubernetes/google-kubernetes-engine-clusters-can-have-up-to-15000-nodes https://cloud.google.com/blog/products/containers-kubernetes... https://www.infoq.com/presentations/alibaba-kubernetes/ https://www.infoq.com/presentations/alibaba-kubernetes/
  - dharmab 6y ago
    Even those numbers (10k to 15k nodes and 100k containers) are smaller than what a great Mesos framework was capable of. Of course, this mattered to only a very small number of organizations.
  - madhadron 6y ago
    Yes, 10k to 15k machines is a relatively small cluster in my world.
thamer 6y ago
Wayback Machine has it[1], but there's not much more content than in Gruber's summary. [1] https://web.archive.org/web/20150429225603/https://mesosphere.com/blog/2015/04/23/apple-details-j-a-r-v-i-s-the-mesos-framework-that-runs-siri/ https://web.archive.org/web/20150429225603/https://mesospher...

jeffbee 6y ago

OK but what was the utilization? I'm not really sure K8s is state-of-the-art either. There are published research papers about very-large-scale clusters with 80%+ resource utilization.

dharmab 6y ago
In our production experience, utilization had far more to do with the service owners (or autoscalers/auto-tuners) correctly choosing the cgroups and CPU scheduler allocations, as well as the kernel settings for cgroup slicing and CPU scheduler. We had Mesos clusters with 3% utilization and have Kubernetes clusters with 95%+ utilization. But we also have Kubernetes clusters with <10% utilization.

madhadron 6y ago

To be fair, Kubernetes right now only schedules relatively small clusters. But it turns out that the majority of the world is not Facebook or Google and only needs relatively small clusters.

davidopp__ 6y ago
> To be fair, Kubernetes right now only schedules relatively small clusters. This is not really true: https://news.ycombinator.com/item?id=25907312 https://news.ycombinator.com/item?id=25907312 https://cloud.google.com/blog/products/containers-kubernetes/google-kubernetes-engine-clusters-can-have-up-to-15000-nodes https://cloud.google.com/blog/products/containers-kubernetes... https://www.infoq.com/presentations/alibaba-kubernetes/ https://www.infoq.com/presentations/alibaba-kubernetes/
- dharmab 6y ago
  Even those numbers (10k to 15k nodes and 100k containers) are smaller than what a great Mesos framework was capable of. Of course, this mattered to only a very small number of organizations.
- madhadron 6y ago
  Yes, 10k to 15k machines is a relatively small cluster in my world.

thamer 6y ago

Wayback Machine has it[1], but there's not much more content than in Gruber's summary. [1] https://web.archive.org/web/20150429225603/https://mesosphere.com/blog/2015/04/23/apple-details-j-a-r-v-i-s-the-mesos-framework-that-runs-siri/ https://web.archive.org/web/20150429225603/https://mesospher...