4 ms·

Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads

54 points by r4um 8y ago

dominotw 8y ago
CMU has a "self driving" db with the same name https://github.com/cmu-db/peloton https://github.com/cmu-db/peloton
- JumpCrisscross 8y ago
  Also, there's a B2C company called Peloton [1]. [1] https://www.onepeloton.com https://www.onepeloton.com
SteveNuts 8y ago
I'm always bummed when I don't see Nomad listed in blog posts like these. It has quite a few powerful features that people overlook.
elvinyung 8y ago
>Kubernetes hasn’t been able to scale to the large clusters that Uber requires, i.e. 10,000-plus, nor does it support elastic resource sharing. It is also not the ideal scheduler for batch workloads due to the high-churn nature of batch jobs. Feels like a semi-weird justification for NIH. Isn't there pretty active work in addressing most/all of those objections in k8s?
- tapoxi 8y ago
  Didn't they also have a weird "Postgres can't suit our needs so we're switching to MySQL" argument?
  - cphoover 8y ago
    yurp
- cdoxsey 8y ago
  Yes, though the problems are real. In its current form Kubernetes can only handle a cluster size of 5,000 nodes: https://kubernetes.io/docs/setup/cluster-large/ https://kubernetes.io/docs/setup/cluster-large/ and high-churn batch workloads don't work well. Kubernetes does support elastic resource sharing (assuming I understand what they need) via horizontal pod scaling and the cluster autoscaler: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler https://github.com/kubernetes/autoscaler/tree/master/cluster.... Although there are rough edges here, I was surprised by the quality of the code in the cluster autoscaler. I kept thinking of new edge cases and expected it to fall over, only to find out that they'd thought of that problem and it handled it gracefully. For batch workloads something like Argo can help: https://github.com/argoproj/argo https://github.com/argoproj/argo. So basically I agree, it probably would've made more sense to build something on top of Kubernetes. But Uber's a big company with a lot of engineers so I guess they have the resources to roll their own solution.
romed 8y ago
It's interesting how big of a difference it makes when you allow local disk I/O, and you need to schedule around it. At Google they disintermediate storage and all (not really all) I/O goes to their cluster FS (Colossus, sort of). They don't have to schedule around I/O resources because every process has access to the full I/O resources of the entire cluster at any time from any node. By contrast as soon as you let some open source or off-the-shelf commercial thing leak into your operations, it will demand ordinary POSIX disk I/O and then you've got big problems. I propose that some companies would actually be better off concentrating on disintermediated storage more, and I/O workload scheduling less.
- nemothekid 8y ago
  >I propose that some companies would actually be better off concentrating on disintermediated storage more, and I/O workload scheduling less. I feel like this isn't possible outside of Google. I've never had good performance (well any performance that you would feel comfortable running a database on) on any storage medium that wasn't POSIX disk I/O. I may be ignorant, but I've never seen a good disintermediate storage solution that is performant on something like AWS.
- makmanalp 8y ago
  > all (not really all) I/O goes to their cluster FS (Colossus, sort of). Well - sorta. Cluster FS just means disk I/O is now network I/O (EBS is a bit like this). Now you push the hairy problem to a different area: your network topology starts to matter a lot. You might end up having to think about not scheduling too many I/O heavy jobs behind the same switch or something. AND disk I/O is now competing with network I/O. AND it might now be hard to predict the destination of where all that I/O is going to be when scheduling the job, whereas previously it used to just be local. So the problem doesn't go away. But maybe you say it's just easier to wire everything up with gigE / infiniband so that you're massively under-capacity, and call it a day, and that might be a fair tradeoff.
- closeparen 8y ago
  The “Storage as a Service” needs to run somewhere. You can’t escape from running a storage aware scheduler. Either it will be the same as your primary scheduler, or storage services will be special snowflakes.
crb 8y ago
Nothing in the article about this being open source or available to the public?
- packetslave 8y ago
  why did you think it would be?
mistrial9 8y ago
from the article -- Peloton is built on top of Mesos
- detaro 8y ago
  They use mesos, and the article explains why they added something to it.

54 points by r4um 8y ago

dominotw 8y ago

CMU has a "self driving" db with the same name https://github.com/cmu-db/peloton https://github.com/cmu-db/peloton

JumpCrisscross 8y ago
Also, there's a B2C company called Peloton [1]. [1] https://www.onepeloton.com https://www.onepeloton.com

SteveNuts 8y ago

I'm always bummed when I don't see Nomad listed in blog posts like these. It has quite a few powerful features that people overlook.

elvinyung 8y ago

>Kubernetes hasn’t been able to scale to the large clusters that Uber requires, i.e. 10,000-plus, nor does it support elastic resource sharing. It is also not the ideal scheduler for batch workloads due to the high-churn nature of batch jobs. Feels like a semi-weird justification for NIH. Isn't there pretty active work in addressing most/all of those objections in k8s?

tapoxi 8y ago
Didn't they also have a weird "Postgres can't suit our needs so we're switching to MySQL" argument?
- cphoover 8y ago
  yurp
cdoxsey 8y ago
Yes, though the problems are real. In its current form Kubernetes can only handle a cluster size of 5,000 nodes: https://kubernetes.io/docs/setup/cluster-large/ https://kubernetes.io/docs/setup/cluster-large/ and high-churn batch workloads don't work well. Kubernetes does support elastic resource sharing (assuming I understand what they need) via horizontal pod scaling and the cluster autoscaler: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler https://github.com/kubernetes/autoscaler/tree/master/cluster.... Although there are rough edges here, I was surprised by the quality of the code in the cluster autoscaler. I kept thinking of new edge cases and expected it to fall over, only to find out that they'd thought of that problem and it handled it gracefully. For batch workloads something like Argo can help: https://github.com/argoproj/argo https://github.com/argoproj/argo. So basically I agree, it probably would've made more sense to build something on top of Kubernetes. But Uber's a big company with a lot of engineers so I guess they have the resources to roll their own solution.

romed 8y ago

It's interesting how big of a difference it makes when you allow local disk I/O, and you need to schedule around it. At Google they disintermediate storage and all (not really all) I/O goes to their cluster FS (Colossus, sort of). They don't have to schedule around I/O resources because every process has access to the full I/O resources of the entire cluster at any time from any node. By contrast as soon as you let some open source or off-the-shelf commercial thing leak into your operations, it will demand ordinary POSIX disk I/O and then you've got big problems. I propose that some companies would actually be better off concentrating on disintermediated storage more, and I/O workload scheduling less.

nemothekid 8y ago
>I propose that some companies would actually be better off concentrating on disintermediated storage more, and I/O workload scheduling less. I feel like this isn't possible outside of Google. I've never had good performance (well any performance that you would feel comfortable running a database on) on any storage medium that wasn't POSIX disk I/O. I may be ignorant, but I've never seen a good disintermediate storage solution that is performant on something like AWS.
makmanalp 8y ago
> all (not really all) I/O goes to their cluster FS (Colossus, sort of). Well - sorta. Cluster FS just means disk I/O is now network I/O (EBS is a bit like this). Now you push the hairy problem to a different area: your network topology starts to matter a lot. You might end up having to think about not scheduling too many I/O heavy jobs behind the same switch or something. AND disk I/O is now competing with network I/O. AND it might now be hard to predict the destination of where all that I/O is going to be when scheduling the job, whereas previously it used to just be local. So the problem doesn't go away. But maybe you say it's just easier to wire everything up with gigE / infiniband so that you're massively under-capacity, and call it a day, and that might be a fair tradeoff.
closeparen 8y ago
The “Storage as a Service” needs to run somewhere. You can’t escape from running a storage aware scheduler. Either it will be the same as your primary scheduler, or storage services will be special snowflakes.

crb 8y ago

Nothing in the article about this being open source or available to the public?

packetslave 8y ago
why did you think it would be?

mistrial9 8y ago

from the article -- Peloton is built on top of Mesos

detaro 8y ago
They use mesos, and the article explains why they added something to it.