8 ms·
Open Sourcing Peloton, Uber’s Unified Resource Scheduler
- eropple 8y agoMy first reaction was "this just sounds like Mesos?". And it's cited in their page (which on first read I thought meant they were trying to act as a single pane of glass for Mesos/k8s/etc.): https://eng.uber.com/peloton/ https://eng.uber.com/peloton/ In the OP blog post though, they assert "to our knowledge, there is no other open source scheduler which combines all types of workloads for web-scale companies like Uber." And then, when you dig...it's just Mesos. They built a framework for Mesos. So, that's cool. But man, the puff piecery borders on dishonesty. I mean--Singularity has existed, and is implemented at very large scales, for a while. I'm sure Peloton is a fine scheduler, but there's a lot of huffing-one's-own-farts in the documentation here.
- grillorafael 8y agoI second you
- jypepin 8y agovery early on in the article there is a "Alternate cluster schedulers" section which cites mesos.
- rhizome 8y agoI think eropple's point might be that it's listed as an alternate without a (or much) difference.
- subway 8y agoSingularity isn't a scheduler. It's generally invoked by traditional hpc schedulers like moab or slurm. edit: clearly I'm thinking of a different Singularity than you.
- eropple 8y agoYeah - on a phone but I'm referring to the Mesos framework by HubSpot.
- r_klancer 8y agoImmediate reaction: Well, now there's Peloton, the fitness tech company, Peloton, the self-driving truck caravan company, and Peloton, the cluster scheduler...
- jonny_eh 8y agoWhere do all these names come from? Is it Latin or Greek or something?
- ericlewis 8y agohttps://en.wikipedia.org/wiki/Peloton https://en.wikipedia.org/wiki/Peloton I would guess
- chris_mc 8y agoI searched on google, a "peloton" is "the main field or group of cyclists in a race".
- WorldMaker 8y agoPeloton is a somewhat common French word for "ball" that became common in English as a sports term for a grouping in a bicycle race, by way of the Tour de France. https://www.merriam-webster.com/dictionary/peloton https://www.merriam-webster.com/dictionary/peloton
- jeromegv 8y agoSmall detail, that's not what it means in French, it's basically used the same was as in English (a group of cyclist, group of race car drivers, etc.) It is however pelota, ball, in Spanish.
- vimslayer 8y agoIn addition to what others have said here, "peloton" also means "fearless" in Finnish. I have no idea if or why they would've used that or if they're just referring to the cycling thing, but I guess "fearless" could also be kind of fitting for this project.
- dtrailin 8y agoPeloton should make a framework called Uber in response.
- platinumrad 8y agoThe term "peloton" obviously predates the company of the same name...
- fenwick67 8y agoThe term "uber" obviously predates the company of the same name...
- platinumrad 8y agoAnd nobody is wringing their hands when something is given that name.
- deleted 8y ago[deleted]
- zachrose 8y agoAt first glance I thought the "resources" here might be Uber drivers, which would be interesting. Is this thing/Apache Mesos abstract enough to allow for such a use?
- spyspy 8y agoIs it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.
- throwawaymath 8y agoI wouldn't say this is a rewrite of existing technology. They borrowed concepts from other well-known open source projects, but this is substantially a wrapper around Mesos, not a competing project. The technical overview of Peloton[1] is more clear about this than the open source announcement, which is what's featured here. _________________ 1. https://eng.uber.com/peloton/ https://eng.uber.com/peloton/
- sciurus 8y agoThanks for sharing! That is definitely a better link than the one posted.
- pj_mukh 8y agoAnyone who had to take an Uber after they switched away from Google Maps and onto their in-house half-baked mapping/navigation solution knows this is a huge problem. I'm assuming like all mapping solutions it'll get better but for now, it's just full of bad routes, over-optimizing turns, out-of-date detours (for MONTHS!) and non-sensical U-turns
- 0xDEFC0DE 8y agoNot to defend Uber/NIH syndrome, but if Google wanted to specifically charge Uber more for maps because of usage, could they legally do that? Maybe the rewrites are risk management?
- giancarlostoro 8y agoGoogle increased maps cost for everybody though, not just Uber. https://gadgets.ndtv.com/apps/features/google-maps-apis-new-pricing-impact-1907242 https://gadgets.ndtv.com/apps/features/google-maps-apis-new-...
- frostyj 8y agoJust call it (Uber) customized mesos. I find this article somehow deceiving and boring. I am pretty sure I can run this peloton thingy with most Mesos API calls.
- a-wu 8y agoIt pretty much is, but the appeal (at least to me) is abstracting away all the framework writing. It seems like it’s easier to run small, unique jobs on it similar to something like YARN.
- anonfmrtser 8y agoThis sounds a bit like Two Sigma’s Cook scheduler for Mesos. https://github.com/twosigma/Cook https://github.com/twosigma/Cook
- a-wu 8y agoI’ve worked with Mesos pretty extensively before and when Uber first announced Peloton last year I was intrigued. Peloton seems to be a wrapper around Mesos that allows for running smaller, unique jobs without having to write a Mesos framework for each. Writing a Mesos framework for every small job you have can get annoying when you just want to define how your job should run and don’t really care for the resources or task allocation of the job, and it seems like Peloton solves this on Mesos. It’s similar to YARN but not limited to Hadoop. It would have been useful for the project that I worked on because it was more geared for our use case and shifting from Mesos to k8s would’ve been a huge engineering project.
- filereaper 8y agoMesosphere created dcos-commons to reduce the friction of getting frameworks up and running. [1] https://github.com/mesosphere/dcos-commons https://github.com/mesosphere/dcos-commons
- ProAm 8y agoWould people use open source stuff from a morally questionable company? Especially when its just a re-write of existing technology posted to a different github repo? EDIT People get so up in arms about Google and Microsoft working with China and the military, but Uber has done some horrendous stuff on their own. Just curious where people think the line is OK to be.
- mandelbrotwurst 8y agoReact and Angular are quite popular.
- baroffoos 8y agoUsing their open source stuff doesn't fund them to do bad things.
- mandelbrotwurst 8y agoArguably it does indirectly - e.g. being able to say that you're responsible for Popular Tool X might help your brand, make it easier to make sales, etc
- matz1 8y agoThe thing with mortality is it's highly subjective. What horrible for one person is perfectly acceptable for other.
- notyourday 8y agoYou mean not a bike company?!
- thekhatribharat 8y agoWhat exactly is Uber running in its clusters? - Route Optimization - Demand Forecasting - Rider Hotspot Prediction This post doesn't exactly tell us the true nature of their workloads (other than the crude categorization - batch, stateful, stateless), nor does it talk about the inflection points where off-the-shelf solutions don't cut it anymore and such customization is required. I mean some before & after numbers / graphs on resource utilization would have really helped.