3 ms·
Sparrow: Distributed, Low Latency Scheduling (2013) [pdf]
- npalli 3y agoFrom 2013. Doesn't seem like it was really updated. https://github.com/radlab/sparrow?tab=readme-ov-file https://github.com/radlab/sparrow?tab=readme-ov-file
- kthejoker2 3y agoIt's more interesting archaeologically as a paper written by the creators of Spark and founders of Databricks, Matei Zaharia, Ion Stoica, Patrick Wendell (and it thanks Ali Ghodsi and cites Reynold Xin) Lots of interesting papers around that time for microtask scheduling ...
- sitkack 3y agoWhy did you omit the main author, Kay Ousterhout?
- kthejoker2 3y agoBecause she isn't one of the founders of Databricks. No slight intended, it's Kay's project, but any interest in Sparrow in 2023 is purely for nostalgia.
- maxmcd 3y agoThis is a bit unrelated, but I've been looking into AWS Lambda or fly.io style scheduling recently. This specific response from and AWS Lambda engineer has been floating around in my brain: https://news.ycombinator.com/item?id=39501188 https://news.ycombinator.com/item?id=39501188 It seems like there is quite a bit of similarity between the type of scheduling that FAAS platforms need to do and the type of scheduling that Sparrow/Spark needs to do. Anyone know if these are truly related research spaces, or do they have enough differences that they can be viewed as distinct? Is it worth looking to data analytics frameworks for FAAS scheduling inspiration?
- robertkoss 3y agoAnother Matei Paper :)
- rfonseca 3y agoReally great research work from Kay Ousterhout. Sparrow extends the "power of two choices" [1] to a more general formulation, and makes it really cheap to find very good choices (i.e., low-load servers) to run a task, without having to check the load on every server. As the number of tasks and number of servers grow, keeping an accurate picture of the load on every server makes the scheduler become a bottleneck. Kay's work showed a very interesting point in this design space for fast and high-quality scheduling. Her PhD work went on to really understand the performance of complex distributed programs, like large Spark jobs, and a lot of at least earlier performance instrumentation in Spark is due to her.[2] [1] https://brooker.co.za/blog/2012/01/17/two-random.html https://brooker.co.za/blog/2012/01/17/two-random.html [2] http://kayousterhout.org/ http://kayousterhout.org/