4 ms·

When you explain the purpose of COST in this way, it is insightful and the value becomes very apparent. Noting that so many systems have not been compared again

by bmh100 12y ago

When you explain the purpose of COST in this way, it is insightful and the value becomes very apparent. Noting that so many systems have not been compared against a single thread is important, since many developers likely would assume that. Do you feel that your COST work helps provide scalability break points where a developer could choose one system over another when performance is the primary concern? I can see that being helpful in the prototype and early scaling stages, as well as for ad-hoc work.

mrry 12y ago
I think in its current form, COST simply provides a reality check: for some problems, a given system may be completely inappropriate when a simple single-machine implementation exists. Along with raising standards for distributed systems evaluation, I hope that it encourages people to innovate on the single-machine implementations: for example, the GAS idea in PowerGraph [1] is an antecedent of the Hilbert-curve implementation in Frank's first blog post, and it would have been great if the original GAS implementation were separable from the particular distributed system for which it was built. I fantasize about this work inspiring a competition like the sort benchmark [2], but for single-threaded graph algorithms. Not only would this encourage people to develop usable single-threaded implementations, but it have the side effect of increasing the COST for all distributed implementations.... As for providing scalability break points, I think this work goes some of the way, but doesn't take into account the fact that the input data for a particular real-world computation might not reside in a conveniently-compressed format on a local SSD. Extracting the Common Crawl from an HDFS cluster just to replicate the blog post's experiments could take much longer than the computation itself; extracting the social graph from Facebook's Tao [3] to do offline processing might be similarly non-trivial. Academic experiments don't typically take this cost into account, but recent work like Musketeer [4] has started to consider it, and even offer ways to automate the system choice problem. [1] https://www.usenix.org/conference/osdi12/technical-sessions/presentation/gonzalez https://www.usenix.org/conference/osdi12/technical-sessions/... [2] http://www.sortbenchmark.org/ http://www.sortbenchmark.org/ [3] https://www.usenix.org/conference/atc13/technical-sessions/presentation/bronson https://www.usenix.org/conference/atc13/technical-sessions/p... [4] http://www.cl.cam.ac.uk/research/srg/netos/camsas/musketeer/ http://www.cl.cam.ac.uk/research/srg/netos/camsas/musketeer/

by bmh100 12y ago

mrry 12y ago

I think in its current form, COST simply provides a reality check: for some problems, a given system may be completely inappropriate when a simple single-machine implementation exists. Along with raising standards for distributed systems evaluation, I hope that it encourages people to innovate on the single-machine implementations: for example, the GAS idea in PowerGraph [1] is an antecedent of the Hilbert-curve implementation in Frank's first blog post, and it would have been great if the original GAS implementation were separable from the particular distributed system for which it was built. I fantasize about this work inspiring a competition like the sort benchmark [2], but for single-threaded graph algorithms. Not only would this encourage people to develop usable single-threaded implementations, but it have the side effect of increasing the COST for all distributed implementations.... As for providing scalability break points, I think this work goes some of the way, but doesn't take into account the fact that the input data for a particular real-world computation might not reside in a conveniently-compressed format on a local SSD. Extracting the Common Crawl from an HDFS cluster just to replicate the blog post's experiments could take much longer than the computation itself; extracting the social graph from Facebook's Tao [3] to do offline processing might be similarly non-trivial. Academic experiments don't typically take this cost into account, but recent work like Musketeer [4] has started to consider it, and even offer ways to automate the system choice problem. [1] https://www.usenix.org/conference/osdi12/technical-sessions/presentation/gonzalez https://www.usenix.org/conference/osdi12/technical-sessions/... [2] http://www.sortbenchmark.org/ http://www.sortbenchmark.org/ [3] https://www.usenix.org/conference/atc13/technical-sessions/presentation/bronson https://www.usenix.org/conference/atc13/technical-sessions/p... [4] http://www.cl.cam.ac.uk/research/srg/netos/camsas/musketeer/ http://www.cl.cam.ac.uk/research/srg/netos/camsas/musketeer/