I think in its current form, COST simply provides a reality check: for some problems, a given system may be completely inappropriate when a simple single-machin

I think in its current form, COST simply provides a reality check: for some problems, a given system may be completely inappropriate when a simple single-machine implementation exists. Along with raising standards for distributed systems evaluation, I hope that it encourages people to innovate on the single-machine implementations: for example, the GAS idea in PowerGraph [1] is an antecedent of the Hilbert-curve implementation in Frank's first blog post, and it would have been great if the original GAS implementation were separable from the particular distributed system for which it was built. I fantasize about this work inspiring a competition like the sort benchmark [2], but for single-threaded graph algorithms. Not only would this encourage people to develop usable single-threaded implementations, but it have the side effect of increasing the COST for all distributed implementations.... As for providing scalability break points, I think this work goes some of the way, but doesn't take into account the fact that the input data for a particular real-world computation might not reside in a conveniently-compressed format on a local SSD. Extracting the Common Crawl from an HDFS cluster just to replicate the blog post's experiments could take much longer than the computation itself; extracting the social graph from Facebook's Tao [3] to do offline processing might be similarly non-trivial. Academic experiments don't typically take this cost into account, but recent work like Musketeer [4] has started to consider it, and even offer ways to automate the system choice problem. [1] https://www.usenix.org/conference/osdi12/technical-sessions/presentation/gonzalez https://www.usenix.org/conference/osdi12/technical-sessions/... [2] http://www.sortbenchmark.org/ http://www.sortbenchmark.org/ [3] https://www.usenix.org/conference/atc13/technical-sessions/presentation/bronson https://www.usenix.org/conference/atc13/technical-sessions/p... [4] http://www.cl.cam.ac.uk/research/srg/netos/camsas/musketeer/ http://www.cl.cam.ac.uk/research/srg/netos/camsas/musketeer/