3 ms·
Note that the author himself is working on a framework for distributed differential dataflow computations (dataflow computations in which it is very cheap to ma
by lambda 9y ago
Note that the author himself is working on a framework for distributed differential dataflow computations (dataflow computations in which it is very cheap to make small updates to the inputs and see the differences in the outputs), and he has plans to apply the same metrics (comparing to a single-threaded implementation) to his own work once he gets a little further:
https://github.com/frankmcsherry/blog/blob/master/posts/2017-03-28.md https://github.com/frankmcsherry/blog/blob/master/posts/2017...
I think that answering the "fastest way to do X in a cluster" question is only particularly interesting if X is something that you need to do in a cluster. But clearly they are testing X that is capable of being done on a mid-range laptop. There's a real question of whether the X that they are testing will actually scale to larger datasets than the laptop (or a decently beefy single node) could handle; it's entirely possible that the cluster solution will blow up at that point. And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice.
- kbenson 9y ago> And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice. As the author hi,self points out, amusingly. These numbers are for sure more interesting. SEED does better than any number I can actually achieve on my laptop. That demonstrates non-triviality, in that their system isn't just a laptop computing the answer and 156 cores mining bitcoin. I really enjoyed the snark in this, and the series of blog posts he links in the beginning. I tend to be nice to a fault myself unless someone has shown themselves to not be acting in kind (and even then I confirm), but I admit I take quite a bit of voyeuristic pleasure in people being assholes in smart and interesting ways. Vicariously living and all that, etc...