3 ms·
Yikes, I had to stop when the author mentioned MapReduce for computing shortest distance. That's one of the problems for which MapReduce approach is extraordina
by bitL 11y ago
Yikes, I had to stop when the author mentioned MapReduce for computing shortest distance. That's one of the problems for which MapReduce approach is extraordinarily bad.
- taeric 11y agoSadly.... I'm noticing that folks that have done one MapReduce job are convinced it is the way to go for all jobs. It is scary just how fad based development can be.
- dimi-31 11y agoTo be fair, abstracting that fact away, the rest of the post is still an interesting thought experiment.
- ignoramous 11y agoA better approach at the scale being discussed in the article would be to...?
- karussell 11y agoWhat does 'scale' mean :) ? Solving graph problems are best done in-memory with a big machine (RAM!), otherwise you are an order of magnitude slower (at least) if you try to distribute. But if you really have no choice I would have a look if some of the graph databases has a good distributed model. My gut feeling tells me that even a bad approach there is faster than spark ...
- ignoramous 11y agoI guess Titan DB would fit the bill: http://thinkaurelius.github.io/titan/ http://thinkaurelius.github.io/titan/ Facebook has blogged about scaling Apache Giraph to insane number of Vertices/Edges.