3 ms·
There has been some work about using zero-copy RDMA and RoCE (which bypass the kernel as a side effect) for analytics. Hadoop in particular is so slow that the
by wmf 12y ago
There has been some work about using zero-copy RDMA and RoCE (which bypass the kernel as a side effect) for analytics. Hadoop in particular is so slow that the kernel is unlikely to be a bottleneck, but more optimized runtimes like Spark might benefit.
- srean 12y agoI dont understand why is Hadoop so freaking slow. I am no fan of Java (to put it mildly) but Java does fairly well to keep in the 70~80% of a C++ code at the cost of 4 to 5 times more memory. My experience is that Hadoop is 4 to 6 times slower. Is it because of bad choice of internal algorithms, bad choice of internal data structures ? Bad I/O design ? Given the popularity it enjoys, and given its age, its a little frightening how much worse Hadoop is in comparison to some proprietary implementations. My hunch is that Hadoop's slowdown is in the shuffle phase, which is where faster network data transfer can help. I like abstractions that spark exposes but it still needs a lot of engineering to catchup. I have anecdotes where Spark is slower than Hadoop by quite a bit. All my experience is with Hadoop 1.x. Is Hadoop 2.x much better ?