3 ms·

Too late to join the story, but I am really curious if datacenter nodes intended for heavy mapreduce use implement this layer in user space. The bottleneck for

by srean 12y ago

Too late to join the story, but I am really curious if datacenter nodes intended for heavy mapreduce use implement this layer in user space. The bottleneck for such processes is typically network I/O and I can imagine that taking control of the network in the user space might offer some modest to significant wins. For Hadoop in particular network packets needs to traverse quite a few layers before it is accessible to the application. Has anyone done this sort of a thing for mapreduce. Pointers to any writeup would be awesome. In fact TCP itself might be an overkill for mapreduce. The reduce functions used are typically associative and commutative. So as long as the keys and values are contained entirely within a datapacket, proper sequencing is not even needed. Any sequence would suffice.

meowface 12y ago
I believe Google has experimented with this sort of thing, as have a few other companies that sell a supposedly more efficient/faster userspace TCP stack as an enterprise product.
wmf 12y ago
There has been some work about using zero-copy RDMA and RoCE (which bypass the kernel as a side effect) for analytics. Hadoop in particular is so slow that the kernel is unlikely to be a bottleneck, but more optimized runtimes like Spark might benefit.
- srean 12y ago
  I dont understand why is Hadoop so freaking slow. I am no fan of Java (to put it mildly) but Java does fairly well to keep in the 70~80% of a C++ code at the cost of 4 to 5 times more memory. My experience is that Hadoop is 4 to 6 times slower. Is it because of bad choice of internal algorithms, bad choice of internal data structures ? Bad I/O design ? Given the popularity it enjoys, and given its age, its a little frightening how much worse Hadoop is in comparison to some proprietary implementations. My hunch is that Hadoop's slowdown is in the shuffle phase, which is where faster network data transfer can help. I like abstractions that spark exposes but it still needs a lot of engineering to catchup. I have anecdotes where Spark is slower than Hadoop by quite a bit. All my experience is with Hadoop 1.x. Is Hadoop 2.x much better ?
donavanm 12y ago
There are a couple implementations of userland ip/tcp out there. As I recall at least one or two are fairly direct ports of a BSD network stack in to a library. These pair up with concepts like intels dpdk to move networking closer to the business logic. Here are some slides for one version: http://www.bsdcan.org/2014/schedule/attachments/260_libuinet_bsdcan2014.pdf http://www.bsdcan.org/2014/schedule/attachments/260_libuinet... Edit: with regards to motivation it's always been something along the lines of a network appliance that I've seen. The mainline linux network stack is more than capable of doing many millions of packets per second over hundreds of thousands of concurrent streams. The network stack will not be the limitation in something like batch processing.

by srean 12y ago

meowface 12y ago

I believe Google has experimented with this sort of thing, as have a few other companies that sell a supposedly more efficient/faster userspace TCP stack as an enterprise product.

wmf 12y ago

There has been some work about using zero-copy RDMA and RoCE (which bypass the kernel as a side effect) for analytics. Hadoop in particular is so slow that the kernel is unlikely to be a bottleneck, but more optimized runtimes like Spark might benefit.

srean 12y ago
I dont understand why is Hadoop so freaking slow. I am no fan of Java (to put it mildly) but Java does fairly well to keep in the 70~80% of a C++ code at the cost of 4 to 5 times more memory. My experience is that Hadoop is 4 to 6 times slower. Is it because of bad choice of internal algorithms, bad choice of internal data structures ? Bad I/O design ? Given the popularity it enjoys, and given its age, its a little frightening how much worse Hadoop is in comparison to some proprietary implementations. My hunch is that Hadoop's slowdown is in the shuffle phase, which is where faster network data transfer can help. I like abstractions that spark exposes but it still needs a lot of engineering to catchup. I have anecdotes where Spark is slower than Hadoop by quite a bit. All my experience is with Hadoop 1.x. Is Hadoop 2.x much better ?

donavanm 12y ago

There are a couple implementations of userland ip/tcp out there. As I recall at least one or two are fairly direct ports of a BSD network stack in to a library. These pair up with concepts like intels dpdk to move networking closer to the business logic. Here are some slides for one version: http://www.bsdcan.org/2014/schedule/attachments/260_libuinet_bsdcan2014.pdf http://www.bsdcan.org/2014/schedule/attachments/260_libuinet... Edit: with regards to motivation it's always been something along the lines of a network appliance that I've seen. The mainline linux network stack is more than capable of doing many millions of packets per second over hundreds of thousands of concurrent streams. The network stack will not be the limitation in something like batch processing.