3 ms·

You seem to be very knowledgeable about Hadoop. Could you tell us a bit about the technical reasons why its performance is so god-awfully underwhelming. Is it a

by srean 12y ago

You seem to be very knowledgeable about Hadoop. Could you tell us a bit about the technical reasons why its performance is so god-awfully underwhelming. Is it a wrong choice of algorithms, internal data structure, architecture, design. I have used Google's implementation of mapreduce and then current version of Hadoop. Hadoop's performance was a piece of crap in comparison, and yes the the bottleneck was shuffle. With Java, I would have expected a 20~30% hit not a slowdown by 4~6 times and using roughly that many times more memory. Weirdly enough, I have heard that Python streaming on Hadoop outperforms the Java Hadoop, but I have not tried this myself. At that time UIUC's sector would outperform Hadoop too, and that was an university project maintained by a grad student. Finally has this changed with Hadoop 2.* And in case you have experienced Googles's mapreduce and Hadoop (which seems likely) I would like to know more.

x0x0 12y ago
Yahoo's hadoop has a variety of issues. I'd rather not say company names, but this comes from working at a company that had hadoop in production by 2007 and last I heard was running 10+pb datastores with 10k+ hadoop cores. Yahoo's hadoop writes all output (from mappers and reducers) to local disks which are typically spinning rust. This is an anti-pattern: you run multiple mappers per box, typically 1.1-1.3 per core, and you have multiple jobs running simultaneously. You then perform a disk based merge sort, and then every reducer must connect to every mapper to get the data associated with its keys. The second you get a hot disk, whole jobs get choked because of the api invariants: all mappers complete before you can finish (or perhaps start) sort, all sorts complete before reducers, all reducers finish before the next mapper pass. Further, avoiding having every reducer have to have high throughput network to every mapper is key; the hadoop design makes it difficult to optimize over network topology like higher within-rack bandwidth. There are far better designs implemented: never write to local disk from the mappers, instead writing to a global dfs then performing the sort in the dfs. I worked for a company that took hadoop 0.17 and made it roughly 10x more performant (and not on toy datasets: on petabyte scale datasets) with roughly 10 engineer-years with this design. I think it's also fair to say that the quality of engineers on hadoop, particularly when they worked at yahoo, was... subpar. There was a ton of low hanging fruit. Also, yahoo's workload wasn't, or so I've heard, highly cpu bound. They where using hadoop at least in part just for hdfs capacity. So if you aren't cpu constrained, a lot of optimizations don't matter. Better companies have to run at say 95%+ cpu or their cfo won't authorize more boxes. You can also go after things like optimizing your compressors / decompressors, which are probably by far the hottest bits of code. Get icc, break out vtune, hire a really good optimizing dev to go after even small gains, etc. Finally, the hdfs design is pretty shitty, on two fronts: the namenode gets slow (choking everyone), and the disk-based format is missing lots of things to enable optimizations like certain types of append, or indexing. Indexing is incredibly powerful for things like join, enabling a map-side join instead of having to read all your data once, sort by join keys, then reread in your reducers. Another problem with hdfs design is capacity adding; hdfs has shitty rebalancing tools. The typical problem is this: hot data is new data. So if you add capacity to hdfs, your old capacity is say 80% full and your new capacity is 0% full so where does all new data go? To the new capacity. Which is quickly 100% saturated and choking your jobs. Properly rebalancing a dfs is a hard problem on it's own. Further, hdfs' use of triple replication (well, you can actually set the replication level) instead of better algorithms like erasure encoding does no favors to performance as well. edit: by the way, never writing to local disk and instead going from mappers straight to dfs then performing a rough/fine grained sort in the dfs is a lot like google's design. You do trade network i/o for disk i/o, but that's often easier to scale and, as mentioned, easier to optimize over your network topology. Finally, you can do multi-pass sorts partially locally by running sorters on your disk nodes.
- srean 12y ago
  I already upvoted, but this needs more, thanks for the excellent read. Some of the inefficiencies you describe are pretty crazy ! If any of the optimizations you did are documented in blogs, would love to have a pointer. @cbsmith Just in case you have links handy could you post them. Much appreciated.
  - cbsmith 12y ago
    These issues with Hadoop and HDFS in particular are pretty well documented even by people in the Hadoop project. There was also a bunch of blog posts posted at Yahoo shortly after Cutting's departure that looked at a bunch of other issues with the IO subsystem. Much of this stuff has been heavily refactored, particularly with Hadoop 2.0, so the problems are at least much smaller than they used to be. Early Hadoop was not exactly a masterful work in terms of the code & algos. What was great about it was the community that was built around it.
  - x0x0 12y ago
    see eg quantcast's experiences with why hadoop is slow on big jobs [1], the design of their sorter architecture [2], and some benchmark wanking [3] [1] https://www.quantcast.com/inside-quantcast/2012/10/letter-from-the-big-data-front-hadoops-big-scalability-challenge/?jump=engblog https://www.quantcast.com/inside-quantcast/2012/10/letter-fr... [2] https://www.quantcast.com/inside-quantcast/2012/10/quantsort-revolution-in-map-reduce-performance-and-operation/ https://www.quantcast.com/inside-quantcast/2012/10/quantsort... [3] https://www.quantcast.com/inside-quantcast/2013/12/petabyte-sort/ https://www.quantcast.com/inside-quantcast/2013/12/petabyte-...
- cbsmith 12y ago
  > I'd rather not say company names, but this comes from working at a company that had hadoop in production by 2007 and last I heard was running 10+pb datastores with 10k+ hadoop cores. I'm going to say... Netseer. ;-)
- cbsmith 12y ago
  > Another problem with hdfs design is capacity adding; hdfs has shitty rebalancing tools. The typical problem is this: hot data is new data. So if you add capacity to hdfs, your old capacity is say 80% full and your new capacity is 0% full so where does all new data go? To the new capacity. Which is quickly 100% saturated and choking your jobs. Properly rebalancing a dfs is a hard problem on it's own. Further, hdfs' use of triple replication (well, you can actually set the replication level) instead of better algorithms like erasure encoding does no favors to performance as well. The tools for HDFS balancing are quite primitive, which is why a lot of people like using MapR or Cassandra, but the rebalancing tools built in to HDFS after a 0.18.x along with a few good practices would get you there... just not the most efficient way possible.

by srean 12y ago

x0x0 12y ago

Yahoo's hadoop has a variety of issues. I'd rather not say company names, but this comes from working at a company that had hadoop in production by 2007 and last I heard was running 10+pb datastores with 10k+ hadoop cores. Yahoo's hadoop writes all output (from mappers and reducers) to local disks which are typically spinning rust. This is an anti-pattern: you run multiple mappers per box, typically 1.1-1.3 per core, and you have multiple jobs running simultaneously. You then perform a disk based merge sort, and then every reducer must connect to every mapper to get the data associated with its keys. The second you get a hot disk, whole jobs get choked because of the api invariants: all mappers complete before you can finish (or perhaps start) sort, all sorts complete before reducers, all reducers finish before the next mapper pass. Further, avoiding having every reducer have to have high throughput network to every mapper is key; the hadoop design makes it difficult to optimize over network topology like higher within-rack bandwidth. There are far better designs implemented: never write to local disk from the mappers, instead writing to a global dfs then performing the sort in the dfs. I worked for a company that took hadoop 0.17 and made it roughly 10x more performant (and not on toy datasets: on petabyte scale datasets) with roughly 10 engineer-years with this design. I think it's also fair to say that the quality of engineers on hadoop, particularly when they worked at yahoo, was... subpar. There was a ton of low hanging fruit. Also, yahoo's workload wasn't, or so I've heard, highly cpu bound. They where using hadoop at least in part just for hdfs capacity. So if you aren't cpu constrained, a lot of optimizations don't matter. Better companies have to run at say 95%+ cpu or their cfo won't authorize more boxes. You can also go after things like optimizing your compressors / decompressors, which are probably by far the hottest bits of code. Get icc, break out vtune, hire a really good optimizing dev to go after even small gains, etc. Finally, the hdfs design is pretty shitty, on two fronts: the namenode gets slow (choking everyone), and the disk-based format is missing lots of things to enable optimizations like certain types of append, or indexing. Indexing is incredibly powerful for things like join, enabling a map-side join instead of having to read all your data once, sort by join keys, then reread in your reducers. Another problem with hdfs design is capacity adding; hdfs has shitty rebalancing tools. The typical problem is this: hot data is new data. So if you add capacity to hdfs, your old capacity is say 80% full and your new capacity is 0% full so where does all new data go? To the new capacity. Which is quickly 100% saturated and choking your jobs. Properly rebalancing a dfs is a hard problem on it's own. Further, hdfs' use of triple replication (well, you can actually set the replication level) instead of better algorithms like erasure encoding does no favors to performance as well. edit: by the way, never writing to local disk and instead going from mappers straight to dfs then performing a rough/fine grained sort in the dfs is a lot like google's design. You do trade network i/o for disk i/o, but that's often easier to scale and, as mentioned, easier to optimize over your network topology. Finally, you can do multi-pass sorts partially locally by running sorters on your disk nodes.

srean 12y ago
I already upvoted, but this needs more, thanks for the excellent read. Some of the inefficiencies you describe are pretty crazy ! If any of the optimizations you did are documented in blogs, would love to have a pointer. @cbsmith Just in case you have links handy could you post them. Much appreciated.
- cbsmith 12y ago
  These issues with Hadoop and HDFS in particular are pretty well documented even by people in the Hadoop project. There was also a bunch of blog posts posted at Yahoo shortly after Cutting's departure that looked at a bunch of other issues with the IO subsystem. Much of this stuff has been heavily refactored, particularly with Hadoop 2.0, so the problems are at least much smaller than they used to be. Early Hadoop was not exactly a masterful work in terms of the code & algos. What was great about it was the community that was built around it.
- x0x0 12y ago
  see eg quantcast's experiences with why hadoop is slow on big jobs [1], the design of their sorter architecture [2], and some benchmark wanking [3] [1] https://www.quantcast.com/inside-quantcast/2012/10/letter-from-the-big-data-front-hadoops-big-scalability-challenge/?jump=engblog https://www.quantcast.com/inside-quantcast/2012/10/letter-fr... [2] https://www.quantcast.com/inside-quantcast/2012/10/quantsort-revolution-in-map-reduce-performance-and-operation/ https://www.quantcast.com/inside-quantcast/2012/10/quantsort... [3] https://www.quantcast.com/inside-quantcast/2013/12/petabyte-sort/ https://www.quantcast.com/inside-quantcast/2013/12/petabyte-...
cbsmith 12y ago
> I'd rather not say company names, but this comes from working at a company that had hadoop in production by 2007 and last I heard was running 10+pb datastores with 10k+ hadoop cores. I'm going to say... Netseer. ;-)
cbsmith 12y ago
> Another problem with hdfs design is capacity adding; hdfs has shitty rebalancing tools. The typical problem is this: hot data is new data. So if you add capacity to hdfs, your old capacity is say 80% full and your new capacity is 0% full so where does all new data go? To the new capacity. Which is quickly 100% saturated and choking your jobs. Properly rebalancing a dfs is a hard problem on it's own. Further, hdfs' use of triple replication (well, you can actually set the replication level) instead of better algorithms like erasure encoding does no favors to performance as well. The tools for HDFS balancing are quite primitive, which is why a lot of people like using MapR or Cassandra, but the rebalancing tools built in to HDFS after a 0.18.x along with a few good practices would get you there... just not the most efficient way possible.