2 ms·
I already upvoted, but this needs more, thanks for the excellent read. Some of the inefficiencies you describe are pretty crazy ! If any of the optimizations yo
by srean 12y ago
I already upvoted, but this needs more, thanks for the excellent read. Some of the inefficiencies you describe are pretty crazy ! If any of the optimizations you did are documented in blogs, would love to have a pointer.
@cbsmith Just in case you have links handy could you post them. Much appreciated.
- cbsmith 12y agoThese issues with Hadoop and HDFS in particular are pretty well documented even by people in the Hadoop project. There was also a bunch of blog posts posted at Yahoo shortly after Cutting's departure that looked at a bunch of other issues with the IO subsystem. Much of this stuff has been heavily refactored, particularly with Hadoop 2.0, so the problems are at least much smaller than they used to be. Early Hadoop was not exactly a masterful work in terms of the code & algos. What was great about it was the community that was built around it.
- x0x0 12y agosee eg quantcast's experiences with why hadoop is slow on big jobs [1], the design of their sorter architecture [2], and some benchmark wanking [3] [1] https://www.quantcast.com/inside-quantcast/2012/10/letter-from-the-big-data-front-hadoops-big-scalability-challenge/?jump=engblog https://www.quantcast.com/inside-quantcast/2012/10/letter-fr... [2] https://www.quantcast.com/inside-quantcast/2012/10/quantsort-revolution-in-map-reduce-performance-and-operation/ https://www.quantcast.com/inside-quantcast/2012/10/quantsort... [3] https://www.quantcast.com/inside-quantcast/2013/12/petabyte-sort/ https://www.quantcast.com/inside-quantcast/2013/12/petabyte-...