I did a bunch of work on managing/anticipating the power draw. ( https://news.ycombinator.com/item?id=13747481 https://news.ycombinator.com/item?id=13747481 ).

I did a bunch of work on managing/anticipating the power draw. ( https://news.ycombinator.com/item?id=13747481 https://news.ycombinator.com/item?id=13747481 ). Something nobody talks about is how grossly wasteful & inefficient Hadoop is. There's a bunch of cited papers (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.580.5685&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.58... ) that study power consumption of Hadoop nodes, but nothing on how to actually reduce the consumption. So here's something you can personally do to make a dent to this problem. Most of the Hadoop jobs that you write will involve some statistical summary over a dataset. Find the total, or the mean, median, 90th quantile, whatever. Writing a Hadoop map-reduce job is the single worst way to do this.Almost always, you can sample say a 1000 points, get a kernel density estimate via Parzen & then use a table. All quantiles, order statistics, functions of order stats...these can be hand computed for several univariate distributions & their location-scale families, and your real-life data can easily be bounded above below by these estimates. So you can get to > 95% accuracy just by hand-calculating. I can go into details if you like, but I suspect most of you already know how to do this.