In these discussions it is mandatory to quote this paper: http://research.microsoft.com/pubs/163083/hotcbp12%20final.pdf http://research.microsoft.com/pubs/1630

In these discussions it is mandatory to quote this paper: http://research.microsoft.com/pubs/163083/hotcbp12%20final.pdf http://research.microsoft.com/pubs/163083/hotcbp12%20final.p...: "We completely agree that Hadoop on a cluster is the right solution for jobs where the input data is multi-terabyte or larger. However, in this position paper we ask if this is the right path for general purpose data analytics? Evidence suggests that many MapReduce-like jobs process relatively small input data sets (less than 14 GB). Memory has reached a GB/$ ratio such that it is now technically and financially feasible to have servers with 100s GB of DRAM. We therefore ask, should we be scaling by using single machines with very large memories rather than clusters? We conjecture that, in terms of hardware and programmer time, this may be a better option for the majority of data processing jobs." Their data is based on Hadoop jobs running at Yahoo, Facebook, and Microsoft -- companies most would agree do have real Big Data -- and they find the median job size is <14GB.