Yes, after doing "big data" at Google, my theory is that 95% of the time you need ONE MapReduce: stratified sampling. After that, you just do everything on a s

Yes, after doing "big data" at Google, my theory is that 95% of the time you need ONE MapReduce: stratified sampling. After that, you just do everything on a single beefy multicore box. Unfortunately most databases do not support efficient stratified sampling, although it would be trivial to add as far as I can see. It needs to be hooked into the same place as GROUP BY. ---- As an example, pick 1000 URLs from every domain, 1000 clicks to every URL, 1000 web pages in every language, 1000 people from every country, etc. All of these things have a "long tail", so you need stratified sampling over the categories, not just straight sampling. But if you have that, you can answer almost any question with 1 GB of data IMO. I think there are two kinds of big data: 1) Production Services like Maps and search, which have to touch every byte 2) Decision making. This can always be done with something like 1 GB of data, or often 1 MB or 10 KB. You never need 1 TB of data to make a decision about anything.