Read http://infolab.stanford.edu/~ullman/mmds.html http://infolab.stanford.edu/~ullman/mmds.html a few times and you'll know more about manipulating large data

Read http://infolab.stanford.edu/~ullman/mmds.html http://infolab.stanford.edu/~ullman/mmds.html a few times and you'll know more about manipulating large data sets than most people. If you're just looking for a quick example: imagine you want to parse 6,000 webserver logs (each being 100+GB) to determine the most popular pages, referrers and user agents. The map phase extracts all fields you want and the reduce phase counts all the extracted unique values. "mapreduce" is about processing at scale. You can always do the same thing on a smaller set of data locally (and usually with just sed, awk, sort, and uniq).