I disagree with the "big data" line in the sand being whether "a kid doing a paper run could afford your servers". Using spot instances on AWS, you can have a 1

I disagree with the "big data" line in the sand being whether "a kid doing a paper run could afford your servers". Using spot instances on AWS, you can have a 100 machine cluster with 1.5TB RAM for only $3 per hour. Working at Common Crawl, some of the coolest projects I've seen have been side projects or weekend projects by interested volunteers. WikiReverse[1] cost only $64USD to parse the metadata of all 3.6 billion pages (even cheaper if you avoided EMR fees) whilst Yelp extracted 748 million US phone numbers from 2 billion pages for $10.60USD. These days, big data (regardless of how you define the term) is within the reach of the kid with a paper route. [1]: https://wikireverse.org/ https://wikireverse.org/ [2]: http://engineeringblog.yelp.com/2015/03/analyzing-the-web-for-the-price-of-a-sandwich.html http://engineeringblog.yelp.com/2015/03/analyzing-the-web-fo...