5 ms·

You can rent a 128gb machine at Hetzner for $130 a month. I think we can all agree to draw a line in the sand and say if a kid doing a paper run could afford y

You can rent a 128gb machine at Hetzner for $130 a month. I think we can all agree to draw a line in the sand and say if a kid doing a paper run could afford your servers, you don't get to call it big data. (not to take away from the linked project, which looks technically awesome)

Smerity 11y ago
I disagree with the "big data" line in the sand being whether "a kid doing a paper run could afford your servers". Using spot instances on AWS, you can have a 100 machine cluster with 1.5TB RAM for only $3 per hour. Working at Common Crawl, some of the coolest projects I've seen have been side projects or weekend projects by interested volunteers. WikiReverse[1] cost only $64USD to parse the metadata of all 3.6 billion pages (even cheaper if you avoided EMR fees) whilst Yelp extracted 748 million US phone numbers from 2 billion pages for $10.60USD. These days, big data (regardless of how you define the term) is within the reach of the kid with a paper route. [1]: https://wikireverse.org/ https://wikireverse.org/ [2]: http://engineeringblog.yelp.com/2015/03/analyzing-the-web-for-the-price-of-a-sandwich.html http://engineeringblog.yelp.com/2015/03/analyzing-the-web-fo...
- deleted 11y ago
  [deleted]
riquito 11y ago
I appreciate the analogy but it's not like if the server were free big data would disappear, isn't it?
threeseed 11y ago
Big Data as the industry sees it is more than just the size of the data. It is about the nasty work of data validation, consolidation, ETL, unification e.g. single customer view, analytics etc. Some of these can be technically challenging even on multi gigabyte files and require a range of multi disciplinary techniques. And Big Data increasingly means taking old EDW concepts and reimagining them for a more real time world.
- rubyfan 11y ago
  No offense but what you just described is everything that is wrong with enterprise "Big Data" today. What many consultants are holding up as a big data panacea is just elbow grease and institutional fortitude... there is no free lunch.

Smerity 11y ago

I disagree with the "big data" line in the sand being whether "a kid doing a paper run could afford your servers". Using spot instances on AWS, you can have a 100 machine cluster with 1.5TB RAM for only $3 per hour. Working at Common Crawl, some of the coolest projects I've seen have been side projects or weekend projects by interested volunteers. WikiReverse[1] cost only $64USD to parse the metadata of all 3.6 billion pages (even cheaper if you avoided EMR fees) whilst Yelp extracted 748 million US phone numbers from 2 billion pages for $10.60USD. These days, big data (regardless of how you define the term) is within the reach of the kid with a paper route. [1]: https://wikireverse.org/ https://wikireverse.org/ [2]: http://engineeringblog.yelp.com/2015/03/analyzing-the-web-for-the-price-of-a-sandwich.html http://engineeringblog.yelp.com/2015/03/analyzing-the-web-fo...

deleted 11y ago
[deleted]

riquito 11y ago

I appreciate the analogy but it's not like if the server were free big data would disappear, isn't it?

threeseed 11y ago

Big Data as the industry sees it is more than just the size of the data. It is about the nasty work of data validation, consolidation, ETL, unification e.g. single customer view, analytics etc. Some of these can be technically challenging even on multi gigabyte files and require a range of multi disciplinary techniques. And Big Data increasingly means taking old EDW concepts and reimagining them for a more real time world.

rubyfan 11y ago
No offense but what you just described is everything that is wrong with enterprise "Big Data" today. What many consultants are holding up as a big data panacea is just elbow grease and institutional fortitude... there is no free lunch.