3 ms·
I appreciate the definition of Big Data as requiring >1 machine. However... Small Data, that which traditional researchers handle, is normally much, much small
by ronaldx 13y ago
I appreciate the definition of Big Data as requiring >1 machine.
However... Small Data, that which traditional researchers handle, is normally much, much smaller than that: perhaps 10-10000 data points (and most often on the small end of that). An experienced researcher can essentially can know everything about this data set, including its outliers and quirky points, and get a good sense of it by drawing out simple graphs.
There is clearly some disconnect between these two ideas: is that "Medium Data"?
I would accept a concept of "Big Data" as data that cannot easily be eyeballed to get a sense of what's going on, so 10000+ points would count (under some circumstances). Maybe the concept of "six sigma" is useful - enough data that you would reasonably expect a six sigma outlier.
Mathematically/statistically, the storage limit is not a particularly important milestone: the ideas and methods don't change once you reach this scale (except for potential parallelisation).
- gtrubetskoy 13y agoI was just going to post the ">1 machine" definition when I saw your comment. I think there are also at least two meanings of "Big Data". The more popular one is simply a trendy name for good old and boring "statistics", but with a twist that the data comes by way of the Internet, social media, all that. The second one (and a little closer to my heart) is what ">1 machine" means from a developer/sysadmin perspective. This is where the hadoops, hives, cassandras, etc. come into play, and it's A LOT to learn, even for seasoned developers. I think it's also a little intimidating for people who have become very comfortable with the typical rdbms stack. Parallel processing can be hard to understand, it's not something you can tinker with on your laptop over the weekend, and it's not surprising to hear all the "your big data thing is stupid" comments.