4 ms·
The misconceptions about big data are similar to those surrounding the word science. Many people associate "science" with things: cells, microscopes, the inner
by hawkharris 13y ago
The misconceptions about big data are similar to those surrounding the word science.
Many people associate "science" with things: cells, microscopes, the inner workings of the body. But science isn't a set of things; it's a process, a method of thinking, that can be applied to any facet of life.
Big data is similar, in my opinion. It's not so much about the stuff — the size or diversity of a company's datasets. It has more to do with the types of observations you're making and the statistical methods involved.
This distinction is important for two reasons:
1. If Big Data is recognized as a process rather than a circumstance, businesses will be more deliberate in deciding whether to use the methods. They will weigh the benefits of, say, MapReduce against other approaches.
2. The idea that "Big Data" techniques have everything to do with size is somewhat misleading. A comprehensive query of a 50,000 user dataset can be more computationally expensive than a simple operation on a 100,000-record dataset.
- mxfh 13y agoIt's the misconception that measurable observations equal the real distribution of the underlying events. Even professional data people often get that wrong, and it's not strictly limited to big data. One of the most obvious examples was this one: A data set of all known meteorite landings[1] turns into "Every meteorite fall on earth mapped" [2] with looks like a world population maps sprinkled with some deserts known for their meteorite hunter tourism. The actual distribution can be theoretically described as a curve falling towards the poles.[3] While this example is pretty obvious, one could expect similar observation biases in other data sources. A danger lies where data analyst do not bother to investigate what their data actually represents and then go on to present their conclusions like it would be some kind of universal truth. [1]http://visualizing.org/datasets/meteorite-landings http://visualizing.org/datasets/meteorite-landings [2]http://www.theguardian.com/news/datablog/interactive/2013/feb/15/meteorite-fall-map http://www.theguardian.com/news/datablog/interactive/2013/fe... [3]http://articles.adsabs.harvard.edu//full/1964Metic...2..271H/0000276.000.html http://articles.adsabs.harvard.edu//full/1964Metic...2..271H... previous discussion of this: https://news.ycombinator.com/item?id=5240782 https://news.ycombinator.com/item?id=5240782
- greenyoda 13y agoYou have the same problem with historical global temperature data: weather stations tend to be in or near populated areas, which excludes oceans (70% of the earth's surface area) and huge, sparsely populated regions like the Arctic, the Antarctic, deserts, rain forests, remote mountain ranges like the Himalayas and Andes, etc.
- Varcht 13y agoAgreed, the buzzword 'Big Data' has nothing to do with the actual size of a given dataset except that it is about gathering as much data ( really metadata) as possible and finding novel ways to extract value from that data.