3 ms·
It's the misconception that measurable observations equal the real distribution of the underlying events. Even professional data people often get that wrong, an
by mxfh 13y ago
It's the misconception that measurable observations equal the real distribution of the underlying events. Even professional data people often get that wrong, and it's not strictly limited to big data.
One of the most obvious examples was this one:
A data set of all known meteorite landings[1] turns into
"Every meteorite fall on earth mapped" [2] with looks like a world population maps sprinkled with some deserts known for their meteorite hunter tourism.
The actual distribution can be theoretically described as a curve falling towards the poles.[3]
While this example is pretty obvious, one could expect similar observation biases in other data sources. A danger lies where data analyst do not bother to investigate what their data actually represents and then go on to present their conclusions like it would be some kind of universal truth.
[1]http://visualizing.org/datasets/meteorite-landings http://visualizing.org/datasets/meteorite-landings
[2]http://www.theguardian.com/news/datablog/interactive/2013/feb/15/meteorite-fall-map http://www.theguardian.com/news/datablog/interactive/2013/fe...
[3]http://articles.adsabs.harvard.edu//full/1964Metic...2..271H/0000276.000.html http://articles.adsabs.harvard.edu//full/1964Metic...2..271H...
previous discussion of this: https://news.ycombinator.com/item?id=5240782 https://news.ycombinator.com/item?id=5240782
- greenyoda 13y agoYou have the same problem with historical global temperature data: weather stations tend to be in or near populated areas, which excludes oceans (70% of the earth's surface area) and huge, sparsely populated regions like the Arctic, the Antarctic, deserts, rain forests, remote mountain ranges like the Himalayas and Andes, etc.