3 ms·
Generating Datasets with Identical Statistics Through Simulated Annealing (2017)
- dspillett 5y agoThis has been posted a few times and seemingly not got the attention it deserves, the first post back in 2017 is the only one with comments: https://news.ycombinator.com/item?id=14243950 https://news.ycombinator.com/item?id=14243950
- nomilk 5y agoWe learn that 13 very different looking datasets can all have identical X and Y means, X and Y standard deviations, and correlations. This is very insightful. What are the real world applications and/or implications?
- pfortuny 5y agoImplications: statistics only say very little, data visualization may help understanding. Hidden correlations are a thing. Hidden variabilities too.
- wodenokoto 5y agoMatt Parker of Numberphile fame, did an interview with the authors two years ago: https://youtu.be/iwzzv1biHv8 https://youtu.be/iwzzv1biHv8
- mjb 5y agoI absolutely love the The Datasaurus Dozen. Like Anscombe's Quartet, it reminds us that summary statistics are all misleading in one way or another, and there's no substitute for looking at our data from multiple angles, and often no substitute for just looking at our data with our eyes. For example, one thing that comes up really often is folks saying not to use the mean as a summary statistic, often because they claim it's too sensitive to outliers. Through the same lens, you could equally say that the median is not sensitive enough to outliers, and that order statistics in general (like percentiles) elide a lot of the structure of the underlying distribution that may really matter in practice. The point is that both of these view are correct, and that there is no summary statistic that captures everything about a distribution. They're tools, and you need to pick the right one. But if you're looking to understand your data holistically, all of them are going to mislead you in one way or another.