3 ms·
See the "Law of large numbers" - https://en.wikipedia.org/wiki/Law_of_large_numbers https://en.wikipedia.org/wiki/Law_of_large_numbers Saying you need more dat
by judge2020 8y ago
See the "Law of large numbers" - https://en.wikipedia.org/wiki/Law_of_large_numbers https://en.wikipedia.org/wiki/Law_of_large_numbers
Saying you need more data is like saying you need to flip a quarter 500 million times to get a better percentage estimate of heads vs. tails compares to 1 million coin flips. After a certain point, having more data only helps with identifying outliers and changes in behavior over time (when dealing with human/natural data).
- bunderbunder 8y agoI think, more to the point, is that more data really is not better data. Oftentimes it's worse -- the larger the volume of data you collect, the harder (and more expensive) it can be to manage sources of systematic bias in your data collection. One of the more extreme examples of this situation is Big Data, where everything is a convenience sample, and therefore nothing is particularly trustworthy. Combine it with standard statistical hypothesis testing, whose p-values, being designed for the size of experiment that one could realistically manage in a lab setting circa 1910, are as much a reflection of your sample size as of anything else, and you're at great risk of leading yourself into a tar pit.
- mr_toad 8y agoDepends what you’re looking for. If you’re trying to measure a very slight bias then more observations can help. Now imagine that you want to measure that bias across different types of coins and year of minting. Does the bias between coins of the same type vary more than across types? Curse of dimensionality starts to raise its ugly head.
- Scene_Cast2 8y agoYou're using low-dimensional intuition. If your data has a non-linear relationship between 1000 features (or more), then 500 million samples is quantifiably better than 1 million samples. If these are boolean features, you have 2^1000 possible data points. Your samples are extremely sparse. The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data.
- throwawayjava 8y ago> The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data. I'm not really sure what "low-dimensional intuition" means, but I pretty regularly build models that do not "assume nonlinear relationships between all parameters".
- jacques_chester 8y agoMy understanding is: It's a fancy way of saying "lots of variables", because a lot of these problems are converted into linear algebra representations (which can have geometric analogies, hence: dimensions).
- toufiqbarhamov 8y agoI think the Wikipedia article on the Curse covers it pretty well. https://en.m.wikipedia.org/wiki/Curse_of_dimensionality https://en.m.wikipedia.org/wiki/Curse_of_dimensionality The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data become sparse. This sparsity is problematic for any method that requires statistical significance. In order to obtain a statistically sound and reliable result, the amount of data needed to support the result often grows exponentially with the dimensionality. Also, organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data, however, all objects appear to be sparse and dissimilar in many ways, which prevents common data organization strategies from being efficient.
- tomnipotent 8y agoMeaning that you can run PCA and the first few components will explain a vast majority of the variance. Some things may have so many moving pieces that it's just not possible to grok how all the variables interact to get the results we see.
- gnulinux 8y agoAs the dimension d goes to infinity, if you sample integers from d-multivariate normal distribution, your samples will collect near the unit d-sphere; whereas in low dimensions they'd collect near the origin. This is because as d goes to infinity, your samples become more and more far apart. This is a topological intuition of why higher dimensions don't work like lower dimensions.
- gnulinux 8y agoI think this intuition is fundamentally wrong and misleading. This intuition really only works in low dimensions. It is not necessarily true that more data is better but you cannot blindly invoke the law of large numbers here. As someone here pointed out in high dim data, diversity is more important than numbers.