4 ms·

The paper is talking about a closely related, but rather different topic than the video (relying on your summary, I did not watch it). The paper is mostly conc

The paper is talking about a closely related, but rather different topic than the video (relying on your summary, I did not watch it). The paper is mostly concerned with model misspecification, in the sense that your model is not actually capable of describing reality (no matter what parameters you set it to). The first example uses the OLS model, e.g. Y = βX + ε where the implicit assumption is that your observations (Y) are purely a linear function of your input(s) (X) plus a independent, equal variance, normally distributed error. Let's say your data really does fit a straight line, but the later data points you get are really high variance. Intuitively, including the later data points will make your estimated parameters more noisy than if we had simply thrown them out. So in this sense, 'more data is worse', because your model assumes all the data points are the same, but the later points you collected are actually worse (noisier). There are other more obvious ways this happen - for example, if you assume that wealth is normally distributed, then try to estimate the distribution's parameters, then any small sample + any billionaire will get you nonsense — your model will be very intolerant of sampling noise compared to a better model. The video's observation only deals with data: you might prefer quality over quantity. E.g. compare a wider cone centered at zero (the true average) vs. a narrow cone that's centered away from zero - depending on what we care about, we may prefer some (hopefully uncorrelated) noise in our model rather than learning some systematic untruth. So these are two different ways more data can be counterproductive: either your model is biased in a way that more data can be hurtful, or your sampling procedure for larger sets is increasingly biased. Note that older stats/econometric techniques are more vulnerable to model misspecification because they have many fewer parameters than, say, a DNN; conversely, more modern ML techniques are able to closely replicate input data. This helps explain the difference in focus.

charlysl 7y ago
Thank you for your patient explanation, it seems to me then, if I understood, that the paper isn't relevant to what was being explained in the lecture, data acquisition/sampling, but rather to a later stage in the Data Science lifecycle, prediction/modeling.
em500 7y ago
> Let's say your data really does fit a straight line, but the later data points you get are really high variance. Intuitively, including the later data points will make your estimated parameters more noisy than if we had simply thrown them out. > So in this sense, 'more data is worse', because your model assumes all the data points are the same, but the later points you collected are actually worse (noisier). The first example in the paper makes it less-than-obvious by writing in formula form that the error noise variance increases with time, but what's described is essentially this picture: https://en.wikipedia.org/wiki/Heteroscedasticity#/media/File:Hsked_residual_compare.svg https://en.wikipedia.org/wiki/Heteroscedasticity#/media/File... So it's intuitively easy to grok that if the more recent variances are bad enough it's probably better not to use them at all (or to properly downweight them like with MLE or GLS). The described result is a bit more subtle though. The classical results are that with heteroskedasticity, OLS is not an efficient estimator [1] (you could do better with for example GLS/WLS), but it's still unbiased [2] (the estimated beta's are centered around the true beta's) and consistent [3] (the probability limit of the OLS beta is the true beta by the Law of Large Numbers). These are what the paper calls "robustness" of OLS in the first example. The consistency of OLS even with heteroskedastic errors is what makes it surprising that adding more (true!) observations can make your estimates worse. Intuitively most people expect consistent estimators to get better with more observations. [1] https://en.wikipedia.org/wiki/Efficient_estimator https://en.wikipedia.org/wiki/Efficient_estimator [2] https://en.wikipedia.org/wiki/Bias_of_an_estimator https://en.wikipedia.org/wiki/Bias_of_an_estimator [3] https://en.wikipedia.org/wiki/Consistent_estimator https://en.wikipedia.org/wiki/Consistent_estimator

charlysl 7y ago

Thank you for your patient explanation, it seems to me then, if I understood, that the paper isn't relevant to what was being explained in the lecture, data acquisition/sampling, but rather to a later stage in the Data Science lifecycle, prediction/modeling.

em500 7y ago

> Let's say your data really does fit a straight line, but the later data points you get are really high variance. Intuitively, including the later data points will make your estimated parameters more noisy than if we had simply thrown them out. > So in this sense, 'more data is worse', because your model assumes all the data points are the same, but the later points you collected are actually worse (noisier). The first example in the paper makes it less-than-obvious by writing in formula form that the error noise variance increases with time, but what's described is essentially this picture: https://en.wikipedia.org/wiki/Heteroscedasticity#/media/File:Hsked_residual_compare.svg https://en.wikipedia.org/wiki/Heteroscedasticity#/media/File... So it's intuitively easy to grok that if the more recent variances are bad enough it's probably better not to use them at all (or to properly downweight them like with MLE or GLS). The described result is a bit more subtle though. The classical results are that with heteroskedasticity, OLS is not an efficient estimator [1] (you could do better with for example GLS/WLS), but it's still unbiased [2] (the estimated beta's are centered around the true beta's) and consistent [3] (the probability limit of the OLS beta is the true beta by the Law of Large Numbers). These are what the paper calls "robustness" of OLS in the first example. The consistency of OLS even with heteroskedastic errors is what makes it surprising that adding more (true!) observations can make your estimates worse. Intuitively most people expect consistent estimators to get better with more observations. [1] https://en.wikipedia.org/wiki/Efficient_estimator https://en.wikipedia.org/wiki/Efficient_estimator [2] https://en.wikipedia.org/wiki/Bias_of_an_estimator https://en.wikipedia.org/wiki/Bias_of_an_estimator [3] https://en.wikipedia.org/wiki/Consistent_estimator https://en.wikipedia.org/wiki/Consistent_estimator