4 ms·

ML is not causing a reproducibility crisis, it just exposes one that is already there. > If we had an additional dataset would we see the same scientific disco

ML is not causing a reproducibility crisis, it just exposes one that is already there. > If we had an additional dataset would we see the same scientific discovery or principle on the same dataset? The same holds true for traditional science based on traditional statistics. It just seems that traditional datasets are under less scrutiny of reproducibility and are taken more easily at face value.

rocqua 8y ago
A specific issue with machine learning is overfitting and non-interpretability. The first means it is possible to get results that don't generalize (even if they survive cross validation). The second means it is a lot harder to detect use of correlations that cannot possibly be causal.
- a_bonobo 8y ago
  One of the issues with problems of reproducibility in the social sciences/psychology is that early studies usually choose WEIRD (white/educated from a rich/industrialised/democractic country) subjects, which are often uni students who are very different from the rest of the world. One article: https://slate.com/technology/2013/05/weird-psychology-social-science-researchers-rely-too-much-on-western-college-students.html https://slate.com/technology/2013/05/weird-psychology-social... Could you not interpret this, in the widest sense, as overfitting interpretations to a 'weird' dataset, with the results that do not generalize, even though the stats (in this case, likely t-test) say everything is fine? In which case overfitting isn't a ML-only problem?
  - rocqua 8y ago
    Overfitting is about reading too much in a properly sampled data set from any distribution. It comes about when your model has so many parameters it can be made to fit anything. Regularization keeps it in check, but it remains an issue. The example you cite has an issue earlier in the chain. Here, we are dealing with a biased sample from some distrubution. The results from that won't generalize to the unbiased distrubution, no matter how great your model is.

rocqua 8y ago

A specific issue with machine learning is overfitting and non-interpretability. The first means it is possible to get results that don't generalize (even if they survive cross validation). The second means it is a lot harder to detect use of correlations that cannot possibly be causal.

a_bonobo 8y ago
One of the issues with problems of reproducibility in the social sciences/psychology is that early studies usually choose WEIRD (white/educated from a rich/industrialised/democractic country) subjects, which are often uni students who are very different from the rest of the world. One article: https://slate.com/technology/2013/05/weird-psychology-social-science-researchers-rely-too-much-on-western-college-students.html https://slate.com/technology/2013/05/weird-psychology-social... Could you not interpret this, in the widest sense, as overfitting interpretations to a 'weird' dataset, with the results that do not generalize, even though the stats (in this case, likely t-test) say everything is fine? In which case overfitting isn't a ML-only problem?
- rocqua 8y ago
  Overfitting is about reading too much in a properly sampled data set from any distribution. It comes about when your model has so many parameters it can be made to fit anything. Regularization keeps it in check, but it remains an issue. The example you cite has an issue earlier in the chain. Here, we are dealing with a biased sample from some distrubution. The results from that won't generalize to the unbiased distrubution, no matter how great your model is.