6 ms·

I'm not sure I really agree with this, because in science studies are biased towards aligning with the results of previous studies. This can be due to many fact

I'm not sure I really agree with this, because in science studies are biased towards aligning with the results of previous studies. This can be due to many factors: authors' expectations, journals like to publish positive over negative results, peer reviewers will look more critically (in an analytical sense) at work that disagrees with accepted research, etc.... It can be very hard for many reasons to stand up and say "this prior work is wrong".

soVeryTired 8y ago
I know someone who quit her PhD in part because she couldn't replicate someone else's experimental work on rat behaviour. If the replication crisis had been as well-known back then, she might have fared better.
SubiculumCode 8y ago
The argument is not that all is perfect in science. It is not. The argument is that strong replication for each study is not required for progress. Back to my analogy to the algorithms like Adaboost. Why ever aggregate over a a bunch of weak classifiers over a single strong one? Well, please correct me if I am wrong (machine learning is not my field), but the primary advantage is computational cost. Sometimes, for the same overall classification performance, training hundreds or thousands of weak classifiers requires fewer computations than training the strong classifier. To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice? Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.
- PeterisP 8y ago
  The ensemble advantage is completely the other way around - we might choose to use a large ensemble of classifiers because they can get slightly better results than the best single strong classifier we can make; and we might choose not to use an ensemble because of computational cost reasons, especially because you train a model once but infer forever (and likely on more limited hardware) and inference for a hundred classifiers takes a hundred times more computing power even if you skimped on training length. A good example is the Netflix Prize, where the best accuracy was achieved by a large ensemble, but the practical implementation afterwards chose a slightly less accurate non-ensemble approach for performance reasons. What you describe sometimes happens in a tradeoff between very different ML models (e.g. an ensemble of decision trees versus a single deep neural network has the properties you describe), but within any single paradigm (an ensemble of neural networks versus one NN with more training time) it's the other way around.
  - SubiculumCode 8y ago
    Always willing to learn more. I was going off of https://en.wikipedia.org/wiki/AdaBoost https://en.wikipedia.org/wiki/AdaBoost which mentions "Unlike neural networks and SVMs, the AdaBoost training process selects only those features known to improve the predictive power of the model, reducing dimensionality and potentially improving execution time as irrelevant features need not be computed." edit: When I read about ensemble theory, you receive support for the increased computational cost. Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model, so ensembles may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. Fast algorithms such as decision trees are commonly used in ensemble methods (for example Random Forest), although slower algorithms can benefit from ensemble techniques as well. But I found this interesting: Empirically, ensembles tend to yield better results when there is a significant diversity among the models.[4][5] Many ensemble methods, therefore, seek to promote diversity among the models they combine.[6][7] Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees).[8] Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity.[9] ..which under my analogy would seem to suggest that exact replication is less productive than having a diversity of study designs.
    - AstralStorm 8y ago
      Not quite. If you read the latter paragraph closely, you would note that an ensemble of stronger models performs better than a multitude of dumb models. Then there is a problem that an ensemble of biased classifiers (mostly in the same ditection in this case - positivity bias) will magnify the bias. This is a reason why metaanalyses have to get at the actual raw data to pool and analyse as well as correct for multiple biases. Even then the process is not perfect.

soVeryTired 8y ago

I know someone who quit her PhD in part because she couldn't replicate someone else's experimental work on rat behaviour. If the replication crisis had been as well-known back then, she might have fared better.

SubiculumCode 8y ago

The argument is not that all is perfect in science. It is not. The argument is that strong replication for each study is not required for progress. Back to my analogy to the algorithms like Adaboost. Why ever aggregate over a a bunch of weak classifiers over a single strong one? Well, please correct me if I am wrong (machine learning is not my field), but the primary advantage is computational cost. Sometimes, for the same overall classification performance, training hundreds or thousands of weak classifiers requires fewer computations than training the strong classifier. To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice? Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.

PeterisP 8y ago
The ensemble advantage is completely the other way around - we might choose to use a large ensemble of classifiers because they can get slightly better results than the best single strong classifier we can make; and we might choose not to use an ensemble because of computational cost reasons, especially because you train a model once but infer forever (and likely on more limited hardware) and inference for a hundred classifiers takes a hundred times more computing power even if you skimped on training length. A good example is the Netflix Prize, where the best accuracy was achieved by a large ensemble, but the practical implementation afterwards chose a slightly less accurate non-ensemble approach for performance reasons. What you describe sometimes happens in a tradeoff between very different ML models (e.g. an ensemble of decision trees versus a single deep neural network has the properties you describe), but within any single paradigm (an ensemble of neural networks versus one NN with more training time) it's the other way around.
- SubiculumCode 8y ago
  Always willing to learn more. I was going off of https://en.wikipedia.org/wiki/AdaBoost https://en.wikipedia.org/wiki/AdaBoost which mentions "Unlike neural networks and SVMs, the AdaBoost training process selects only those features known to improve the predictive power of the model, reducing dimensionality and potentially improving execution time as irrelevant features need not be computed." edit: When I read about ensemble theory, you receive support for the increased computational cost. Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model, so ensembles may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. Fast algorithms such as decision trees are commonly used in ensemble methods (for example Random Forest), although slower algorithms can benefit from ensemble techniques as well. But I found this interesting: Empirically, ensembles tend to yield better results when there is a significant diversity among the models.[4][5] Many ensemble methods, therefore, seek to promote diversity among the models they combine.[6][7] Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees).[8] Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity.[9] ..which under my analogy would seem to suggest that exact replication is less productive than having a diversity of study designs.
  - AstralStorm 8y ago
    Not quite. If you read the latter paragraph closely, you would note that an ensemble of stronger models performs better than a multitude of dumb models. Then there is a problem that an ensemble of biased classifiers (mostly in the same ditection in this case - positivity bias) will magnify the bias. This is a reason why metaanalyses have to get at the actual raw data to pool and analyse as well as correct for multiple biases. Even then the process is not perfect.