6 ms·

I just want to add that almost all fields have serious problems with reproducibility. In my case: distributed machine learning... You may think otherwise becaus

I just want to add that almost all fields have serious problems with reproducibility. In my case: distributed machine learning... You may think otherwise because subjectivity is not involved, but many "top-tier" papers publish code and release parameters that, if runnable, do not reproduce the paper's results.)

mjburgess 6y ago
Err.. well social psych and ML don't constitute "almost all fields". If someone asked me for a list of where to find pseudo-science, it would be fairly small set: social psych, nutrition, AI.
- elcritch 6y ago
  Biological fields (cancer drugs, early stage drug research, etc) often don’t perform much better. In many other hard science fields, like say graphene production via CVP (as a friend worked in), researchers intentionally make their work almost impossible to replicate. The papers look fine but the authors quickly learn which parameters to exclude to make it almost impossible to reproduce. Otherwise they’d loose competitive advantage, and potential money from chip manufacturers, etc. Pretty much all people from toddlers to phd researchers are masters of game theory, be it conscious or not. Luckily there are also a decent number of scientists who try and counterbalance those urges, but I believe the current socioeconomics are pushing them out. Hopefully I’m wrong there.
  - mjburgess 6y ago
    RCTs are category-changing if your field uses RCTs, it's a science. Whether the results are occasionally, due to incompetence or stats, wrong. The issue with social psych et al. is that they are methodologically /incapable/ of producing reproducible results. It's association modelling of extremely complex domain's. Essentially superstitous.
    - elcritch 6y ago
      Agreed, the amount of superstitious in many fields is mind boggling.. Even then in many fields like comp. sci. the role of RCT's are pretty fuzzy (and many results like OO, FP are more superstitious than scientific). So I do consider much of C.S. to be more akin to philosophy than science. Still in say Physics the concept of RCT's don't apply directly, but the effect is the same underlying hypothesis -> experiment -> re-hypothesize -> experiment cycle. That cycle requires predictions of novel events which fulfills the same role, otherwise it's just data fitting. One "technology" that could strongly improve our ability to do proper science without RCT's or even improve RCT or hypothesis formulation would be properly integrating causation into statistics (particularly via Bayesian methods). See the work of Judea Pearl [1] for a digest of current work on causal maths. Then scientific fields can clearly define the degree of effect or causation for various results. Even in medicine/biology RCT's don't always confirm a theory as you can have overlapping effects where the tested variable does work but not for reasons in a given hypothesis. For example see Statins & cholesterol which work to a degree but largely because cholesterol is a necessary but not only ingredient to plaque formation and heart disease. RCT's proved early results of Statins, but new more powerful ones failed to materialize benefits. It turns out it's much more complicated (one random google result plucked for context [2]). I imagine those cases in medicine as similar to "microbenchmarking" in C.S. where it's easy to see that small scale microbenchmarks are true but fail to improve the overall system performance. Sometimes they make the overall system worse. Medicine in particular doesn't do enough real science. RCT's are only performed at a "microbenchmark" level but rarely performed for the overall system, partly due to cost and difficulty. Hence why I think something like causal maths could really help society formally and explicitly estimate causal properties from groups of RCT's. Sorry for the semi-rant, but it's been bouncing around my head for a few weeks now! 1: http://cdar.berkeley.edu/wp-content/uploads/2017/04/Lisa-Goldberg-reviews-The-Book-of-Why.pdf http://cdar.berkeley.edu/wp-content/uploads/2017/04/Lisa-Gol... 2: https://www.statnews.com/2019/04/03/statin-heart-disease-prevention-more-than-medicine/ https://www.statnews.com/2019/04/03/statin-heart-disease-pre...
- dependenttypes 6y ago
  Remember https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
nspattak 6y ago
I suspect, though I haven't tested as I can not, that this would be the case for MOST papers in every field that uses code to reach its results.
ternaus 6y ago
And many Machine Learning paper's do not have code at all. Or promise to release the code, but never do this, like: https://github.com/shepnerd/blindinpainting_vcnet https://github.com/shepnerd/blindinpainting_vcnet

mjburgess 6y ago

Err.. well social psych and ML don't constitute "almost all fields". If someone asked me for a list of where to find pseudo-science, it would be fairly small set: social psych, nutrition, AI.

elcritch 6y ago
Biological fields (cancer drugs, early stage drug research, etc) often don’t perform much better. In many other hard science fields, like say graphene production via CVP (as a friend worked in), researchers intentionally make their work almost impossible to replicate. The papers look fine but the authors quickly learn which parameters to exclude to make it almost impossible to reproduce. Otherwise they’d loose competitive advantage, and potential money from chip manufacturers, etc. Pretty much all people from toddlers to phd researchers are masters of game theory, be it conscious or not. Luckily there are also a decent number of scientists who try and counterbalance those urges, but I believe the current socioeconomics are pushing them out. Hopefully I’m wrong there.
- mjburgess 6y ago
  RCTs are category-changing if your field uses RCTs, it's a science. Whether the results are occasionally, due to incompetence or stats, wrong. The issue with social psych et al. is that they are methodologically /incapable/ of producing reproducible results. It's association modelling of extremely complex domain's. Essentially superstitous.
  - elcritch 6y ago
    Agreed, the amount of superstitious in many fields is mind boggling.. Even then in many fields like comp. sci. the role of RCT's are pretty fuzzy (and many results like OO, FP are more superstitious than scientific). So I do consider much of C.S. to be more akin to philosophy than science. Still in say Physics the concept of RCT's don't apply directly, but the effect is the same underlying hypothesis -> experiment -> re-hypothesize -> experiment cycle. That cycle requires predictions of novel events which fulfills the same role, otherwise it's just data fitting. One "technology" that could strongly improve our ability to do proper science without RCT's or even improve RCT or hypothesis formulation would be properly integrating causation into statistics (particularly via Bayesian methods). See the work of Judea Pearl [1] for a digest of current work on causal maths. Then scientific fields can clearly define the degree of effect or causation for various results. Even in medicine/biology RCT's don't always confirm a theory as you can have overlapping effects where the tested variable does work but not for reasons in a given hypothesis. For example see Statins & cholesterol which work to a degree but largely because cholesterol is a necessary but not only ingredient to plaque formation and heart disease. RCT's proved early results of Statins, but new more powerful ones failed to materialize benefits. It turns out it's much more complicated (one random google result plucked for context [2]). I imagine those cases in medicine as similar to "microbenchmarking" in C.S. where it's easy to see that small scale microbenchmarks are true but fail to improve the overall system performance. Sometimes they make the overall system worse. Medicine in particular doesn't do enough real science. RCT's are only performed at a "microbenchmark" level but rarely performed for the overall system, partly due to cost and difficulty. Hence why I think something like causal maths could really help society formally and explicitly estimate causal properties from groups of RCT's. Sorry for the semi-rant, but it's been bouncing around my head for a few weeks now! 1: http://cdar.berkeley.edu/wp-content/uploads/2017/04/Lisa-Goldberg-reviews-The-Book-of-Why.pdf http://cdar.berkeley.edu/wp-content/uploads/2017/04/Lisa-Gol... 2: https://www.statnews.com/2019/04/03/statin-heart-disease-prevention-more-than-medicine/ https://www.statnews.com/2019/04/03/statin-heart-disease-pre...
dependenttypes 6y ago
Remember https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...

nspattak 6y ago

I suspect, though I haven't tested as I can not, that this would be the case for MOST papers in every field that uses code to reach its results.

ternaus 6y ago

And many Machine Learning paper's do not have code at all. Or promise to release the code, but never do this, like: https://github.com/shepnerd/blindinpainting_vcnet https://github.com/shepnerd/blindinpainting_vcnet