6 ms·

This is not pyschology's replication crisis. Every field has replication problems, psychology is just one of the only fields taking it seriously.

acover 8y ago
Source? I don't hear about rampant replication problems in chemistry/physics. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- nordsieck 8y ago
  This[0] is a great video talking about problems with lots of studies in biology-adjacent fields including, but not limited to, psychology and nutrition. [0] John Ioannidis: "Reproducible Research: True or False?" https://www.youtube.com/watch?v=GPYzY9I78CI https://www.youtube.com/watch?v=GPYzY9I78CI
  - acover 8y ago
    Interesting! Sadly medicine has a replication crisis. I don't think chemistry or physics does.
- mcguire 8y ago
  I'll just leave a mention of the current state of theoretical physics here: https://www.nature.com/articles/d41586-018-05374-9 https://www.nature.com/articles/d41586-018-05374-9
  - tormeh 8y ago
    That's a criticism of a bunch of heuristics we've had for selecting promising theories. No popular theory has been proven wrong. It's just that we've been stuck for a really long time. And we need some kind of heuristics, otherwise we're just going to overfit to our data.
- schuetze 8y ago
  The replicability of the biomedical sciences is just as bad if not worse than in psychology. [1] http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... [2] https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on...
  - acover 8y ago
    Yes, that is covered in my wikipedia link and a comment below. There is a replication crisis in many fields. Not every field.
    - thaumasiotes 8y ago
      There is a replication crisis in every field that is not disciplined by a need to achieve externally-defined goals. Chemistry is in good shape because people apply it in order to e.g. create steel. The difference between a scientist and an engineer is that the scientist answers the question "if I do X, what will happen?", and the engineer answers the question "what do I do in order to get Y to happen?". But if there are no engineers working off of the results, the scientist is free to say anything. Sadly, as pointed out elsewhere in the subthread, this is a sufficient but not a necessary condition. Medicine has plenty of well-defined goals, but it doesn't know how to meet them. The cheap wisdom there is: if you want something badly enough, there will always be someone willing to sell it to you, whether or not they actually have it.
- iron0013 8y ago
  If you're interested in machine learning/deep learning/artificial intelligence, for example, I've got bad news about a lot of those papers that appear at NIPS...
  - Baeocystin 8y ago
    I've yet to see anything from the current boom be anything more than empirical curve fitting in fancy clothes. I've lived through enough AI winters to not be surprised. Still disappointed.
    - thanatropism 8y ago
      UMAP is pretty cool. It's like the realization of a dream that's been hanging around (when it was called "multidimensional scaling") since the mid-20th century.
izzydata 8y ago
I don't think mathematics has a replication crises.
- 3825782957 8y ago
  Many of our field's greatest contributions to date are due to the perpetual nagging of pedantic zealots who condemn the laxness of other mathematicians accepting aesthetic brevity as a permissible excuse for unspecified leaps in reasoning. The history of logic is a completely uninterrupted episode of crisis in replication, where the creative genius of intuition is targeted for termination as we insist upon rigorous definitions and disciplined mechanistic proof as the official workhorses while only regarding the art of surreal hypothesis divination and strongarmed notational analogies as just temporarily embarrassing stopgaps until we can tease out the procedure for automating the prodigies while managing the resultant verbosity of minutely incremental 'drudgery' through the factorization of symmetry/redundancy and begin the descent into madness of hierarchies of meta proof and possibilities of hypercomputation. Maybe this is out of envy against the more naturally gifted such as Ramanujan, or rather out of desire to retire to the beach someday without neglecting one's responsibility in the search for ultimate knowledge, but more likely just the practicality of desiring an objective reality check when the deepest dives into the abyss of specialization and abstraction render oneself completely lonely and isolated from the checks and balances of haphazard social consensus in peer review or even the _physical_ universe's fairly arbitrary and low-precision demands in constraining and verifying empirical modeling/prediction that more casual disciplines such as sociology or physics typically set as the highest caliber of evidence.
  - btrask 8y ago
    Pardon my aesthetic judgment, but this is a beautiful comment.

acover 8y ago

Source? I don't hear about rampant replication problems in chemistry/physics. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

nordsieck 8y ago
This[0] is a great video talking about problems with lots of studies in biology-adjacent fields including, but not limited to, psychology and nutrition. [0] John Ioannidis: "Reproducible Research: True or False?" https://www.youtube.com/watch?v=GPYzY9I78CI https://www.youtube.com/watch?v=GPYzY9I78CI
- acover 8y ago
  Interesting! Sadly medicine has a replication crisis. I don't think chemistry or physics does.
mcguire 8y ago
I'll just leave a mention of the current state of theoretical physics here: https://www.nature.com/articles/d41586-018-05374-9 https://www.nature.com/articles/d41586-018-05374-9
- tormeh 8y ago
  That's a criticism of a bunch of heuristics we've had for selecting promising theories. No popular theory has been proven wrong. It's just that we've been stuck for a really long time. And we need some kind of heuristics, otherwise we're just going to overfit to our data.
schuetze 8y ago
The replicability of the biomedical sciences is just as bad if not worse than in psychology. [1] http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... [2] https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on...
- acover 8y ago
  Yes, that is covered in my wikipedia link and a comment below. There is a replication crisis in many fields. Not every field.
  - thaumasiotes 8y ago
    There is a replication crisis in every field that is not disciplined by a need to achieve externally-defined goals. Chemistry is in good shape because people apply it in order to e.g. create steel. The difference between a scientist and an engineer is that the scientist answers the question "if I do X, what will happen?", and the engineer answers the question "what do I do in order to get Y to happen?". But if there are no engineers working off of the results, the scientist is free to say anything. Sadly, as pointed out elsewhere in the subthread, this is a sufficient but not a necessary condition. Medicine has plenty of well-defined goals, but it doesn't know how to meet them. The cheap wisdom there is: if you want something badly enough, there will always be someone willing to sell it to you, whether or not they actually have it.
iron0013 8y ago
If you're interested in machine learning/deep learning/artificial intelligence, for example, I've got bad news about a lot of those papers that appear at NIPS...
- Baeocystin 8y ago
  I've yet to see anything from the current boom be anything more than empirical curve fitting in fancy clothes. I've lived through enough AI winters to not be surprised. Still disappointed.
  - thanatropism 8y ago
    UMAP is pretty cool. It's like the realization of a dream that's been hanging around (when it was called "multidimensional scaling") since the mid-20th century.

izzydata 8y ago

I don't think mathematics has a replication crises.

3825782957 8y ago
Many of our field's greatest contributions to date are due to the perpetual nagging of pedantic zealots who condemn the laxness of other mathematicians accepting aesthetic brevity as a permissible excuse for unspecified leaps in reasoning. The history of logic is a completely uninterrupted episode of crisis in replication, where the creative genius of intuition is targeted for termination as we insist upon rigorous definitions and disciplined mechanistic proof as the official workhorses while only regarding the art of surreal hypothesis divination and strongarmed notational analogies as just temporarily embarrassing stopgaps until we can tease out the procedure for automating the prodigies while managing the resultant verbosity of minutely incremental 'drudgery' through the factorization of symmetry/redundancy and begin the descent into madness of hierarchies of meta proof and possibilities of hypercomputation. Maybe this is out of envy against the more naturally gifted such as Ramanujan, or rather out of desire to retire to the beach someday without neglecting one's responsibility in the search for ultimate knowledge, but more likely just the practicality of desiring an objective reality check when the deepest dives into the abyss of specialization and abstraction render oneself completely lonely and isolated from the checks and balances of haphazard social consensus in peer review or even the _physical_ universe's fairly arbitrary and low-precision demands in constraining and verifying empirical modeling/prediction that more casual disciplines such as sociology or physics typically set as the highest caliber of evidence.
- btrask 8y ago
  Pardon my aesthetic judgment, but this is a beautiful comment.