3 ms·

Is this study reproducible? If not it’s another example of the crisis in psychology: https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.

Is this study reproducible? If not it’s another example of the crisis in psychology: https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis

JackFr 1y ago
It seems unlikely. Some control groups might be nice. Send two more groups on a week long retreat, tell them some quack theory and see if they improve.
genewitch 1y ago
it's all science, not just "psychology." i'm working my way through [1] to try and internalize the ability to recognize the weasel. The part about a test that's 80% accurate for positive, and 9.6% false positive rate made intuitive sense to me, and yet something like "15% of doctors get it right." The excuses abound: You need a "lab", there's no money in replication, "science marches on[2]", and so on. It's just a cover for an alternate economy, at this point. I'm sorry if you're a scientist doing actual science and your entire foundation is being chipped away at by these heathens. [1] https://www.norvig.com/experiment-design.html posted here earlier this week. [2] https://tvtropes.org/pmwiki/pmwiki.php/Main/ScienceMarchesOn - "New discoveries make old theories obsolete and stories retroactively inaccurate."
- aYsY4dDQ2NrcNzA 1y ago
  I agree it’s a crisis, and I’ve wondered what we should do about this. Any thoughts?
  - genewitch 1y ago
    It'd take something like encouraging through monetization the replication of papers / experiments, with completely transparent methods (i.e. a video archive, or such). However i presuppose that we can also fix greed, payola (and other bribery), litigation of results that some don't like, and so on. I don't think those are solvable (in my lifetime, probably.) So an alternative would be for a non-profit with large funding to open laboratories around the planet and run replication experiments; this would need to be their only purpose. You'd have to figure out how to convince people to do the thankless work of proving or disproving others' experiments, designs, and so on. Even if you somehow could assign prestige for being the hardest-nosed re-researcher in the program... And even then, you'll still have people that believe outlandish, patently unbelievable things. You'll still be unable to be 100% sure that something is correct.
mrguyorama 1y ago
It was never even published. Also people vastly misunderstand the replication crisis. "Soft" sciences are harder to actually do science with. It's harder to measure things of humans vs an electron for example. But even in chemistry, there's a very high chance that if you take a random paper, even from the 60s say, and try to do it yourself you do not get the result from the paper, but that doesn't mean the paper is wrong, it can just as easily mean that the paper did not include some important environmental condition or procedure or whatever. NileRed and the guy from Explosions and Fire have both had difficulty reproducing random chemistry papers. Does that mean chemistry is this big lie and narrative propped up by lying academics? No, it means science is freaking hard. Every paper has a nonzero "failed replication" rate. Realms of science where it is harder to measure things and control the experiment have vastly higher expected rates of replication failure. People who do science every day understand this and work with it, and it is not a problem. Consider this; China has a HUGE replication problem with their science, with significantly more actual fraud than we have in the US. Has it hampered their advancement? Science is usually self correcting in that, if you put out fraud, anyone trying to build off your work will fail. This is what happened with several past famous fraudsters in science. Scientists do not even consider "peer review" to be a real quality signal of any sort. It basically wasn't a formal system until the 60s. They just read the paper.

Is this study reproducible? If not it’s another example of the crisis in psychology: https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis

JackFr 1y ago

It seems unlikely. Some control groups might be nice. Send two more groups on a week long retreat, tell them some quack theory and see if they improve.

genewitch 1y ago

it's all science, not just "psychology." i'm working my way through [1] to try and internalize the ability to recognize the weasel. The part about a test that's 80% accurate for positive, and 9.6% false positive rate made intuitive sense to me, and yet something like "15% of doctors get it right." The excuses abound: You need a "lab", there's no money in replication, "science marches on[2]", and so on. It's just a cover for an alternate economy, at this point. I'm sorry if you're a scientist doing actual science and your entire foundation is being chipped away at by these heathens. [1] https://www.norvig.com/experiment-design.html posted here earlier this week. [2] https://tvtropes.org/pmwiki/pmwiki.php/Main/ScienceMarchesOn - "New discoveries make old theories obsolete and stories retroactively inaccurate."

aYsY4dDQ2NrcNzA 1y ago
I agree it’s a crisis, and I’ve wondered what we should do about this. Any thoughts?
- genewitch 1y ago
  It'd take something like encouraging through monetization the replication of papers / experiments, with completely transparent methods (i.e. a video archive, or such). However i presuppose that we can also fix greed, payola (and other bribery), litigation of results that some don't like, and so on. I don't think those are solvable (in my lifetime, probably.) So an alternative would be for a non-profit with large funding to open laboratories around the planet and run replication experiments; this would need to be their only purpose. You'd have to figure out how to convince people to do the thankless work of proving or disproving others' experiments, designs, and so on. Even if you somehow could assign prestige for being the hardest-nosed re-researcher in the program... And even then, you'll still have people that believe outlandish, patently unbelievable things. You'll still be unable to be 100% sure that something is correct.

mrguyorama 1y ago

It was never even published. Also people vastly misunderstand the replication crisis. "Soft" sciences are harder to actually do science with. It's harder to measure things of humans vs an electron for example. But even in chemistry, there's a very high chance that if you take a random paper, even from the 60s say, and try to do it yourself you do not get the result from the paper, but that doesn't mean the paper is wrong, it can just as easily mean that the paper did not include some important environmental condition or procedure or whatever. NileRed and the guy from Explosions and Fire have both had difficulty reproducing random chemistry papers. Does that mean chemistry is this big lie and narrative propped up by lying academics? No, it means science is freaking hard. Every paper has a nonzero "failed replication" rate. Realms of science where it is harder to measure things and control the experiment have vastly higher expected rates of replication failure. People who do science every day understand this and work with it, and it is not a problem. Consider this; China has a HUGE replication problem with their science, with significantly more actual fraud than we have in the US. Has it hampered their advancement? Science is usually self correcting in that, if you put out fraud, anyone trying to build off your work will fail. This is what happened with several past famous fraudsters in science. Scientists do not even consider "peer review" to be a real quality signal of any sort. It basically wasn't a formal system until the 60s. They just read the paper.