5 ms·

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual arti

by qnsi 5y ago

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility crisis is real and it sucks for all the people that read individual books and don't know about reproducibility. I feel people that write those books should also put themself into higher standards and stop writing narrative books based on individual nonreplicated studies. You can read about reproducibility crisis here: https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/ https://www.theatlantic.com/science/archive/2018/11/psycholo...

dalbasal 5y ago
I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the scientific method where they don't work well. Say you are studying the effects of a breakfast program, 2 hr recess or push-up regiment on high schoolers. The study is well constructed, faithfully conducted, etc. You find interesting, potentially actionable ways of reducing obesity, improving psych health or something. Do/Should we even expect such a study to replicate in 1990s Tokyo, 2000s Miami, School A, School B, etc. Does that mean this study should not be conducted? Does it mean we can't generate knowledge and understanding of things this way? Is there any daylight between "anecdote" and a result that qualifies for publication in Physics? A lot of articles published in many fields feel like they are being hammered into a Popperian mold they don't fit into. Everyone knows it, but no one is willing to say "this method doesn't work here. we're actually doing a different thing." Instead, they sort of eye roll the article into the frame and let readers unscramble it for themselves.
- rwbhn 5y ago
  If the results are not replicable, how can they be actionable?
  - ethanbond 5y ago
    Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve changed a parameter that happens to be important: now find it!
    - jtbayly 5y ago
      Or it means there was fraud, which is why they are ultimately not able to be separated.
    - DanielBMarkham 5y ago
      Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency) I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before. Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place. Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.
stewbrew 5y ago
It is wrong to believe just because a result cannot be replicated the original result is wrong or fraudulent. It can be totally normal and is to be expected that a replication study fails to yield the same result just out of chance etc. Sometimes things are even more complicated than you already believe them to be. And it may take more than just a replication study to say a study is wrong or fraudulent.
- beagle3 5y ago
  It is wrong if it doesn’t replicate; it might not be fraudulent. But it is not right. If it doesn’t replicate, it should be relegated to “less than observational” level evidence. Why would you believe that the assumed intervention had the claimed effect, if it doesn’t have it when you try again?
  - stewbrew 5y ago
    In the present case, replication didn't fail because of statistics but because the experiment was impossible to reproduce. If it fails to replicate certain statistical outcomes, things are more complicated.
- goatlover 5y ago
  It means the original result is an anomaly that doesn’t fit into a scientific understanding. Maybe there was an error in the experimental setup, or some unknown factor was at play, or the situation was unique. Whatever the case, nothing scientific can be determined if it can’t be replicated. It’s akin to the WOW SETI signal.
gexla 5y ago
Another issue is books written by journalists blowing up ideas they find interesting from research and the pulling things out of their ass. Look at emotional intelligence (EQ) as an example. One of the early books on the subject written by a journalist which was based on a shaky foundation and then added a bunch more sugar. It might as well be a self-help movement which could spawn coaches, authors courses and certifications.

by qnsi 5y ago

dalbasal 5y ago

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the scientific method where they don't work well. Say you are studying the effects of a breakfast program, 2 hr recess or push-up regiment on high schoolers. The study is well constructed, faithfully conducted, etc. You find interesting, potentially actionable ways of reducing obesity, improving psych health or something. Do/Should we even expect such a study to replicate in 1990s Tokyo, 2000s Miami, School A, School B, etc. Does that mean this study should not be conducted? Does it mean we can't generate knowledge and understanding of things this way? Is there any daylight between "anecdote" and a result that qualifies for publication in Physics? A lot of articles published in many fields feel like they are being hammered into a Popperian mold they don't fit into. Everyone knows it, but no one is willing to say "this method doesn't work here. we're actually doing a different thing." Instead, they sort of eye roll the article into the frame and let readers unscramble it for themselves.

rwbhn 5y ago
If the results are not replicable, how can they be actionable?
- ethanbond 5y ago
  Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve changed a parameter that happens to be important: now find it!
  - jtbayly 5y ago
    Or it means there was fraud, which is why they are ultimately not able to be separated.
  - DanielBMarkham 5y ago
    Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency) I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before. Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place. Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.

stewbrew 5y ago

It is wrong to believe just because a result cannot be replicated the original result is wrong or fraudulent. It can be totally normal and is to be expected that a replication study fails to yield the same result just out of chance etc. Sometimes things are even more complicated than you already believe them to be. And it may take more than just a replication study to say a study is wrong or fraudulent.

beagle3 5y ago
It is wrong if it doesn’t replicate; it might not be fraudulent. But it is not right. If it doesn’t replicate, it should be relegated to “less than observational” level evidence. Why would you believe that the assumed intervention had the claimed effect, if it doesn’t have it when you try again?
- stewbrew 5y ago
  In the present case, replication didn't fail because of statistics but because the experiment was impossible to reproduce. If it fails to replicate certain statistical outcomes, things are more complicated.
goatlover 5y ago
It means the original result is an anomaly that doesn’t fit into a scientific understanding. Maybe there was an error in the experimental setup, or some unknown factor was at play, or the situation was unique. Whatever the case, nothing scientific can be determined if it can’t be replicated. It’s akin to the WOW SETI signal.

gexla 5y ago

Another issue is books written by journalists blowing up ideas they find interesting from research and the pulling things out of their ass. Look at emotional intelligence (EQ) as an example. One of the early books on the subject written by a journalist which was based on a shaky foundation and then added a bunch more sugar. It might as well be a self-help movement which could spawn coaches, authors courses and certifications.