4 ms·

I know that psychology is particularly devasted by the replication crises but aren't branches of research affected as well?

EnFinlay 8y ago
Absolutely, and we should have a lot of skepticism to conclusions from those branches of research as well.
TangoTrotFox 8y ago
I think one of the problems "we" have is that bias confirmation is a hell of a drug. By "we" I mean individuals, researchers, companies, corporations, governments, everybody. When we see something that intuitively makes sense to us, especially one that confirms our biases - we tend to want to believe it, because we all believe what we think is true and so things that support what we believe must therefore be true, according to what we believe. Some word salad there, but I think that's reasonably clear. Like you mention the replication crisis should be really leaving most of all psychology results, not only past but also present, in serious doubt -- let alone anything contingent on those past results. But again the problem is that when new research comes out that confirms our biases, we don't expend the energy to challenge it. And it seems that many in science are more concerned with themselves than their science - something that the replication crisis provides a great deal of evidence for. People aren't putting out trash science by accident. They need to publish, and trash science is what gets published. And the replication crisis hasn't changed this. There seems to have been, at best, token efforts to try to more fully ensure the truthfulness of what's being published. So we continue to believe what we want to believe, with science increasingly falling victim to the act of starting at a conclusion and working your way backwards.
cultus 8y ago
Medicine is being hit almost as bad. In general, fields where a lot of studies have low statistical power(usually due to noisy data and/or small sample sizes) are the hardest hit. Statistical power is the chance that a significant result is found if the effect is indeed significant (one minus the chance of a false negative). Counter-intuitively, this increases the chance that a given significant result is invalid. Combine that with non-reporting of negative results and you basically have a huge pile of bullshit.
- ItsMe000001 8y ago
  I would extend to to "statistics based knowledge" in general. If you don't understand the mechanism it's not real knowledge. For example: You have a drug that 100% helps in a certain context (of what the problem is and the person's genetics plus maybe even epigenetics, and maybe even things like what they eat and what environment they are exposed to). However, clinical trial studies don't go very deep in separating different kinds of people. We just don't know enough, we don't know how to even measure most thing, and when we do it's extremely costly. And since we don't understand the mechanism - if we did we didn't have to go through the trials - they are needed because even we have _a_ mechanism we are not sure what else the drug does in the body, or about follow-up and higher order effects - we would not know what to look for anyway. Plus, without full understanding of the mechanism it's hard to combine the new "knowledge" with other knowledge. The experiment you gained the data from is very specific and results are hard to generalize. So the result is the drug will be a complete failure, because we are unable to tell which people would benefit. That's always the problem: When you don't have a very good understanding of the mechanism and all the consequences you have to be lucky that the population you study is more or less the correct one. You don't even know when you got the wrong one. If the drug - but same in any other field that uses statistics - fails for 95% of people you may still have a hit for a sub-population (it's not as easy as "it's the other 5%" of course). There are people working specifically on looking closer at some "failed drugs". They have been able to find a few "miracle drugs" that way. They only help certain people (they use genetic testing) but when they do they do great. But they failed their initial clinical trials big time. Over the years I have gotten much more skeptical of all statistics based "knowledge". Reading those studies always leaves a strange taste in my mouth. Something doesn't feel right. It does not taste like knowledge. I see the relevance given that it often is the only way to make practical progress of course.
  - cultus 8y ago
    >That's always the problem: When you don't have a very good understanding of the mechanism and all the consequences you have to be lucky that the population you study is more or less the correct one. You don't even know when you got the wrong one. If the drug - but same in any other field that uses statistics - fails for 95% of people you may still have a hit for a sub-population (it's not as easy as "it's the other 5%" of course). Good point, but this kind of subgroup analysis also runs into the multiple comparisons problem. If you test all kinds of subgroups (or just do a lot of any kind of tests), chances are good that you will run into false positives. In such scenarios, it becomes more probable than not that a given positive result is in fact false. Combine that with the base-rate fallacy (the good majority of drug trials yield null results, but this is not taken into account) and you are in really bad shape. There was a replication study a while ago that tried to replicate IIRC 50 or so "landmark" cancer studies, and only came up with significant results in 6 cases. Bayesian methods go a long way towards solving these issues, but there is no cure for low-power studies. They just can't tell you much, and will lead you heavily astray if you don't properly account for multiple comparisons, etc. >Over the years I have gotten much more skeptical of all statistics based "knowledge". Reading those studies always leaves a strange taste in my mouth. Something doesn't feel right. It does not taste like knowledge. I see the relevance given that it often is the only way to make practical progress of course. Clinical studies of new drugs have uncertainty. Thus any decision-making based on results must take this uncertainty into account. You can't get away from statistics here. The bad feeling in your mouth may be from the unintuitive and usually inappropriate use of P-values and null-hypothesis statistical tests. The vast majority of researchers are actually completely mistaken on what P-values even mean. Most think that they are "the chance of a false positive" or something similar, which is completely wrong. Bayesian methods help here, because they are the only valid way of combining past information with new information.
rossdavidh 8y ago
Personally, I wonder if it's not simply that psychology is the first field to investigate the issue seriously.

EnFinlay 8y ago

Absolutely, and we should have a lot of skepticism to conclusions from those branches of research as well.

TangoTrotFox 8y ago

I think one of the problems "we" have is that bias confirmation is a hell of a drug. By "we" I mean individuals, researchers, companies, corporations, governments, everybody. When we see something that intuitively makes sense to us, especially one that confirms our biases - we tend to want to believe it, because we all believe what we think is true and so things that support what we believe must therefore be true, according to what we believe. Some word salad there, but I think that's reasonably clear. Like you mention the replication crisis should be really leaving most of all psychology results, not only past but also present, in serious doubt -- let alone anything contingent on those past results. But again the problem is that when new research comes out that confirms our biases, we don't expend the energy to challenge it. And it seems that many in science are more concerned with themselves than their science - something that the replication crisis provides a great deal of evidence for. People aren't putting out trash science by accident. They need to publish, and trash science is what gets published. And the replication crisis hasn't changed this. There seems to have been, at best, token efforts to try to more fully ensure the truthfulness of what's being published. So we continue to believe what we want to believe, with science increasingly falling victim to the act of starting at a conclusion and working your way backwards.

cultus 8y ago

Medicine is being hit almost as bad. In general, fields where a lot of studies have low statistical power(usually due to noisy data and/or small sample sizes) are the hardest hit. Statistical power is the chance that a significant result is found if the effect is indeed significant (one minus the chance of a false negative). Counter-intuitively, this increases the chance that a given significant result is invalid. Combine that with non-reporting of negative results and you basically have a huge pile of bullshit.

ItsMe000001 8y ago
I would extend to to "statistics based knowledge" in general. If you don't understand the mechanism it's not real knowledge. For example: You have a drug that 100% helps in a certain context (of what the problem is and the person's genetics plus maybe even epigenetics, and maybe even things like what they eat and what environment they are exposed to). However, clinical trial studies don't go very deep in separating different kinds of people. We just don't know enough, we don't know how to even measure most thing, and when we do it's extremely costly. And since we don't understand the mechanism - if we did we didn't have to go through the trials - they are needed because even we have _a_ mechanism we are not sure what else the drug does in the body, or about follow-up and higher order effects - we would not know what to look for anyway. Plus, without full understanding of the mechanism it's hard to combine the new "knowledge" with other knowledge. The experiment you gained the data from is very specific and results are hard to generalize. So the result is the drug will be a complete failure, because we are unable to tell which people would benefit. That's always the problem: When you don't have a very good understanding of the mechanism and all the consequences you have to be lucky that the population you study is more or less the correct one. You don't even know when you got the wrong one. If the drug - but same in any other field that uses statistics - fails for 95% of people you may still have a hit for a sub-population (it's not as easy as "it's the other 5%" of course). There are people working specifically on looking closer at some "failed drugs". They have been able to find a few "miracle drugs" that way. They only help certain people (they use genetic testing) but when they do they do great. But they failed their initial clinical trials big time. Over the years I have gotten much more skeptical of all statistics based "knowledge". Reading those studies always leaves a strange taste in my mouth. Something doesn't feel right. It does not taste like knowledge. I see the relevance given that it often is the only way to make practical progress of course.
- cultus 8y ago
  >That's always the problem: When you don't have a very good understanding of the mechanism and all the consequences you have to be lucky that the population you study is more or less the correct one. You don't even know when you got the wrong one. If the drug - but same in any other field that uses statistics - fails for 95% of people you may still have a hit for a sub-population (it's not as easy as "it's the other 5%" of course). Good point, but this kind of subgroup analysis also runs into the multiple comparisons problem. If you test all kinds of subgroups (or just do a lot of any kind of tests), chances are good that you will run into false positives. In such scenarios, it becomes more probable than not that a given positive result is in fact false. Combine that with the base-rate fallacy (the good majority of drug trials yield null results, but this is not taken into account) and you are in really bad shape. There was a replication study a while ago that tried to replicate IIRC 50 or so "landmark" cancer studies, and only came up with significant results in 6 cases. Bayesian methods go a long way towards solving these issues, but there is no cure for low-power studies. They just can't tell you much, and will lead you heavily astray if you don't properly account for multiple comparisons, etc. >Over the years I have gotten much more skeptical of all statistics based "knowledge". Reading those studies always leaves a strange taste in my mouth. Something doesn't feel right. It does not taste like knowledge. I see the relevance given that it often is the only way to make practical progress of course. Clinical studies of new drugs have uncertainty. Thus any decision-making based on results must take this uncertainty into account. You can't get away from statistics here. The bad feeling in your mouth may be from the unintuitive and usually inappropriate use of P-values and null-hypothesis statistical tests. The vast majority of researchers are actually completely mistaken on what P-values even mean. Most think that they are "the chance of a false positive" or something similar, which is completely wrong. Bayesian methods help here, because they are the only valid way of combining past information with new information.

rossdavidh 8y ago

Personally, I wonder if it's not simply that psychology is the first field to investigate the issue seriously.