5 ms·

How much of this is the result of outright fraud? I know Hanlon's razor and all, but I find it extremely difficult to believe that virtually everybody in this f

How much of this is the result of outright fraud? I know Hanlon's razor and all, but I find it extremely difficult to believe that virtually everybody in this field is incompetent. It seems like the majority of these highly intelligent people, despite spending a decade in training, were essentially statistically illiterate, even though statistics are absolutely central to their field. That's not a catastrophe in itself, you can always hit up the "real" statistician two floors down...but it seems like they didn't want to bother with that. Did they overestimate their ability or did they know they were serving up shitty research? The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky)...I'm not sure how to interpret this. Did they get lucky and stumble upon the one area of psychological research that had a "high yield"? Did they implement a superior research methodology that nobody could/wanted to imitate?

jMyles 11y ago
I've always felt like Hanlon's razor needs some kind of clause to account for corporatacracy. It's difficult to figure out how to attribute enforced monopolies, which take a great deal of planning and political cunning to establish, to incompetence.
mettamage 11y ago
Where I study psychology, I'm noticing that the pace of learning statistics is slower compared to statistics classes in my CS program. The fact that I'm doing multiple programs is a hint in itself. For example, we receive lessons about normality assumptions but do not receive lessons about non-parametric statistics. Furthermore, whenever I gave critique about these kind of things they were always shoved away as if they were a minor inconvenience.
DanBC 11y ago
Have a look at this what this Harvard neuroscientist says about failed replication. It's pretty scary. http://web.archive.org/web/20150115080542/http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm http://web.archive.org/web/20150115080542/http://wjh.harvard...
- bobcostas55 11y ago
  Well that makes me think the fraud angle is fairly likely...
  - DanBC 11y ago
    It made me think of incompetence, but I guess fraud fits too. The rejection of scientific method was what scared me.
  - analog31 11y ago
    I don't know about neuroscience in particular, but most professions have a code of ethics that forbids you from working in an area where you're not professionally qualified. This specifically addresses the gray area between incompetence and fraud.
- sambe 11y ago
  I find this scary because the author seems to attribute negative failure of replication to incompetence in a way that he doesn't treat the original success, and essentially ends up rejecting all failed replications - how many good results are sufficient vs how many bad results? Is that what you meant?
  - DanBC 11y ago
    Yes. It's a scary rejection of the scientific method.
    - conistonwater 11y ago
      How did that guy become a professor of psychology at Harvard? (The archived link itself is dead, but he's listed as a PI here: http://www.wjh.harvard.edu/~scanlab/people.html http://www.wjh.harvard.edu/~scanlab/people.html)
- astazangasta 11y ago
  If people cannot replicate your methods the fault is yours, period. The claim that it is "too hard" is bullshit. Write better methods. Science is nothing without replication. There are plenty of people writing vague descriptions of their methods to protect their status. These people are shit scientists.
mhuffman 11y ago
Not to try to derail the conversation, but I have noticed a very poor understanding of statistics and experimentation in all of the "soft" sciences, in particular, the social sciences. It is not uncommon to see published and "peer reviewed" work that makes sweeping claims off a tiny amount of self-reported data with nothing else to back it up. I have seen plenty of stuff that people on the "hard" sciences side of campuses would be embarrassed to even take responsibility for ... not to say they don't try to slip one by every now and again, but it seems systemic in the "soft" sciences.
- KingMob 11y ago
  This is endemic to all the sciences, not just the social ones. Biology has just as poor (if not worse) an understanding of stats, and even worse pressures in terms of the money at stake (medicine, pharmaceuticals). Maybe physics, and to some extent, chemistry, are immune, I'm less familiar with them. The issue is less stats, and more the pressure to publish, file drawer effects, poor career prospects for replication work relative to original research, fishing without multiple comparisons correction, etc. etc. etc. All of which combines to result in a lot of scientists looking the other way because the rules for stats and the rules for tenure aren't in alignment. I saw all this first-hand when I was a grad student in cognitive neuroscience. Don't for a second believe this is limited to psychology. If anything, psychology is being more honest and grappling with these issues first.
contravariant 11y ago
It doesn't require any malicious intent. An alternative possibility is that the things they were researching just weren't likely in the first place. If an effect that only has a 5% chance of being true in the first place, then even if you confirm it with a 95% confidence the chance that it is a fluke could still be as high as 50%. This can be made worse if you test several different hypothesis, only report successful experiments, and don't replicate them.
TeMPOraL 11y ago
When I look at the people I know personally who study psychology, sociology and related fields and see what they learn, I don't have any problems with believing that virtually everybody in the field is incompetent. It starts with the little things. I remember bringing up on HN the bullshit surveys most students do for their Master's thesis that are statistically worthless for the reasons said students don't understand. I've been told that hey, it's MSc, not a PhD, it doesn't have to be research-quality, etc. But hell, if people are willing to rely on bullshit methodologies for their Master's, when exactly do you think they'll learn the right way of doing things? Does the statistics god dispense additional levels of enlightenment when someone crosses the PhD threshold? I don't think so. And honestly, it's not that much better in the hard sciences as well, on the undergraduate level. Maybe they won't let you get away with bullshit stats, but they'll happily let you get away with fabricating results and tweaking data to fit the model. Studying today is about only one thing - do everything you can to pass the exam. Research ethics? Never heard of it.
analog31 11y ago
I know people in the life sciences who admit to frustration over the pressure to publish and the abuse of statistics. But I don't know if statistics is the entire problem. It seems to me that a feature of good work in the "hard" sciences is that a problem will be approached from a bunch of different angles, so that the result doesn't really depend solely on statistics. Synthetic chemists have to perform multiple orthogonal tests in order to announce a new compound, such as IR, NMR, X-Ray crystallography, and elemental analysis, on top of describing a mechanism for the compound's formation and performing reactions with the compound if possible. Traditionally, to prove discovery of a new element, it wasn't good enough to see a blip on a mass spec. You had to do chemistry with it. In my view a problem with data driven science is that you can do statistics without even knowing what you're actually studying. But if you don't know what you're studying, then you can't attack a problem from multiple angles because you don't even know if your tests are all measuring the same thing. You also have no tools to ask the hard questions about whether the results really make sense.
jrochkind1 11y ago
I don't think it's just this field. Statistics are routinely abused in nearly every field of science, alas. The OP mentions findings about how forcing pre-registration on clicnicaltrials.gov seems to change results. Those are, of course, medical research studies. A discussion of those findings regarding pre-registration was on HN 10 days ago. https://news.ycombinator.com/item?id=10080680 https://news.ycombinator.com/item?id=10080680 I do not think it is conscious 'fraud' exactly. Sloppy statistics have just become routinized in standard scientific research in many fields. And yes, that sloppiness almost always errs in the direction of making it easier to show positive results, of course. But I don't think the researchers realize they are doing something wrong exactly, it's just totally routinized.
anindyabd 11y ago
> The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky) This doesn't surprise me at all. Kahneman wrote a book, "Thinking, Fast and Slow", in which he spends a long time talking about how most psychologists and social scientists do not understand the basics of statistics and therefore end up with flawed conclusions.

jMyles 11y ago

I've always felt like Hanlon's razor needs some kind of clause to account for corporatacracy. It's difficult to figure out how to attribute enforced monopolies, which take a great deal of planning and political cunning to establish, to incompetence.

mettamage 11y ago

Where I study psychology, I'm noticing that the pace of learning statistics is slower compared to statistics classes in my CS program. The fact that I'm doing multiple programs is a hint in itself. For example, we receive lessons about normality assumptions but do not receive lessons about non-parametric statistics. Furthermore, whenever I gave critique about these kind of things they were always shoved away as if they were a minor inconvenience.

DanBC 11y ago

Have a look at this what this Harvard neuroscientist says about failed replication. It's pretty scary. http://web.archive.org/web/20150115080542/http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm http://web.archive.org/web/20150115080542/http://wjh.harvard...

bobcostas55 11y ago
Well that makes me think the fraud angle is fairly likely...
- DanBC 11y ago
  It made me think of incompetence, but I guess fraud fits too. The rejection of scientific method was what scared me.
- analog31 11y ago
  I don't know about neuroscience in particular, but most professions have a code of ethics that forbids you from working in an area where you're not professionally qualified. This specifically addresses the gray area between incompetence and fraud.
sambe 11y ago
I find this scary because the author seems to attribute negative failure of replication to incompetence in a way that he doesn't treat the original success, and essentially ends up rejecting all failed replications - how many good results are sufficient vs how many bad results? Is that what you meant?
- DanBC 11y ago
  Yes. It's a scary rejection of the scientific method.
  - conistonwater 11y ago
    How did that guy become a professor of psychology at Harvard? (The archived link itself is dead, but he's listed as a PI here: http://www.wjh.harvard.edu/~scanlab/people.html http://www.wjh.harvard.edu/~scanlab/people.html)
astazangasta 11y ago
If people cannot replicate your methods the fault is yours, period. The claim that it is "too hard" is bullshit. Write better methods. Science is nothing without replication. There are plenty of people writing vague descriptions of their methods to protect their status. These people are shit scientists.

mhuffman 11y ago

Not to try to derail the conversation, but I have noticed a very poor understanding of statistics and experimentation in all of the "soft" sciences, in particular, the social sciences. It is not uncommon to see published and "peer reviewed" work that makes sweeping claims off a tiny amount of self-reported data with nothing else to back it up. I have seen plenty of stuff that people on the "hard" sciences side of campuses would be embarrassed to even take responsibility for ... not to say they don't try to slip one by every now and again, but it seems systemic in the "soft" sciences.

KingMob 11y ago
This is endemic to all the sciences, not just the social ones. Biology has just as poor (if not worse) an understanding of stats, and even worse pressures in terms of the money at stake (medicine, pharmaceuticals). Maybe physics, and to some extent, chemistry, are immune, I'm less familiar with them. The issue is less stats, and more the pressure to publish, file drawer effects, poor career prospects for replication work relative to original research, fishing without multiple comparisons correction, etc. etc. etc. All of which combines to result in a lot of scientists looking the other way because the rules for stats and the rules for tenure aren't in alignment. I saw all this first-hand when I was a grad student in cognitive neuroscience. Don't for a second believe this is limited to psychology. If anything, psychology is being more honest and grappling with these issues first.

contravariant 11y ago

It doesn't require any malicious intent. An alternative possibility is that the things they were researching just weren't likely in the first place. If an effect that only has a 5% chance of being true in the first place, then even if you confirm it with a 95% confidence the chance that it is a fluke could still be as high as 50%. This can be made worse if you test several different hypothesis, only report successful experiments, and don't replicate them.

TeMPOraL 11y ago

When I look at the people I know personally who study psychology, sociology and related fields and see what they learn, I don't have any problems with believing that virtually everybody in the field is incompetent. It starts with the little things. I remember bringing up on HN the bullshit surveys most students do for their Master's thesis that are statistically worthless for the reasons said students don't understand. I've been told that hey, it's MSc, not a PhD, it doesn't have to be research-quality, etc. But hell, if people are willing to rely on bullshit methodologies for their Master's, when exactly do you think they'll learn the right way of doing things? Does the statistics god dispense additional levels of enlightenment when someone crosses the PhD threshold? I don't think so. And honestly, it's not that much better in the hard sciences as well, on the undergraduate level. Maybe they won't let you get away with bullshit stats, but they'll happily let you get away with fabricating results and tweaking data to fit the model. Studying today is about only one thing - do everything you can to pass the exam. Research ethics? Never heard of it.

analog31 11y ago

I know people in the life sciences who admit to frustration over the pressure to publish and the abuse of statistics. But I don't know if statistics is the entire problem. It seems to me that a feature of good work in the "hard" sciences is that a problem will be approached from a bunch of different angles, so that the result doesn't really depend solely on statistics. Synthetic chemists have to perform multiple orthogonal tests in order to announce a new compound, such as IR, NMR, X-Ray crystallography, and elemental analysis, on top of describing a mechanism for the compound's formation and performing reactions with the compound if possible. Traditionally, to prove discovery of a new element, it wasn't good enough to see a blip on a mass spec. You had to do chemistry with it. In my view a problem with data driven science is that you can do statistics without even knowing what you're actually studying. But if you don't know what you're studying, then you can't attack a problem from multiple angles because you don't even know if your tests are all measuring the same thing. You also have no tools to ask the hard questions about whether the results really make sense.

jrochkind1 11y ago

I don't think it's just this field. Statistics are routinely abused in nearly every field of science, alas. The OP mentions findings about how forcing pre-registration on clicnicaltrials.gov seems to change results. Those are, of course, medical research studies. A discussion of those findings regarding pre-registration was on HN 10 days ago. https://news.ycombinator.com/item?id=10080680 https://news.ycombinator.com/item?id=10080680 I do not think it is conscious 'fraud' exactly. Sloppy statistics have just become routinized in standard scientific research in many fields. And yes, that sloppiness almost always errs in the direction of making it easier to show positive results, of course. But I don't think the researchers realize they are doing something wrong exactly, it's just totally routinized.

anindyabd 11y ago

> The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky) This doesn't surprise me at all. Kahneman wrote a book, "Thinking, Fast and Slow", in which he spends a long time talking about how most psychologists and social scientists do not understand the basics of statistics and therefore end up with flawed conclusions.