6 ms·

I think bad statistics education is just a facade that hides what is really happening. Now I don't have proof, but I think the majority of these "errors" are do

by cop359 15y ago

I think bad statistics education is just a facade that hides what is really happening. Now I don't have proof, but I think the majority of these "errors" are done on purpose. It's far better to fudge your math, get amazing (and wrong) conclusions and then get published in Nature then it is to not get published in Nature. The prestige of getting published in Nature or Science far outweighs the criticism you will get for forging or manipulating your data. In large part because the later can almost never be proven. You can always say you just made a mistake or plead ignorance.

scott_s 15y ago
1. Outright forgery or manipulation of data is a huge black mark on a researcher. Huge. It lasts forever, and is much bigger than the prestige from a single published article, where ever it was published. 2. http://en.wikipedia.org/wiki/Hanlons_razor http://en.wikipedia.org/wiki/Hanlons_razor
- forensic 15y ago
  Every scientist who has been caught forging data, has basically just claimed it was an accident. Whether they got "punished" or not depended purely on their political connections, not their behaviour. What's worse than the possibility of a black mark on your career? Not having a career at all. Which is what will happen if you don't publish.
  - scott_s 15y ago
    You're basically making an economic argument, where you assume the risks for cheating outweigh the consequences of being caught. But your support for that argument is the claim that the risk of being caught forging data is less than the risk of not having a career, but that is a false dichotomy. There is a middle ground, which is publishing valid but less sensational results. In other words, the choice is never "Forge these results, or have no career."
    - forensic 15y ago
      > In other words, the choice is never "Forge these results, or have no career." Yes it is, because in order to have a career, it is not enough to just publish. You have to publish in prestigious journals and you have to publish original, groundbreaking, important, and meaningful research. A tiny percentage of academics actually end up keeping their job until retirement. If you don't plan on perishing, you publish in prestigious journals by whatever means necessary. Whatever. Means. Necessary. "Accidentally" using bad statistics is the kind of "accident" that gets you tenure. It's significantly easier to forge groundbreaking, important, prestigious research than it is to actually do this research. Scientific fraud is epidemic precisely because of the economics of universities.
      - kd1220 15y ago
        I experienced this mindset in my graduate studies. Many graduate advisors spend so much time writing grant proposals and hobnobbing with government project managers that they didn't care what result their experiments got, as long as it was positive. I lost interest in being a researcher primarily because it's no different than working at a company - except you get paid less. I never saw any researchers or students falsify or make up data, but often I saw experiments performed without defining a hypothesis first. The hypothesis was created a posteriori to match any interesting correlations that could be combed from the data. I felt like I was a marketer trying to find a reason to promote a product. This particular approach might be standard when dealing with in-vivo experiments (on human subjects). It's a time-intensive process and you can't just rerun the experiment like you can on an artificial system. However, I didn't feel like it was the correct way to do research. There's been renewed interest in recent years of creating a journal of failed experiments for various research areas, but it never seems to catch on. It would be invaluable, but there are too many egotistical people in academia who don't want to be associated with failure in any way.
        PakG1 15y ago
        http://www.phdcomics.com/comics/archive.php?comicid=1431 http://www.phdcomics.com/comics/archive.php?comicid=1431
- joe_the_user 15y ago
  Having noticed "Hanlon's Razor" becoming the default reaction for a number of HN posters, I have to say what I feel is really bad about it when it applied to complex human endeavors. When one deals with modern institutional corruption, what one is dealing with is a complex combination of incompetence, overt deception of others, self-deception and simulated incompetence. Sure, incompetence is in the mix but when you use a construct like "Hanlon's Razor" to "label and forget" the situation, you hide the full situation. The complexities of these situations are there, there is no razor for a priori untangling.
  - scott_s 15y ago
    I think of it as a bias towards suspecting incompetence rather than a bias towards suspecting malice. I think people are too quick to assign narrative to situations, and malice is a stronger narrative than incompetence.
mbreese 15y ago
These aren't errors that would be caused my manipulating your data. If you were manipulating your data, you'd have made sure that your results were significant with the correct tests. These are errors where people didn't use the proper test. At the worst, you could only claim that people only submitted the results of the test that made their research look better than it otherwise would have been (with the correct test). In this case, I think it is more of an issue with the reviewers catching the problems than the authors deliberately misleading.
- jules 15y ago
  Actually, that is a huge problem. You have to decide on the test you are going to use before you see the data. After you've seen the data you can always come up with a specially made statistical test that will prove anything you want with any p-value you want.
- jvm 15y ago
  Actually, in the case of this specific error, reviewers are often shut down by editors who want to publish the finding. It happened to my friend's advisor recently: she called them out for it and the editor said it would be published anyway.
hardtke 15y ago
Errors like this have a life of their own -- there is no ill intent. Once a mistake is made in a seminal paper, most papers that build on that research will usually copy the error. Most authors do not test the statistical assumptions made in the literature and copy the statistical techniques used previously. I once traced an absurd FDA policy decision to a simple statistical mistake that had passed unchallenged through multiple papers and review committees.
jeltz 15y ago
I do not think it is outright fraud but rather are result of publication bias. People tend to publish results more often than the lack of a result, and in areas of research where there is much noise compared to signal statistical errors will stand out as interesting. This is also worsened by that today you have to publish often so you do not have the time to check the result before publishing.
locci 15y ago
After reading "A Mathematical Model for the Determination of Total Area Under Glucose Tolerance and Other Metabolic Curves"[1] I am unable to see any malice in this sort of things. [1]http://care.diabetesjournals.org/content/17/2/152 http://care.diabetesjournals.org/content/17/2/152
jessriedel 15y ago
Whatever the degree to which these errors are purposeful vs. accidental (a distinction I think can actually be hard to make when you consider how thoroughly and subtly people deceive themselves), remember that it's a hell of a lot easier to improve the statistics education of the researchers than their ethics. And improving their education for some of these simple ideas can be at least as effective; if it's obvious to everyone what the correct statistical technique is (because they've all been a bit better educated), then such papers won't be able to be published with these errors.

by cop359 15y ago

scott_s 15y ago

1. Outright forgery or manipulation of data is a huge black mark on a researcher. Huge. It lasts forever, and is much bigger than the prestige from a single published article, where ever it was published. 2. http://en.wikipedia.org/wiki/Hanlons_razor http://en.wikipedia.org/wiki/Hanlons_razor

forensic 15y ago
Every scientist who has been caught forging data, has basically just claimed it was an accident. Whether they got "punished" or not depended purely on their political connections, not their behaviour. What's worse than the possibility of a black mark on your career? Not having a career at all. Which is what will happen if you don't publish.
- scott_s 15y ago
  You're basically making an economic argument, where you assume the risks for cheating outweigh the consequences of being caught. But your support for that argument is the claim that the risk of being caught forging data is less than the risk of not having a career, but that is a false dichotomy. There is a middle ground, which is publishing valid but less sensational results. In other words, the choice is never "Forge these results, or have no career."
  - forensic 15y ago
    > In other words, the choice is never "Forge these results, or have no career." Yes it is, because in order to have a career, it is not enough to just publish. You have to publish in prestigious journals and you have to publish original, groundbreaking, important, and meaningful research. A tiny percentage of academics actually end up keeping their job until retirement. If you don't plan on perishing, you publish in prestigious journals by whatever means necessary. Whatever. Means. Necessary. "Accidentally" using bad statistics is the kind of "accident" that gets you tenure. It's significantly easier to forge groundbreaking, important, prestigious research than it is to actually do this research. Scientific fraud is epidemic precisely because of the economics of universities.
    - kd1220 15y ago
      I experienced this mindset in my graduate studies. Many graduate advisors spend so much time writing grant proposals and hobnobbing with government project managers that they didn't care what result their experiments got, as long as it was positive. I lost interest in being a researcher primarily because it's no different than working at a company - except you get paid less. I never saw any researchers or students falsify or make up data, but often I saw experiments performed without defining a hypothesis first. The hypothesis was created a posteriori to match any interesting correlations that could be combed from the data. I felt like I was a marketer trying to find a reason to promote a product. This particular approach might be standard when dealing with in-vivo experiments (on human subjects). It's a time-intensive process and you can't just rerun the experiment like you can on an artificial system. However, I didn't feel like it was the correct way to do research. There's been renewed interest in recent years of creating a journal of failed experiments for various research areas, but it never seems to catch on. It would be invaluable, but there are too many egotistical people in academia who don't want to be associated with failure in any way.
      - PakG1 15y ago
        http://www.phdcomics.com/comics/archive.php?comicid=1431 http://www.phdcomics.com/comics/archive.php?comicid=1431
joe_the_user 15y ago
Having noticed "Hanlon's Razor" becoming the default reaction for a number of HN posters, I have to say what I feel is really bad about it when it applied to complex human endeavors. When one deals with modern institutional corruption, what one is dealing with is a complex combination of incompetence, overt deception of others, self-deception and simulated incompetence. Sure, incompetence is in the mix but when you use a construct like "Hanlon's Razor" to "label and forget" the situation, you hide the full situation. The complexities of these situations are there, there is no razor for a priori untangling.
- scott_s 15y ago
  I think of it as a bias towards suspecting incompetence rather than a bias towards suspecting malice. I think people are too quick to assign narrative to situations, and malice is a stronger narrative than incompetence.

mbreese 15y ago

These aren't errors that would be caused my manipulating your data. If you were manipulating your data, you'd have made sure that your results were significant with the correct tests. These are errors where people didn't use the proper test. At the worst, you could only claim that people only submitted the results of the test that made their research look better than it otherwise would have been (with the correct test). In this case, I think it is more of an issue with the reviewers catching the problems than the authors deliberately misleading.

jules 15y ago
Actually, that is a huge problem. You have to decide on the test you are going to use before you see the data. After you've seen the data you can always come up with a specially made statistical test that will prove anything you want with any p-value you want.
jvm 15y ago
Actually, in the case of this specific error, reviewers are often shut down by editors who want to publish the finding. It happened to my friend's advisor recently: she called them out for it and the editor said it would be published anyway.

hardtke 15y ago

Errors like this have a life of their own -- there is no ill intent. Once a mistake is made in a seminal paper, most papers that build on that research will usually copy the error. Most authors do not test the statistical assumptions made in the literature and copy the statistical techniques used previously. I once traced an absurd FDA policy decision to a simple statistical mistake that had passed unchallenged through multiple papers and review committees.

jeltz 15y ago

I do not think it is outright fraud but rather are result of publication bias. People tend to publish results more often than the lack of a result, and in areas of research where there is much noise compared to signal statistical errors will stand out as interesting. This is also worsened by that today you have to publish often so you do not have the time to check the result before publishing.

locci 15y ago

After reading "A Mathematical Model for the Determination of Total Area Under Glucose Tolerance and Other Metabolic Curves"[1] I am unable to see any malice in this sort of things. [1]http://care.diabetesjournals.org/content/17/2/152 http://care.diabetesjournals.org/content/17/2/152

jessriedel 15y ago

Whatever the degree to which these errors are purposeful vs. accidental (a distinction I think can actually be hard to make when you consider how thoroughly and subtly people deceive themselves), remember that it's a hell of a lot easier to improve the statistics education of the researchers than their ethics. And improving their education for some of these simple ideas can be at least as effective; if it's obvious to everyone what the correct statistical technique is (because they've all been a bit better educated), then such papers won't be able to be published with these errors.