3 ms·

> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of

by batbomb 13y ago

> These are the sorts of nuts-and-bolts reproducibility issues that drive researchers crazy, because they can be affected by things like the specific strain of mice you use, where you buy your chemicals, and even the pH of your lab's water supply. No amount of statistical thinking is going to change any of that. This screams systematic error and error propagation to me. It's possible that we don't need to up the p-value, we just need to make sure researchers aren't stupid and can properly account for all sources of errors. The problem is that's often an acquired skill over time, not something younger researchers typically think about, especially those who aren't multidisciplinary.

flatline 13y ago
Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of errors, up to and including blatantly cherry-picking or fabricating results, where the focus should be. There's been a series of recent posts encouraging scientists (and, moreover, journals) to publish negative results, which is probably a good starting point to promote greater transparency and honesty in the paper publishing game.
- apsec112 13y ago
  This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis. For example, you could have an extremely unlikely hypothesis (eg. "dice are controlled by alien telepathy"), test it, and still come out with p < 0.05 because of something much more plausible ("the dice were badly made and are biased"). This is known as the base rate fallacy; an extraordinary claim requires extraordinary proof. Moreover, to prove an extraordinary claim, you must show it is more likely not just compared to chance, but compared to every less extraordinary alternative, including "the experimenters are faking the data". It's also entirely possible to get a result at p < 0.05 that makes the hypothesis you're testing less likely. Suppose you want to know how far away the nearest star is. The value in the textbook is 13.4 light-years, and you think it's really 12,000. You take some measurements, and get values of 19.6, 17.4, 20.1, 20.4 and 18.5. Now, this is a significant result at p < 0.05 - if the star is 13.4 light-years away, then getting these numbers has a probability of less than 5%. However, these results completely rule out the hypothesis you're testing. The numbers you get are pretty unlikely if the real number is 13.4, but extraordinarily unlikely if the real number is 12,000, so this experiment makes the 13.4 number more credible. This kind of thing is why lots of researchers still believe in psychic powers - they keep testing for psychic powers, and keep getting results at p < 0.05, but don't notice their results make psychic powers less plausible. (Not joking - see http://commonsenseatheism.com/wp-content/uploads/2010/11/Wagenmakers-Why-Psychologists-Must-Change-the-Way-They-Analyze-Their-Data.pdf http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... for a detailed explanation.) This is an elementary mistake that every freshman statistics course warns against, and yet it's absolutely pervasive.
  - eli_gottlieb 13y ago
    Right, you want this superhero cape with Bayes' Theorem printed on it? It's got priiiiooooooors.
  - jmpeax 13y ago
    > What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. The hypothesis in question is the null hypothesis which must be assumed true not false for the p-value to mean a 100*p% chance of seeing the observed data. I can see why elementary mistakes are so pervasive.
- hnha 13y ago
  It's not that arbitrary really. 95% equals 2 sigma.
dnautics 13y ago
exactly wrong, it's the multidisciplinary scientists who don't get enough depth or rigor to come close to understanding sources of error and error propagation. The thing I fear is underexperienced multidisciplinary researchers coming up with convoluted procedures that put their hands and eyes on their work in so many places that small sources of error have compounded themselves and entrenched themselves into nooks and crannies.
- haimez 13y ago
  You're right! It's those other jerks getting all their high school statistics wrong. No true scientist (Scotsman) could ever make THAT mistake.
- SamuelMulder 13y ago
  I haven't seen any reason to believe that being multidisciplinary makes this more likely. The whole problem is caused by the fact that most of the single-disciplinary researchers are using bad statistics. Very few degree programs give scientists enough background in statistics and experimental rigor to prevent these problems.
  - dnautics 13y ago
    no doubt, the single-disciplinary researchers use bad statistics, too. My broader point is that multidisciplinary researchers are less likely to use statistics correctly, if for no other reason than that the fact that they are multidisciplinary means that they are less likely to be able to handle depth from a personality/self-habits point of view. Perhaps that's a little bit of projection, but I don't use any but the most rudimentary statistics in my publications and I don't make claims about significance.
    - SamuelMulder 13y ago
      It would be interesting to look into this. My experience has been that multi-disciplinary researchers are more likely to be familiar with a broader range of research and familiar with statistical techniques across their different disciplines. Someone with a narrower perspective is more likely to get into bad habits related to their field. I think your comment about them being less likely to be able to handle depth from a personality/self-habits point of view is really projecting and probably based on experiences with a few bad apples. My experience doesn't reflect that at all. I feel like a lot of the problems with statistics come from overly insular journals and conferences where everyone does it wrong and nobody realizes that because they're all doing the same thing. I've seen this in some computer science fields.
chalst 13y ago
I agree with the diagnosis. The big problem is that it is not possible to be sure that some study accounts for all possible confounding factors, unless you have a correct theory that allows you to determine what factors might be confounding factors. Statistical methodology alone cannot tell you this. Reproducibility by researchers who know no more about the experimental setup than what was published is a better way of guarding against this than upping the significance hurdle. Experienced researchers may have a long history of ignoring confounding factors - I am not sure maturity matters much. I think the best time to have your complacency about constructing statistical investogations torn apart is while you are doing your PhD.

by batbomb 13y ago

flatline 13y ago

Yeah, the 95% confidence interval is completely arbitrary to begin with. Is it really better than, say, 94% -- which will most assuredly not get you published? In many fields, I think that a p-value indicating 95% confidence in the results is fine, you are never going to get 99.999% or even 99% due to the errors inherent in the subject of study, which doesn't invalidate the results out of hand. It's the other host of errors, up to and including blatantly cherry-picking or fabricating results, where the focus should be. There's been a series of recent posts encouraging scientists (and, moreover, journals) to publish negative results, which is probably a good starting point to promote greater transparency and honesty in the paper publishing game.

apsec112 13y ago
This is part of the problem. A p-value of 0.05 absolutely does not (!!!) indicate 95% confidence in the results! What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. It tells you absolutely nothing about, if you observe such-and-such data, then how strongly should we believe the hypothesis. For example, you could have an extremely unlikely hypothesis (eg. "dice are controlled by alien telepathy"), test it, and still come out with p < 0.05 because of something much more plausible ("the dice were badly made and are biased"). This is known as the base rate fallacy; an extraordinary claim requires extraordinary proof. Moreover, to prove an extraordinary claim, you must show it is more likely not just compared to chance, but compared to every less extraordinary alternative, including "the experimenters are faking the data". It's also entirely possible to get a result at p < 0.05 that makes the hypothesis you're testing less likely. Suppose you want to know how far away the nearest star is. The value in the textbook is 13.4 light-years, and you think it's really 12,000. You take some measurements, and get values of 19.6, 17.4, 20.1, 20.4 and 18.5. Now, this is a significant result at p < 0.05 - if the star is 13.4 light-years away, then getting these numbers has a probability of less than 5%. However, these results completely rule out the hypothesis you're testing. The numbers you get are pretty unlikely if the real number is 13.4, but extraordinarily unlikely if the real number is 12,000, so this experiment makes the 13.4 number more credible. This kind of thing is why lots of researchers still believe in psychic powers - they keep testing for psychic powers, and keep getting results at p < 0.05, but don't notice their results make psychic powers less plausible. (Not joking - see http://commonsenseatheism.com/wp-content/uploads/2010/11/Wagenmakers-Why-Psychologists-Must-Change-the-Way-They-Analyze-Their-Data.pdf http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... for a detailed explanation.) This is an elementary mistake that every freshman statistics course warns against, and yet it's absolutely pervasive.
- eli_gottlieb 13y ago
  Right, you want this superhero cape with Bayes' Theorem printed on it? It's got priiiiooooooors.
- jmpeax 13y ago
  > What a p-value means is, if the hypothesis in question is false, then we have an x% chance of seeing the observed data. The hypothesis in question is the null hypothesis which must be assumed true not false for the p-value to mean a 100*p% chance of seeing the observed data. I can see why elementary mistakes are so pervasive.
hnha 13y ago
It's not that arbitrary really. 95% equals 2 sigma.

dnautics 13y ago

exactly wrong, it's the multidisciplinary scientists who don't get enough depth or rigor to come close to understanding sources of error and error propagation. The thing I fear is underexperienced multidisciplinary researchers coming up with convoluted procedures that put their hands and eyes on their work in so many places that small sources of error have compounded themselves and entrenched themselves into nooks and crannies.

haimez 13y ago
You're right! It's those other jerks getting all their high school statistics wrong. No true scientist (Scotsman) could ever make THAT mistake.
SamuelMulder 13y ago
I haven't seen any reason to believe that being multidisciplinary makes this more likely. The whole problem is caused by the fact that most of the single-disciplinary researchers are using bad statistics. Very few degree programs give scientists enough background in statistics and experimental rigor to prevent these problems.
- dnautics 13y ago
  no doubt, the single-disciplinary researchers use bad statistics, too. My broader point is that multidisciplinary researchers are less likely to use statistics correctly, if for no other reason than that the fact that they are multidisciplinary means that they are less likely to be able to handle depth from a personality/self-habits point of view. Perhaps that's a little bit of projection, but I don't use any but the most rudimentary statistics in my publications and I don't make claims about significance.
  - SamuelMulder 13y ago
    It would be interesting to look into this. My experience has been that multi-disciplinary researchers are more likely to be familiar with a broader range of research and familiar with statistical techniques across their different disciplines. Someone with a narrower perspective is more likely to get into bad habits related to their field. I think your comment about them being less likely to be able to handle depth from a personality/self-habits point of view is really projecting and probably based on experiences with a few bad apples. My experience doesn't reflect that at all. I feel like a lot of the problems with statistics come from overly insular journals and conferences where everyone does it wrong and nobody realizes that because they're all doing the same thing. I've seen this in some computer science fields.

chalst 13y ago

I agree with the diagnosis. The big problem is that it is not possible to be sure that some study accounts for all possible confounding factors, unless you have a correct theory that allows you to determine what factors might be confounding factors. Statistical methodology alone cannot tell you this. Reproducibility by researchers who know no more about the experimental setup than what was published is a better way of guarding against this than upping the significance hurdle. Experienced researchers may have a long history of ignoring confounding factors - I am not sure maturity matters much. I think the best time to have your complacency about constructing statistical investogations torn apart is while you are doing your PhD.