4 ms·

> which would mean a 1 in 1.74 million chance that the results occurred by chance rather than being a signal No!! It's the chance that randomness could produce

> which would mean a 1 in 1.74 million chance that the results occurred by chance rather than being a signal No!! It's the chance that randomness could produce a result that large. This distinction sounds pedantic, but misunderstanding this is widespread and leads to fallacies committed frequently by very smart people who ought to know better in many fields.

raldi 14y ago
Aren't those the same thing? How are they different?
- lincolnq 14y ago
  I suspect Almaviva is talking about the biases inherent to publication / talking about results. Consider, by analogy, the event "rolling a six sided die and getting a 6 and announcing that fact to the world". "What is the probability that random events could produce a result that large?": one in six, per die roll. The question excludes the whole "announce it to the world" filter. "What is the probability that these results [getting a six and announcing it] occurred by chance, rather than being a signal?": We have no idea. If the person announced "I'm going to roll one die and announce the results, regardless of the outcome", then it's one in six. If they kept rolling dice until they got a six, then the probability is 1. If they rolled 3 dice, then the probability is 91/216. The point is that the scientific method has all sorts of biases (publication bias, confirmation bias, etc.) and p-values are rarely "probability that the result is wrong".
- brendano 14y ago
  Standard null hypothesis testing -- including what they're talking about here -- focuses on P(results | random noise is active) You're talking about P(the thing we care about | results) The former is more like a simple sanity check. If random noise could have produced what you see, you shouldn't take the results too seriously. The former tells you a little bit about the latter -- which is good, because the latter is what we actually care about. But you can't explicitly compute the latter without making much stronger assumptions like priors and the like. That's why this last step of reasoning is often performed qualitatively. A good stats book to read about all this is Larry Wasserman's http://www.stat.cmu.edu/~larry/all-of-statistics/ http://www.stat.cmu.edu/~larry/all-of-statistics/
- capnrefsmmat 14y ago
  Imagine testing 1000 potential cancer drugs. Only 100 of them actually work, but you don't know that; you have to do the trial. So you get out some petri dishes and start testing. You look for a 5% chance of false positive (p < 0.05), which is the statistical significance level usually used in medical trials. Of the 100 real drugs, you detect all of them. Of the 900 fake drugs, 5% of them falsely appear to work. So you have 145 drugs you think work, only two thirds of which actually work. The chance of any individual drug having obtained its positive results by chance is 31%, not 5%. I have a guide to this here, since it's so common: http://www.refsmmat.com/statistics/#an-introduction-to-data-analysis http://www.refsmmat.com/statistics/#an-introduction-to-data-... (Side note: the average medical trial would only detect 50 of the working drugs, not all 100, so the real situation is even worse.)
  - raldi 14y ago
    > The chance of any individual drug having obtained its positive results by chance is 31% And that's different from the chance that randomness could have produced the effect? How?
    - capnrefsmmat 14y ago
      Sorry, I wasn't very clear. Usually people quote the p value as the chance the result is a fluke; the p value in CERN's case is p = 1/1,740,000. But that's the chance that the effect would be produced if the Higgs did not exist, which is different. By analogy in the medical case, p = 0.05. The incorrect interpretation is that this means only 5% of drugs with statistically significant benefits actually achieved these benefits through luck; rather, the right interpretation is that 5% of the nonfunctional drugs somehow appeared to work. You could also imagine testing 200,000,000 hypotheses which were all completely false. Even if you used CERN's level of statistical significance, you'd still quite likely find one hypothesis which appears to be true, simply by chance. The chance of that hypothesis being false is 100%, despite the significance level of 1 in 1,740,000. So yes, 31% is exactly the chance that randomness produced the effect in the trial. But people will try to tell you that it's actually 5%, and they're wrong.
- Almaviva 14y ago
  A couple of extreme examples: 1. I'm rolling two dice to try and get the highest total. I get two sixes. What is the chance randomness produces this? 1/36, about 0.027. This more than a two sigma result. What is the chance that this is caused by random chance? 2.7%? Nope, 100%. 2. I study the same thing in a million situations in parallel. I take the most extreme result and find that random chance can produce this one time in 1.7 million. It's a five sigma result! What are the chances this result is caused by random chance?

raldi 14y ago

Aren't those the same thing? How are they different?

lincolnq 14y ago
I suspect Almaviva is talking about the biases inherent to publication / talking about results. Consider, by analogy, the event "rolling a six sided die and getting a 6 and announcing that fact to the world". "What is the probability that random events could produce a result that large?": one in six, per die roll. The question excludes the whole "announce it to the world" filter. "What is the probability that these results [getting a six and announcing it] occurred by chance, rather than being a signal?": We have no idea. If the person announced "I'm going to roll one die and announce the results, regardless of the outcome", then it's one in six. If they kept rolling dice until they got a six, then the probability is 1. If they rolled 3 dice, then the probability is 91/216. The point is that the scientific method has all sorts of biases (publication bias, confirmation bias, etc.) and p-values are rarely "probability that the result is wrong".
brendano 14y ago
Standard null hypothesis testing -- including what they're talking about here -- focuses on P(results | random noise is active) You're talking about P(the thing we care about | results) The former is more like a simple sanity check. If random noise could have produced what you see, you shouldn't take the results too seriously. The former tells you a little bit about the latter -- which is good, because the latter is what we actually care about. But you can't explicitly compute the latter without making much stronger assumptions like priors and the like. That's why this last step of reasoning is often performed qualitatively. A good stats book to read about all this is Larry Wasserman's http://www.stat.cmu.edu/~larry/all-of-statistics/ http://www.stat.cmu.edu/~larry/all-of-statistics/
capnrefsmmat 14y ago
Imagine testing 1000 potential cancer drugs. Only 100 of them actually work, but you don't know that; you have to do the trial. So you get out some petri dishes and start testing. You look for a 5% chance of false positive (p < 0.05), which is the statistical significance level usually used in medical trials. Of the 100 real drugs, you detect all of them. Of the 900 fake drugs, 5% of them falsely appear to work. So you have 145 drugs you think work, only two thirds of which actually work. The chance of any individual drug having obtained its positive results by chance is 31%, not 5%. I have a guide to this here, since it's so common: http://www.refsmmat.com/statistics/#an-introduction-to-data-analysis http://www.refsmmat.com/statistics/#an-introduction-to-data-... (Side note: the average medical trial would only detect 50 of the working drugs, not all 100, so the real situation is even worse.)
- raldi 14y ago
  > The chance of any individual drug having obtained its positive results by chance is 31% And that's different from the chance that randomness could have produced the effect? How?
  - capnrefsmmat 14y ago
    Sorry, I wasn't very clear. Usually people quote the p value as the chance the result is a fluke; the p value in CERN's case is p = 1/1,740,000. But that's the chance that the effect would be produced if the Higgs did not exist, which is different. By analogy in the medical case, p = 0.05. The incorrect interpretation is that this means only 5% of drugs with statistically significant benefits actually achieved these benefits through luck; rather, the right interpretation is that 5% of the nonfunctional drugs somehow appeared to work. You could also imagine testing 200,000,000 hypotheses which were all completely false. Even if you used CERN's level of statistical significance, you'd still quite likely find one hypothesis which appears to be true, simply by chance. The chance of that hypothesis being false is 100%, despite the significance level of 1 in 1,740,000. So yes, 31% is exactly the chance that randomness produced the effect in the trial. But people will try to tell you that it's actually 5%, and they're wrong.
Almaviva 14y ago
A couple of extreme examples: 1. I'm rolling two dice to try and get the highest total. I get two sixes. What is the chance randomness produces this? 1/36, about 0.027. This more than a two sigma result. What is the chance that this is caused by random chance? 2.7%? Nope, 100%. 2. I study the same thing in a million situations in parallel. I take the most extreme result and find that random chance can produce this one time in 1.7 million. It's a five sigma result! What are the chances this result is caused by random chance?