6 ms·

So, the magic number here was 5σ, the generally accepted "gold standard" for discovery, which would mean a 1 in 1.74 million chance that the results occurred by

So, the magic number here was 5σ, the generally accepted "gold standard" for discovery, which would mean a 1 in 1.74 million chance that the results occurred by chance rather than being a signal. As other commenters have pointed out, the presenter originally announced a 4.1σ observation, then continued to add data from other experiments until the combined result was 5.1σ. However, right at the end, he added in some additional data which actually brought the significance down to 4.9σ... That's science - you can't ignore data just because it ruins your big presentation. IANAPhysicist, but I'd be interested to know how strict the 5-sigma discovery rule is considered - for example, could they still get a Nobel prize for a 4.9σ announcement? I suppose it's not that big a deal - the LHC is still running, and I'm sure they'll have enough data for a true 5σ announcement soon. Regardless, hats off to all involved, it must be exciting to be at the forefront of human knowledge :)

mkramlich 14y ago
You should have waited to watch the ATLAS woman's presentation afterward. During hers, they concluded with a solid 5 sigma.
chime 14y ago
Atlas is still presenting. So let's wait and see what they came up with. 4.9σ was just from CMS. Edit: It is worth noting that 125GeV fits well within Fermilab's recent announcement: http://news.discovery.com/space/tevatron-data-detects-higgs-boson-existence-120703.html http://news.discovery.com/space/tevatron-data-detects-higgs-...
- dandelany 14y ago
  I think these physicists have been watching too many Steve Jobs presentations. "4.9σ... Oh, but one more thing!" ;) In all seriousness, does anyone know how common it is to do this kind of gradual reveal during scientific presentations vs. stating your basic final results in the introduction? It's kind of fun, but you could cut the tension in that room with a knife!
  - pork 14y ago
    The bravado comes from an ego and pride in ones self worth, not from Jobs' keynotes.
  - chime 14y ago
    The way they handled it was perfect. For over 20 minutes each, the presenters for CMS and Atlas explained their measurement capabilities, methods, recent improvements, and recorded measurements before stating their conclusion. They made sure to cover their bases and justify the conclusion before making a big announcement. I'm pretty sure the abstract for the papers will state the conclusion early on but the initial announcement for the greatest achievement in Particle Physics shouldn't be prefaced with "We found it!" just for the convenience of the impatient.
- typpo 14y ago
  She is working the crowd, so to speak. Here is the final CERN press release, which claims a 5 sigma result: http://press.web.cern.ch/press/PressReleases/Releases2012/PR17.12E.html http://press.web.cern.ch/press/PressReleases/Releases2012/PR...
lmm 14y ago
>IANAPhysicist, but I'd be interested to know how strict the 5-sigma discovery rule is considered - for example, could they still get a Nobel prize for a 4.9σ announcement? It's very much a HEP thing, and done that way because HEP is pretty much all statistical analysis these days. Other fields wouldn't treat the sigmas as the most important thing, and I've heard mutterings that it's not really the most accurate approach - but it is objective and easy to apply.
haberman 14y ago
5σ seems like an unnecessarily high standard to this non-physicist; what's the rationale for that? At 5σ we could publish 1,000 major discoveries a year and have E[false discovery accepted] = 1,740 years.
- JonnieCache 14y ago
  I think it's mainly so they can be as condescending and smug as possible at interdisciplinary conferences.
  - bermanoid 14y ago
    Actually, it's because the rest of you "scientists" end up publishing bullshit results that you got by chance. I'm kind of joking - most other scientists don't collect enough data to have to worry about 1 in 10,000 events happening by chance. In medicine, though, I'm not joking at all - those guys publish absolute statistical garbage all the time, I hesitate to even consider it a science because the data dredging is so bad. I can "prove" just about anything if the publication standard is 95% significance...
- mattheww 14y ago
  The two main rationales are that systematic uncertainties are historically under-estimated and that we are looking in many channels (more than 1000), so it would not be too hard to find a 3 or 4 sigma anomaly. The second part is the so-called "Look Elsewhere Effect." If you hit 5 sigma, you are fairly safe from either of these effects ruining your "discovery."
  - _delirium 14y ago
    > The second part is the so-called "Look Elsewhere Effect." Also more generally known as the "multiple testing" problem, fwiw (not sure why it has a different name in physics, unless I'm missing a subtlety). It's a major problem in "big data" also, where people just data-dredge thousands of possible parameter choices and pairwise correlations, and then report the p<0.01 results that came up, even though you'd expect several false positives just by chance with that methodology.
raphinou 14y ago
I searched but didn't find an explanation of this 5 sigma. Any reference to share?
- vibrunazo 14y ago
  5σ = 1 / 1,744,278 chance this was just luck. http://en.wikipedia.org/wiki/Standard_deviation http://en.wikipedia.org/wiki/Standard_deviation
  - cygx 14y ago
    The question this actually answers is: Assuming it was luck (ie there's no Higgs), how likely is it to get the measured result (or one even more extreme)?
    - tszyn 14y ago
      Exactly. P(results|luck) and P(luck|results) are two very different things. We don't know the probability that the results are due to "just luck" (whatever that means).
      - Almaviva 14y ago
        I think it's quite dramatic how many very intelligent people (up to writers in major publications) are uneducated about this distinction, and how dramatically wrong decisions can be if this is misunderstood. If we could teach this, as well as correlation vs causation, I'm convinced the intelligent public would make much better decisions about many things, medicine and nutrition to name a couple.
  - raphinou 14y ago
    I know standard deviation but I didn't know scientists reported their results talking about x sigmas. Thanks for clearing that up.
SiVal 14y ago
The 5σ convention is more about announcement than about discovery. No one can really say when a discovery is made. As more information comes in, the image can remain fuzzy or can get clearer and clearer. In the latter case, 5σ is a convention as a threshold for when you are entitled to announce a discovery without being blamed for jumping the gun. They'll collect data for many more years before passing the threshold of Nobel-worthy discovery.
ekianjo 14y ago
5σ is not a rule. It's just a measurement of the probability that the result you observe is an accident. It means that it's very unlikely that it was just pure luck. By the way, σ measurements are also used in many engineering jobs, and quality systems measurements. The kind of industry "goal" is to get 6σ (let's say when you manufacture consumer goods in large quantities, like millions every month). But for some other industries, 9σ is the benchmark - and from what I know, air companies have quality systems up to 9σ to ensure the lowest risk of something wrong happening.
- rflrob 14y ago
  How can they possibly have up to 9 sigma? 6 sigma is a failure rate about 1 in 10 billion, whereas 9 sigma would be 1 in 10^19 or so! That's oddly similar to the number of grains of sand on the earth![1] [1] http://www.hawaii.edu/suremath/jsand.html http://www.hawaii.edu/suremath/jsand.html
  - jerf 14y ago
    The math will happily pop out absurd numbers. Remember that all stats carry assumptions with them, such as "this is a Gaussian process from negative to positive infinity" and "everything is absolutely perfectly independent" and other things that break down if your push them hard enough. What that is telling you is just how hard you're pushing the math rather than the real odds. I'm not 1 in 10^19 sure of anything.
  - lolcraft 14y ago
    Yeah, I doubt most pilots have failure rates anywhere near or above 4 sigma.
    - ekianjo 14y ago
      That's not about pilots. It's about equipment failure rates.
  - ekianjo 14y ago
    Where did you get your failure rate at 6 sigma to be 1 in 10 billion ?? 6 Sigma is 3.4 defects per million opportunities. http://en.wikipedia.org/wiki/Six_Sigma http://en.wikipedia.org/wiki/Six_Sigma
Almaviva 14y ago
> which would mean a 1 in 1.74 million chance that the results occurred by chance rather than being a signal No!! It's the chance that randomness could produce a result that large. This distinction sounds pedantic, but misunderstanding this is widespread and leads to fallacies committed frequently by very smart people who ought to know better in many fields.
- raldi 14y ago
  Aren't those the same thing? How are they different?
  - lincolnq 14y ago
    I suspect Almaviva is talking about the biases inherent to publication / talking about results. Consider, by analogy, the event "rolling a six sided die and getting a 6 and announcing that fact to the world". "What is the probability that random events could produce a result that large?": one in six, per die roll. The question excludes the whole "announce it to the world" filter. "What is the probability that these results [getting a six and announcing it] occurred by chance, rather than being a signal?": We have no idea. If the person announced "I'm going to roll one die and announce the results, regardless of the outcome", then it's one in six. If they kept rolling dice until they got a six, then the probability is 1. If they rolled 3 dice, then the probability is 91/216. The point is that the scientific method has all sorts of biases (publication bias, confirmation bias, etc.) and p-values are rarely "probability that the result is wrong".
  - brendano 14y ago
    Standard null hypothesis testing -- including what they're talking about here -- focuses on P(results | random noise is active) You're talking about P(the thing we care about | results) The former is more like a simple sanity check. If random noise could have produced what you see, you shouldn't take the results too seriously. The former tells you a little bit about the latter -- which is good, because the latter is what we actually care about. But you can't explicitly compute the latter without making much stronger assumptions like priors and the like. That's why this last step of reasoning is often performed qualitatively. A good stats book to read about all this is Larry Wasserman's http://www.stat.cmu.edu/~larry/all-of-statistics/ http://www.stat.cmu.edu/~larry/all-of-statistics/
jasonwatkinspdx 14y ago
To be explicit, since this comment is still top voted: This comment was made by someone who only watched the first of 2 presentations. The second presentation showed a 5 sigma result. I'm Incredibly disappointed by the trivial inaccuracy of comments on hacker news lately, and that corrections never get upvoted quickly enough to prevent the spread of misinformation.

mkramlich 14y ago

You should have waited to watch the ATLAS woman's presentation afterward. During hers, they concluded with a solid 5 sigma.

chime 14y ago

Atlas is still presenting. So let's wait and see what they came up with. 4.9σ was just from CMS. Edit: It is worth noting that 125GeV fits well within Fermilab's recent announcement: http://news.discovery.com/space/tevatron-data-detects-higgs-boson-existence-120703.html http://news.discovery.com/space/tevatron-data-detects-higgs-...

dandelany 14y ago
I think these physicists have been watching too many Steve Jobs presentations. "4.9σ... Oh, but one more thing!" ;) In all seriousness, does anyone know how common it is to do this kind of gradual reveal during scientific presentations vs. stating your basic final results in the introduction? It's kind of fun, but you could cut the tension in that room with a knife!
- pork 14y ago
  The bravado comes from an ego and pride in ones self worth, not from Jobs' keynotes.
- chime 14y ago
  The way they handled it was perfect. For over 20 minutes each, the presenters for CMS and Atlas explained their measurement capabilities, methods, recent improvements, and recorded measurements before stating their conclusion. They made sure to cover their bases and justify the conclusion before making a big announcement. I'm pretty sure the abstract for the papers will state the conclusion early on but the initial announcement for the greatest achievement in Particle Physics shouldn't be prefaced with "We found it!" just for the convenience of the impatient.
typpo 14y ago
She is working the crowd, so to speak. Here is the final CERN press release, which claims a 5 sigma result: http://press.web.cern.ch/press/PressReleases/Releases2012/PR17.12E.html http://press.web.cern.ch/press/PressReleases/Releases2012/PR...

lmm 14y ago

>IANAPhysicist, but I'd be interested to know how strict the 5-sigma discovery rule is considered - for example, could they still get a Nobel prize for a 4.9σ announcement? It's very much a HEP thing, and done that way because HEP is pretty much all statistical analysis these days. Other fields wouldn't treat the sigmas as the most important thing, and I've heard mutterings that it's not really the most accurate approach - but it is objective and easy to apply.

haberman 14y ago

5σ seems like an unnecessarily high standard to this non-physicist; what's the rationale for that? At 5σ we could publish 1,000 major discoveries a year and have E[false discovery accepted] = 1,740 years.

JonnieCache 14y ago
I think it's mainly so they can be as condescending and smug as possible at interdisciplinary conferences.
- bermanoid 14y ago
  Actually, it's because the rest of you "scientists" end up publishing bullshit results that you got by chance. I'm kind of joking - most other scientists don't collect enough data to have to worry about 1 in 10,000 events happening by chance. In medicine, though, I'm not joking at all - those guys publish absolute statistical garbage all the time, I hesitate to even consider it a science because the data dredging is so bad. I can "prove" just about anything if the publication standard is 95% significance...
mattheww 14y ago
The two main rationales are that systematic uncertainties are historically under-estimated and that we are looking in many channels (more than 1000), so it would not be too hard to find a 3 or 4 sigma anomaly. The second part is the so-called "Look Elsewhere Effect." If you hit 5 sigma, you are fairly safe from either of these effects ruining your "discovery."
- _delirium 14y ago
  > The second part is the so-called "Look Elsewhere Effect." Also more generally known as the "multiple testing" problem, fwiw (not sure why it has a different name in physics, unless I'm missing a subtlety). It's a major problem in "big data" also, where people just data-dredge thousands of possible parameter choices and pairwise correlations, and then report the p<0.01 results that came up, even though you'd expect several false positives just by chance with that methodology.

raphinou 14y ago

I searched but didn't find an explanation of this 5 sigma. Any reference to share?

vibrunazo 14y ago
5σ = 1 / 1,744,278 chance this was just luck. http://en.wikipedia.org/wiki/Standard_deviation http://en.wikipedia.org/wiki/Standard_deviation
- cygx 14y ago
  The question this actually answers is: Assuming it was luck (ie there's no Higgs), how likely is it to get the measured result (or one even more extreme)?
  - tszyn 14y ago
    Exactly. P(results|luck) and P(luck|results) are two very different things. We don't know the probability that the results are due to "just luck" (whatever that means).
    - Almaviva 14y ago
      I think it's quite dramatic how many very intelligent people (up to writers in major publications) are uneducated about this distinction, and how dramatically wrong decisions can be if this is misunderstood. If we could teach this, as well as correlation vs causation, I'm convinced the intelligent public would make much better decisions about many things, medicine and nutrition to name a couple.
- raphinou 14y ago
  I know standard deviation but I didn't know scientists reported their results talking about x sigmas. Thanks for clearing that up.

SiVal 14y ago

The 5σ convention is more about announcement than about discovery. No one can really say when a discovery is made. As more information comes in, the image can remain fuzzy or can get clearer and clearer. In the latter case, 5σ is a convention as a threshold for when you are entitled to announce a discovery without being blamed for jumping the gun. They'll collect data for many more years before passing the threshold of Nobel-worthy discovery.

ekianjo 14y ago

5σ is not a rule. It's just a measurement of the probability that the result you observe is an accident. It means that it's very unlikely that it was just pure luck. By the way, σ measurements are also used in many engineering jobs, and quality systems measurements. The kind of industry "goal" is to get 6σ (let's say when you manufacture consumer goods in large quantities, like millions every month). But for some other industries, 9σ is the benchmark - and from what I know, air companies have quality systems up to 9σ to ensure the lowest risk of something wrong happening.

rflrob 14y ago
How can they possibly have up to 9 sigma? 6 sigma is a failure rate about 1 in 10 billion, whereas 9 sigma would be 1 in 10^19 or so! That's oddly similar to the number of grains of sand on the earth![1] [1] http://www.hawaii.edu/suremath/jsand.html http://www.hawaii.edu/suremath/jsand.html
- jerf 14y ago
  The math will happily pop out absurd numbers. Remember that all stats carry assumptions with them, such as "this is a Gaussian process from negative to positive infinity" and "everything is absolutely perfectly independent" and other things that break down if your push them hard enough. What that is telling you is just how hard you're pushing the math rather than the real odds. I'm not 1 in 10^19 sure of anything.
- lolcraft 14y ago
  Yeah, I doubt most pilots have failure rates anywhere near or above 4 sigma.
  - ekianjo 14y ago
    That's not about pilots. It's about equipment failure rates.
- ekianjo 14y ago
  Where did you get your failure rate at 6 sigma to be 1 in 10 billion ?? 6 Sigma is 3.4 defects per million opportunities. http://en.wikipedia.org/wiki/Six_Sigma http://en.wikipedia.org/wiki/Six_Sigma

Almaviva 14y ago

> which would mean a 1 in 1.74 million chance that the results occurred by chance rather than being a signal No!! It's the chance that randomness could produce a result that large. This distinction sounds pedantic, but misunderstanding this is widespread and leads to fallacies committed frequently by very smart people who ought to know better in many fields.

raldi 14y ago
Aren't those the same thing? How are they different?
- lincolnq 14y ago
  I suspect Almaviva is talking about the biases inherent to publication / talking about results. Consider, by analogy, the event "rolling a six sided die and getting a 6 and announcing that fact to the world". "What is the probability that random events could produce a result that large?": one in six, per die roll. The question excludes the whole "announce it to the world" filter. "What is the probability that these results [getting a six and announcing it] occurred by chance, rather than being a signal?": We have no idea. If the person announced "I'm going to roll one die and announce the results, regardless of the outcome", then it's one in six. If they kept rolling dice until they got a six, then the probability is 1. If they rolled 3 dice, then the probability is 91/216. The point is that the scientific method has all sorts of biases (publication bias, confirmation bias, etc.) and p-values are rarely "probability that the result is wrong".
- brendano 14y ago
  Standard null hypothesis testing -- including what they're talking about here -- focuses on P(results | random noise is active) You're talking about P(the thing we care about | results) The former is more like a simple sanity check. If random noise could have produced what you see, you shouldn't take the results too seriously. The former tells you a little bit about the latter -- which is good, because the latter is what we actually care about. But you can't explicitly compute the latter without making much stronger assumptions like priors and the like. That's why this last step of reasoning is often performed qualitatively. A good stats book to read about all this is Larry Wasserman's http://www.stat.cmu.edu/~larry/all-of-statistics/ http://www.stat.cmu.edu/~larry/all-of-statistics/

jasonwatkinspdx 14y ago

To be explicit, since this comment is still top voted: This comment was made by someone who only watched the first of 2 presentations. The second presentation showed a 5 sigma result. I'm Incredibly disappointed by the trivial inaccuracy of comments on hacker news lately, and that corrections never get upvoted quickly enough to prevent the spread of misinformation.