7 ms·

The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence. No, that’s not wha

The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence. No, that’s not what this means. But this is about one in 1,000 BBC Science articles that have repeated this confusion. Edited to add: What this actually means is that, conditional on the null hypothesis (that there is nothing there to find), there is a one out of 1,000 chance of seeing an observation such as this by chance. But the BBC article has inverted this. Rather than giving you the probability of the data given the hypothesis, they're purporting to give you the probability of the hypothesis given the data (which in general cannot be objectively stated; this is a standard difficulty in the philosophy of science and statistics): "there is roughly a one in 1,000 chance that the measurement is a statistical coincidence [i.e. the null hypothesis is true and there's nothing to see here]". They make it sound like the scientists are virtually certain that they've found something here, when in reality this is probably nothing (although that's a matter of opinion).

skissane 6y ago
I believe you that isn't what it means, but for those (like myself) who have limited familiarity with this topic, could you please explain what it actually means.
- mhh__ 6y ago
  The p-value of a physics experiment isn't the probability of the result coming from thin air but rather than the probability of a result as or more extreme under the assumption of the null hypothesis (i.e. the effect we are looking for doesn't exist). The rub is that the design of the experiment and thus the hypothesis test is all dependent on various subjective factors, especially the people designing the experiment themselves. I'm not statistician however so presumably I'm just as wrong
- Kaibeezy 6y ago
  Good explanation of sigmas here: https://news.mit.edu/2012/explained-sigma-0209 https://news.mit.edu/2012/explained-sigma-0209 I’m thinking it would be more accurate (by being less precise) for BBC to describe the 1:1000 possibility as just coincidence rather than “statistical” coincidence.
  - spuz 6y ago
    This article seems to agree that 3 sigma simply means "1 in 1000 possibility the result is a coincidence". I'm not sure what difference removing the word "statistical" makes here or why the BBC explanation is wrong.
    - Kaibeezy 6y ago
      Yeah, I’m not exactly getting why BBC is wrong either. Re “statistical”, to me, it sort of implies all sources of error or variability were provably not present (prove a negative) and the only reason for a coincidental result was a statistical anomaly. What’s got to be far more likely is some unaccounted-for variable produced the coincidental result.
    - trehalose 6y ago
      The article does seem to agree, and it's wrong just like the BBC article. Here's an old article from the time of the Higgs Boson discovery that had a great explanation: https://blogs.scientificamerican.com/observations/five-sigmawhats-that/ https://blogs.scientificamerican.com/observations/five-sigma... > Chances are, you heard this month about the discovery of a tiny fundamental physics particle that may be the long-sought Higgs boson. The phrase five-sigma was tossed about by scientists to describe the strength of the discovery. So, what does five-sigma mean? > In short, five-sigma corresponds to a p-value, or probability, of 3x10-7, or about 1 in 3.5 million. This is not the probability that the Higgs boson does or doesn't exist; rather, it is the probability that if the particle does not exist, the data that CERN scientists collected in Geneva, Switzerland, would be at least as extreme as what they observed. "The reason that it's so annoying is that people want to hear declarative statements, like 'The probability that there's a Higgs is 99.9 percent,' but the real statement has an 'if' in there. There's a conditional. There's no way to remove the conditional," says Kyle Cranmer, a physicist at New York University and member of the ATLAS team, one of the two groups that announced the new particle results in Geneva on July 4.
      - spuz 6y ago
        Thanks - I think that does explain some of the potential area of confusion. However, I don't agree that the BBC's explanation differs from what the true meaning is according to this Scientific American article says. The BBC article says: > The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence My interpretation of this is "If there was no new physics, then there is a 1/1000 chance the same result could be measured by coincidence." I understand why some people may interpret this as "There is a 1/1000 chance that there is no new physics" though.
- stdbrouw 6y ago
  The probability of winning the lottery if you're a cheater is high, but the probability of being a cheater if you win the lottery is low. More generally, the probability of A conditional on B is not the same as the probability of B conditional on A. Sometimes they're similar, but it depends on the pre-existing odds of A and B.
- Mordisquitos 6y ago
  In case it helps, following that logic, green jelly beans could cause acne and there would be "roughly a 5% chance that the measurement is a statistical coincidence": https://xkcd.com/882/ https://xkcd.com/882/
flukus 6y ago
How many iterations that could create this result do they run? 1 in 1000 sounds unlikely, but for all I know they have run this experiment 50,000 times.
- DavidSJ 6y ago
  I would guess there are many more than thousands of such experiments that have been run, testing all the various permutations of dozens (or more) of different beyond Standard Model physical theories.
  - dukwon 6y ago
    Testing different models is done in the analysis of the data, not in running the experiment differently. The "experiment" (colliding protons together) has been repeated quadrillions of times at the LHC. But there are only a few parameters you can change about it, namely the energy and how focused the beams are. Altering the detectors to change/improve how data is collected is an expensive and slow process.
    - DavidSJ 6y ago
      Fair enough; what I meant is "multiple testing" in the context of a single overarching experiment, which is I believe what the comment I was replying to was getting at as well.
      - davrosthedalek 6y ago
        In any case, you have a point. Whether multiple analysis, multiple channels or multiple experiments, you have a look-elsewhere effect, and that can easily knock off one or two sigmas. If you look at a 10000 possibilities, you will a couple with 3 sigma difference, just by chance. LHC looks at MANY possibilities. For a while, they tried to keep track to quantify the look-elsewhere effect. I think they have given up on that.
- dalewe 6y ago
  Not sure I entirely understand your question but I'll try to explain a bit anyway: they're using a dataset of 9fb^-1. That means it includes 9 events for a crosssection of 1fb (femtobarn). Now the paper doesn't seem to include what crosssection their events have but if we assume something of the order of magnitude of 1pb then the dataset would contain around 9000 events of interest. Now all that doesn't really matter as this is already factored into the standard deviation of their result. They have enough data to say it's a 3 sigma (being the standard deviation) deviation from the standard model. Now this "1 in 1000" really just means that the null hypothesis (the Standard Model) says there's a 0.1% chance for that result to happen. Of course it could also be that they forgot to factor in some uncertainty of their measurements and their sigma is actually much bigger. But it surely won't be the size of their dataset as this is quite obvious and easy to take into account.
simiones 6y ago
Even after you explain it, I can barely grasp the significance. Given the expected level of understanding of someone reading a BBC article about this topic, I don't think they are wrong at all in explaining it that way - that is, I don't think their intended audience would take the message differently to any extent whatsoever if they had explained the way you do. To be more specific, to the average BBC reader, the phrases The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence. and The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance to have seen this data even if the particle doesn't exist. convey the exact same message, the first one just uses fewer words.
- DavidSJ 6y ago
  Look, that may be so, but if it is, then they should either omit such a line entirely so as not to mislead their readers, select better language so that we don't collapse concepts which are fundamentally, completely, different from each other, or they could try to educate the public on a distinction which is fundamental to science. I mean, we might as well say that the BBC should just use the words "million" and "billion" interchangeably because most readers have no sense of scale. Bottom line: thanks to this line, the reader is likely to think that new physics has very likely been discovered, when the opposite is the case.
  - simiones 6y ago
    I don't agree. The whole paragraph makes that clear - 1 in 1000 is actually pretty good odds that this is just a coincidence, and most people should know this intuitively. > The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence. So people should not get carried away by these findings, according to team leader Prof Chris Parkes, from the University of Manchester.
- davrosthedalek 6y ago
  The problem with it is that layman people believe that both of these statements are equal to "it's a 999/1000 chance that it was something else". Which is not true. The difference in these two formulations is that for b) it's explicit wrong, for a) this is arguably what the article says (which is not what the scientists say)
choeger 6y ago
I did not even see this at first. But they did really write that the chance of "no new physics" is 1:1000, right? So they gave "there is new physics" interpretation a huge probability. Seriously, I completely glanced over this. Slightly embarrassing. In any case, if the LHC runs thousands of such experiments, we do expect such signals, right?
dataflow 6y ago
I don't see how the BBC's statement can translate into P(hypothesis|data). To me it's P(data|H0). Consider this simpler example: "Scientists always assumed their coin was fair. Well, they flipped it 10 times, and it came up heads every time. This is a 3-sigmal result. There's a 1 in 1024 chance of this measurement being a statistical coincidence." The reader would take this "1 in 1024 chance of coincidence" to mean "if the coin was fair, they'd have a 1 in 1024 chance of seeing this event occur". That's P(data|fair) = P(10 heads|fair) = 1/1024. I don't see how you read that and translate it into P(fair|10 heads).
- DavidSJ 6y ago
  If I say: "there's a 25% chance that Pat is from North Dakota", I'm taking as a given that there is a person named Pat, and I'm talking about the chance that Pat, whom we know to exist, is from North Dakota. In this case, grammatically, they're talking about the chance that the measurement (data) is a statistical coincidence, as opposed to not being a statistical coincidence. They're not talking about the chance that the data exists, as opposed to not existing.
  - dataflow 6y ago
    > If I say: "there's a 25% chance that Pat is from North Dakota", I'm taking as a given that there is a person named Pat, and I'm talking about the chance that Pat, whom we know to exist, is from North Dakota. Yes, and if I say "there's a 0.001% chance that these coin flip observations were coincidental", I'm taking it as a given that there were coin flips observed (indeed, there were!) I'm talking about the chance that the observations, which we (indeed) know to exist, were coincidental. That is... literally P(observations | null hypothesis) = P(10 heads | fair). It's clearly not P(null hypothesis | observations) = P(fair | 10 heads)... > In this case, grammatically, they're talking about the chance that the measurement (data) is a statistical coincidence, as opposed to not being a statistical coincidence. They're not talking about the chance that the data exists, as opposed to not existing. Which is... perfectly fine. The data clearly does exist (as would the coin flips in my example), so the chance of the data existing is 100%... there's nothing interesting to talk about there. But the chance of it having been coincidental is 0.001%. This all seems correct to me...

skissane 6y ago

I believe you that isn't what it means, but for those (like myself) who have limited familiarity with this topic, could you please explain what it actually means.

mhh__ 6y ago
The p-value of a physics experiment isn't the probability of the result coming from thin air but rather than the probability of a result as or more extreme under the assumption of the null hypothesis (i.e. the effect we are looking for doesn't exist). The rub is that the design of the experiment and thus the hypothesis test is all dependent on various subjective factors, especially the people designing the experiment themselves. I'm not statistician however so presumably I'm just as wrong
Kaibeezy 6y ago
Good explanation of sigmas here: https://news.mit.edu/2012/explained-sigma-0209 https://news.mit.edu/2012/explained-sigma-0209 I’m thinking it would be more accurate (by being less precise) for BBC to describe the 1:1000 possibility as just coincidence rather than “statistical” coincidence.
- spuz 6y ago
  This article seems to agree that 3 sigma simply means "1 in 1000 possibility the result is a coincidence". I'm not sure what difference removing the word "statistical" makes here or why the BBC explanation is wrong.
  - Kaibeezy 6y ago
    Yeah, I’m not exactly getting why BBC is wrong either. Re “statistical”, to me, it sort of implies all sources of error or variability were provably not present (prove a negative) and the only reason for a coincidental result was a statistical anomaly. What’s got to be far more likely is some unaccounted-for variable produced the coincidental result.
  - trehalose 6y ago
    The article does seem to agree, and it's wrong just like the BBC article. Here's an old article from the time of the Higgs Boson discovery that had a great explanation: https://blogs.scientificamerican.com/observations/five-sigmawhats-that/ https://blogs.scientificamerican.com/observations/five-sigma... > Chances are, you heard this month about the discovery of a tiny fundamental physics particle that may be the long-sought Higgs boson. The phrase five-sigma was tossed about by scientists to describe the strength of the discovery. So, what does five-sigma mean? > In short, five-sigma corresponds to a p-value, or probability, of 3x10-7, or about 1 in 3.5 million. This is not the probability that the Higgs boson does or doesn't exist; rather, it is the probability that if the particle does not exist, the data that CERN scientists collected in Geneva, Switzerland, would be at least as extreme as what they observed. "The reason that it's so annoying is that people want to hear declarative statements, like 'The probability that there's a Higgs is 99.9 percent,' but the real statement has an 'if' in there. There's a conditional. There's no way to remove the conditional," says Kyle Cranmer, a physicist at New York University and member of the ATLAS team, one of the two groups that announced the new particle results in Geneva on July 4.
    - spuz 6y ago
      Thanks - I think that does explain some of the potential area of confusion. However, I don't agree that the BBC's explanation differs from what the true meaning is according to this Scientific American article says. The BBC article says: > The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence My interpretation of this is "If there was no new physics, then there is a 1/1000 chance the same result could be measured by coincidence." I understand why some people may interpret this as "There is a 1/1000 chance that there is no new physics" though.
stdbrouw 6y ago
The probability of winning the lottery if you're a cheater is high, but the probability of being a cheater if you win the lottery is low. More generally, the probability of A conditional on B is not the same as the probability of B conditional on A. Sometimes they're similar, but it depends on the pre-existing odds of A and B.
Mordisquitos 6y ago
In case it helps, following that logic, green jelly beans could cause acne and there would be "roughly a 5% chance that the measurement is a statistical coincidence": https://xkcd.com/882/ https://xkcd.com/882/

flukus 6y ago

How many iterations that could create this result do they run? 1 in 1000 sounds unlikely, but for all I know they have run this experiment 50,000 times.

DavidSJ 6y ago
I would guess there are many more than thousands of such experiments that have been run, testing all the various permutations of dozens (or more) of different beyond Standard Model physical theories.
- dukwon 6y ago
  Testing different models is done in the analysis of the data, not in running the experiment differently. The "experiment" (colliding protons together) has been repeated quadrillions of times at the LHC. But there are only a few parameters you can change about it, namely the energy and how focused the beams are. Altering the detectors to change/improve how data is collected is an expensive and slow process.
  - DavidSJ 6y ago
    Fair enough; what I meant is "multiple testing" in the context of a single overarching experiment, which is I believe what the comment I was replying to was getting at as well.
    - davrosthedalek 6y ago
      In any case, you have a point. Whether multiple analysis, multiple channels or multiple experiments, you have a look-elsewhere effect, and that can easily knock off one or two sigmas. If you look at a 10000 possibilities, you will a couple with 3 sigma difference, just by chance. LHC looks at MANY possibilities. For a while, they tried to keep track to quantify the look-elsewhere effect. I think they have given up on that.
dalewe 6y ago
Not sure I entirely understand your question but I'll try to explain a bit anyway: they're using a dataset of 9fb^-1. That means it includes 9 events for a crosssection of 1fb (femtobarn). Now the paper doesn't seem to include what crosssection their events have but if we assume something of the order of magnitude of 1pb then the dataset would contain around 9000 events of interest. Now all that doesn't really matter as this is already factored into the standard deviation of their result. They have enough data to say it's a 3 sigma (being the standard deviation) deviation from the standard model. Now this "1 in 1000" really just means that the null hypothesis (the Standard Model) says there's a 0.1% chance for that result to happen. Of course it could also be that they forgot to factor in some uncertainty of their measurements and their sigma is actually much bigger. But it surely won't be the size of their dataset as this is quite obvious and easy to take into account.

simiones 6y ago

Even after you explain it, I can barely grasp the significance. Given the expected level of understanding of someone reading a BBC article about this topic, I don't think they are wrong at all in explaining it that way - that is, I don't think their intended audience would take the message differently to any extent whatsoever if they had explained the way you do. To be more specific, to the average BBC reader, the phrases The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence. and The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance to have seen this data even if the particle doesn't exist. convey the exact same message, the first one just uses fewer words.

DavidSJ 6y ago
Look, that may be so, but if it is, then they should either omit such a line entirely so as not to mislead their readers, select better language so that we don't collapse concepts which are fundamentally, completely, different from each other, or they could try to educate the public on a distinction which is fundamental to science. I mean, we might as well say that the BBC should just use the words "million" and "billion" interchangeably because most readers have no sense of scale. Bottom line: thanks to this line, the reader is likely to think that new physics has very likely been discovered, when the opposite is the case.
- simiones 6y ago
  I don't agree. The whole paragraph makes that clear - 1 in 1000 is actually pretty good odds that this is just a coincidence, and most people should know this intuitively. > The measurement from LHCb is three-sigma - meaning there is roughly a one in 1,000 chance that the measurement is a statistical coincidence. So people should not get carried away by these findings, according to team leader Prof Chris Parkes, from the University of Manchester.
davrosthedalek 6y ago
The problem with it is that layman people believe that both of these statements are equal to "it's a 999/1000 chance that it was something else". Which is not true. The difference in these two formulations is that for b) it's explicit wrong, for a) this is arguably what the article says (which is not what the scientists say)

choeger 6y ago

I did not even see this at first. But they did really write that the chance of "no new physics" is 1:1000, right? So they gave "there is new physics" interpretation a huge probability. Seriously, I completely glanced over this. Slightly embarrassing. In any case, if the LHC runs thousands of such experiments, we do expect such signals, right?

dataflow 6y ago

I don't see how the BBC's statement can translate into P(hypothesis|data). To me it's P(data|H0). Consider this simpler example: "Scientists always assumed their coin was fair. Well, they flipped it 10 times, and it came up heads every time. This is a 3-sigmal result. There's a 1 in 1024 chance of this measurement being a statistical coincidence." The reader would take this "1 in 1024 chance of coincidence" to mean "if the coin was fair, they'd have a 1 in 1024 chance of seeing this event occur". That's P(data|fair) = P(10 heads|fair) = 1/1024. I don't see how you read that and translate it into P(fair|10 heads).

DavidSJ 6y ago
If I say: "there's a 25% chance that Pat is from North Dakota", I'm taking as a given that there is a person named Pat, and I'm talking about the chance that Pat, whom we know to exist, is from North Dakota. In this case, grammatically, they're talking about the chance that the measurement (data) is a statistical coincidence, as opposed to not being a statistical coincidence. They're not talking about the chance that the data exists, as opposed to not existing.
- dataflow 6y ago
  > If I say: "there's a 25% chance that Pat is from North Dakota", I'm taking as a given that there is a person named Pat, and I'm talking about the chance that Pat, whom we know to exist, is from North Dakota. Yes, and if I say "there's a 0.001% chance that these coin flip observations were coincidental", I'm taking it as a given that there were coin flips observed (indeed, there were!) I'm talking about the chance that the observations, which we (indeed) know to exist, were coincidental. That is... literally P(observations | null hypothesis) = P(10 heads | fair). It's clearly not P(null hypothesis | observations) = P(fair | 10 heads)... > In this case, grammatically, they're talking about the chance that the measurement (data) is a statistical coincidence, as opposed to not being a statistical coincidence. They're not talking about the chance that the data exists, as opposed to not existing. Which is... perfectly fine. The data clearly does exist (as would the coin flips in my example), so the chance of the data existing is 100%... there's nothing interesting to talk about there. But the chance of it having been coincidental is 0.001%. This all seems correct to me...