5 ms·

I am under the impression that over 3 sigma suggests strong evidence for new physics but not enough to be treated as new physics with high probability, which is

I am under the impression that over 3 sigma suggests strong evidence for new physics but not enough to be treated as new physics with high probability, which is done at 5 sigma. However, all the numbers are, well, arbitrary. By that I mean that there is nothing special about 3 sigma or 5 sigma or p-value less than 0.05. The actual value is arbitrary and used because it has high inertia, i.e. commonly used in literature.

enkid 6y ago
If I understand correctly, "certainly new physics" is not correct. It makes the results very substantial, but you still have systemic things to consider at that point, like misapplication of a mathematical concept that no one has caught, or instrument error.
- PartiallyTyped 6y ago
  That's correct. I should have said, new physics with high confidence. Physicists tend to be very very conservative when it comes to discoveries as there have been some big blunders during the last century.
leereeves 6y ago
In a normal distribution, about 3 out of 1000 values will be over 3 sigma just by random chance. How many measurements do they make at CERN? They probably see random 3 sigma measurements fairly often.
- PartiallyTyped 6y ago
  That's the reason it is expected to have 5 sigma, so that anomalies are extremely rare. The more experiments one makes, the closer one gets to the mean value and this happens exponentially fast. By that, I mean that using some probability theory and some algebra you can get a confidence bound on the accuracy you demand based on the number of experiments that you run.
  - jampekka 6y ago
    I think this happens sublinearly w.r.t sample size.
    - PartiallyTyped 6y ago
      I would like to point you towards Hoeffding's inequality. The probability of deviating more than epsilon from the mean after n experiments is exponential in n.
      - leereeves 6y ago
        I think you're talking about different things. For a fixed range, the probability of being outside that range decays like 1/k^n. But for a fixed probability, the width of the range with that probability decays like 1/sqrt(n).
marcosdumay 6y ago
Due to the sheer number of hypotheses get tested all the time in particle physics, 3 sigma effects appear all the time, and then disappear before getting a larger certainty. That's why it's considered a suggestion, not proof. And yes, the exact numbers are arbitrary, but their ballpark isn't. You reduce the necessary certainty if you have confidence on your priors, and increase it if you test many different hypotheses. The target confidence also varies from one discipline to another based on how much data one can realistically gather, but nearly all of physics falls on the "we can gather enough data" category anyway.
- ramraj07 6y ago
  Well put, but I’ve always felt a middle ground existed where a knowledgeable scientist can make a very educated guess with much lesser statistical significance as long as the experiment is thought through, you are aware of the noise models and alternative explanations have been explored and mostly ruled out. Of course for publication you better whip your ass to whatever sigma they demand, but more often than not there either cause for optimism or pessimism ways before that point.
  - marcosdumay 6y ago
    Not really. The amount of confidence you have grows exponentially with the observed deviation, and that last one grows kinda proportionally to the amount of data you have (if the hypothesis is real, of course). So there is some fine adjustment possible, but it's very small on the experiments where you need a lot of confidence. There is more margin to judge low non-significant results at the low confidence end, as exponentials grow slowly when they are small.
- panda-giddiness 6y ago
  > Due to the sheer number of hypotheses get tested all the time in particle physics, 3 sigma effects appear all the time, and then disappear before getting a larger certainty. See, eg, the look-elsewhere effect [1] or the problem of multiple comparisons [2]. --- [1] https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect [2] https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem

enkid 6y ago

If I understand correctly, "certainly new physics" is not correct. It makes the results very substantial, but you still have systemic things to consider at that point, like misapplication of a mathematical concept that no one has caught, or instrument error.

PartiallyTyped 6y ago
That's correct. I should have said, new physics with high confidence. Physicists tend to be very very conservative when it comes to discoveries as there have been some big blunders during the last century.

leereeves 6y ago

In a normal distribution, about 3 out of 1000 values will be over 3 sigma just by random chance. How many measurements do they make at CERN? They probably see random 3 sigma measurements fairly often.

PartiallyTyped 6y ago
That's the reason it is expected to have 5 sigma, so that anomalies are extremely rare. The more experiments one makes, the closer one gets to the mean value and this happens exponentially fast. By that, I mean that using some probability theory and some algebra you can get a confidence bound on the accuracy you demand based on the number of experiments that you run.
- jampekka 6y ago
  I think this happens sublinearly w.r.t sample size.
  - PartiallyTyped 6y ago
    I would like to point you towards Hoeffding's inequality. The probability of deviating more than epsilon from the mean after n experiments is exponential in n.
    - leereeves 6y ago
      I think you're talking about different things. For a fixed range, the probability of being outside that range decays like 1/k^n. But for a fixed probability, the width of the range with that probability decays like 1/sqrt(n).

marcosdumay 6y ago

Due to the sheer number of hypotheses get tested all the time in particle physics, 3 sigma effects appear all the time, and then disappear before getting a larger certainty. That's why it's considered a suggestion, not proof. And yes, the exact numbers are arbitrary, but their ballpark isn't. You reduce the necessary certainty if you have confidence on your priors, and increase it if you test many different hypotheses. The target confidence also varies from one discipline to another based on how much data one can realistically gather, but nearly all of physics falls on the "we can gather enough data" category anyway.

ramraj07 6y ago
Well put, but I’ve always felt a middle ground existed where a knowledgeable scientist can make a very educated guess with much lesser statistical significance as long as the experiment is thought through, you are aware of the noise models and alternative explanations have been explored and mostly ruled out. Of course for publication you better whip your ass to whatever sigma they demand, but more often than not there either cause for optimism or pessimism ways before that point.
- marcosdumay 6y ago
  Not really. The amount of confidence you have grows exponentially with the observed deviation, and that last one grows kinda proportionally to the amount of data you have (if the hypothesis is real, of course). So there is some fine adjustment possible, but it's very small on the experiments where you need a lot of confidence. There is more margin to judge low non-significant results at the low confidence end, as exponentials grow slowly when they are small.
panda-giddiness 6y ago
> Due to the sheer number of hypotheses get tested all the time in particle physics, 3 sigma effects appear all the time, and then disappear before getting a larger certainty. See, eg, the look-elsewhere effect [1] or the problem of multiple comparisons [2]. --- [1] https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect [2] https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem