3 ms·

Oof, p value misinterpretation: > That result got the community excited — although most physicists were quick to point out that with such a small sample size,

Oof, p value misinterpretation: > That result got the community excited — although most physicists were quick to point out that with such a small sample size, there was still a 10 percent chance that the difference was merely a random fluctuation. (By comparison, the 2012 Higgs boson discovery had less than a 1-in-1 million probability that the signal was due to chance.)

nonbel 9y ago
Yep. When reading the actual papers from the LHC, LIGO, etc groups the p-value/significance aspect seems nearly irrelevant though. My impression is usually they have some set of possible theories that predict the exact result observed, then go on to do more experiments to distinguish between the predictions of those. Really they should just drop the significance testing because it adds nothing and they don't seem any better at interpreting those tests than other fields. If I am wrong about all this and that aspect actually is crucial, then physics is in trouble...
- QAPereo 9y ago
  In HEP, it’s more that the bar is so damned high, and your n equals something in the billions or trillions, and oversight from the community is extreme. The confidence required for example, to make people feel generally positive about the discovery of the Higgs boson, was in the realm of billions to one. Generally there is a model, and dats needs to fit that model to what would be an insane degree of confidence in most other fields. There is a bit more to it than just pure sigma values; it is also a matter of how closely the data fits the model overall, a lack of competing hypotheses to explain said data, and to what degree other evidence exists to support the model. Of course, if the theory being confirmed is complementary to existing physics, that also helps a lot. Having said that, 3 sigma is about when people started to get excited about the Higgs, and 5 sigma was confirmation.
  - sandworm101 9y ago
    5-sigma is a ridiculous level. Im not that confident of my own name, which i spell incorrectly on at least two of every billion forms i fill.
  - nonbel 9y ago
    The problem is that the sigma calculation is only about rejecting the null model (ie, the model of "background noise"). That number is calculated without including info about any theory/evidence regarding the Higgs boson. So, all of the confidence and confirmation regarding the Higg's boson better actually be coming from "how closely the data fits the model overall, a lack of competing hypotheses to explain said data, and to what degree other evidence exists to support the model". As an example, I'd imagine detecting the Higgs boson at 1 eV energy levels is theoretically predicted to be even less likely than billions/trillions to one odds, therefore detector noise would be a more likely explanation for such results, despite the low p-value.
    - mattheww 9y ago
      >That number is calculated without including info about any theory/evidence regarding the Higg's [sic] boson. That's not true. You can see plots in the 2011 and 2012 papers that give these calculations as a function of mass. It's basically impossible to make a mass-independent calculation. (Not withstanding the fact that there's no Higgsless theory to do calculations for the background). >As an example, I'd imagine detecting the Higg's [sic] boson at 1 eV energy levels is theoretically predicted to be even less likely than billions/trillions to one odds, therefore detector noise would be a more likely explanation for such results, despite the low p-value. You're right, the p-value is much lower. Previous experiments have long since excluded such a low Higgs mass. Also, if the Higgs mass were so low, we probably wouldn't exist.
      - nonbel 9y ago
        1) Typos fixed. 2) "That's not true. You can see plots in the 2011 and 2012 papers that give these calculations as a function of mass." - Can you explain what you mean via figure 1 in this paper: https://arxiv.org/abs/1207.7235 https://arxiv.org/abs/1207.7235 ? - I don't see the relevance of calculating p-values as a function of mass to my comment 3) "Not withstanding the fact that there's no Higgsless theory to do calculations for the background" - Then what model did they use to calculate the p-values? (I do not know the details but am fairly certain it is one where there is no Higgs boson at any given mass) 4) "Also, if the Higgs mass were so low, we probably wouldn't exist." - Ok, but I've also read headlines like "CERN proves the universe shouldn't exist", clearly this is just because their model of the universe is wrong. I'm sure in a pinch people could come up with some kind of balancing out of whatever problems would arise from such a small Higgs mass. The point was that assuming the current theory is correct, the Higgs would be a much worse explanation than detector noise.
- haZard_OS 9y ago
  We should just drop the significance testing/reporting period. The ASA's recent statement on p-values is spot on: http://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf http://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
  - nonbel 9y ago
    Significance testing is probably fine as long as you put your theoretical prediction as the null hypothesis.
- posterboy 9y ago
  Measurements are inherently probabilistic. If there was no trouble, there would be no physics.

nonbel 9y ago

Yep. When reading the actual papers from the LHC, LIGO, etc groups the p-value/significance aspect seems nearly irrelevant though. My impression is usually they have some set of possible theories that predict the exact result observed, then go on to do more experiments to distinguish between the predictions of those. Really they should just drop the significance testing because it adds nothing and they don't seem any better at interpreting those tests than other fields. If I am wrong about all this and that aspect actually is crucial, then physics is in trouble...

QAPereo 9y ago
In HEP, it’s more that the bar is so damned high, and your n equals something in the billions or trillions, and oversight from the community is extreme. The confidence required for example, to make people feel generally positive about the discovery of the Higgs boson, was in the realm of billions to one. Generally there is a model, and dats needs to fit that model to what would be an insane degree of confidence in most other fields. There is a bit more to it than just pure sigma values; it is also a matter of how closely the data fits the model overall, a lack of competing hypotheses to explain said data, and to what degree other evidence exists to support the model. Of course, if the theory being confirmed is complementary to existing physics, that also helps a lot. Having said that, 3 sigma is about when people started to get excited about the Higgs, and 5 sigma was confirmation.
- sandworm101 9y ago
  5-sigma is a ridiculous level. Im not that confident of my own name, which i spell incorrectly on at least two of every billion forms i fill.
- nonbel 9y ago
  The problem is that the sigma calculation is only about rejecting the null model (ie, the model of "background noise"). That number is calculated without including info about any theory/evidence regarding the Higgs boson. So, all of the confidence and confirmation regarding the Higg's boson better actually be coming from "how closely the data fits the model overall, a lack of competing hypotheses to explain said data, and to what degree other evidence exists to support the model". As an example, I'd imagine detecting the Higgs boson at 1 eV energy levels is theoretically predicted to be even less likely than billions/trillions to one odds, therefore detector noise would be a more likely explanation for such results, despite the low p-value.
  - mattheww 9y ago
    >That number is calculated without including info about any theory/evidence regarding the Higg's [sic] boson. That's not true. You can see plots in the 2011 and 2012 papers that give these calculations as a function of mass. It's basically impossible to make a mass-independent calculation. (Not withstanding the fact that there's no Higgsless theory to do calculations for the background). >As an example, I'd imagine detecting the Higg's [sic] boson at 1 eV energy levels is theoretically predicted to be even less likely than billions/trillions to one odds, therefore detector noise would be a more likely explanation for such results, despite the low p-value. You're right, the p-value is much lower. Previous experiments have long since excluded such a low Higgs mass. Also, if the Higgs mass were so low, we probably wouldn't exist.
    - nonbel 9y ago
      1) Typos fixed. 2) "That's not true. You can see plots in the 2011 and 2012 papers that give these calculations as a function of mass." - Can you explain what you mean via figure 1 in this paper: https://arxiv.org/abs/1207.7235 https://arxiv.org/abs/1207.7235 ? - I don't see the relevance of calculating p-values as a function of mass to my comment 3) "Not withstanding the fact that there's no Higgsless theory to do calculations for the background" - Then what model did they use to calculate the p-values? (I do not know the details but am fairly certain it is one where there is no Higgs boson at any given mass) 4) "Also, if the Higgs mass were so low, we probably wouldn't exist." - Ok, but I've also read headlines like "CERN proves the universe shouldn't exist", clearly this is just because their model of the universe is wrong. I'm sure in a pinch people could come up with some kind of balancing out of whatever problems would arise from such a small Higgs mass. The point was that assuming the current theory is correct, the Higgs would be a much worse explanation than detector noise.
haZard_OS 9y ago
We should just drop the significance testing/reporting period. The ASA's recent statement on p-values is spot on: http://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf http://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
- nonbel 9y ago
  Significance testing is probably fine as long as you put your theoretical prediction as the null hypothesis.
posterboy 9y ago
Measurements are inherently probabilistic. If there was no trouble, there would be no physics.