9 ms·

I'm a high school science teacher. Could you point me to the typical methods you use to calculate the level of confidence of results at CERN?

by drsopp 8y ago

cshimmin 8y ago
Sure thing! In broad strokes, the most formal way (in our field's typically frequentist paradigm) to calculate confidence/significance is to build a robust statistical model for your entire experiment (including the random effect that various uncertainties, etc will have on the outcome), and then to randomly synthesize many many outcomes of your experiment with respect to a given hypothesis. The outcome of each "pseudoexperiment" is boiled down to a single number, known as a test-statistic, and this way we can generate a distribution of that test statistic that gives you an idea of the probability of any given outcome. Then, when we do the experiment for real and get a single outcome, we can evaluate the compatibility of that outcome with the distribution of possible outcomes sampled from the statistical model (e.g. by calculating a p-value). Generally the Profile Likelihood Ratio is chosen as the test statistic at LHC experiments. This is possibly the most highly-cited statistics paper in our field: https://arxiv.org/abs/1007.1727 https://arxiv.org/abs/1007.1727 It provides asymptotic formulae that allow us to estimate these p-values without having to simulate our experiment tens of thousands of times. But it also gives a nice overview of the different test statistics one might use. "LHC Statistics for Pedestrians" is short and informative, but a bit haphazard and probably assumes you're familiar with a lot of jargon: https://cds.cern.ch/record/1099994/files/p205.pdf https://cds.cern.ch/record/1099994/files/p205.pdf "Practical Statistics for the LHC" is a much more complete and thorough introduction, but it's quite long and technical: https://arxiv.org/abs/1503.07622 https://arxiv.org/abs/1503.07622 "Statistics for searches at the LHC": https://arxiv.org/abs/1307.2487 https://arxiv.org/abs/1307.2487 Similar to the previous one, but maybe more words and less math. Perhaps one of these will resonate with you more than the others! Lastly, here is a public note that was created jointly by the ATLAS and CMS collaborations prior to the Higgs boson discovery. Basically, we got together and came up with a consistent way to present new results from the LHC, and this note documents those protocols: https://cds.cern.ch/record/1379837 https://cds.cern.ch/record/1379837
- drsopp 8y ago
  Wow! This is amazing. Thank you!
- nonbel 8y ago
  >"Question number one would be: Did I or did I not establish a discovery? Question number two would be: How well does my alternate model describe this discovery?" https://cds.cern.ch/record/1099994/files/p205.pdf https://cds.cern.ch/record/1099994/files/p205.pdf I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data. EDIT: For example, with the famous "sun's gravity bending starlight" example they compared the observations with predictions due to Newtonian gravity x with predictions due to Relativity 2x, there was no need to check for zero deflection since there was no theory predicting that. However, if some value (including zero) inconsistent with both was observed a new theory would need to be developed. Then that theory would need to be tested on some other phenomenon.
  - beojan 8y ago
    > I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data. You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis. Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true. For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model.
    - nonbel 8y ago
      >"For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model." The data is consistent with the predictions of both models in this case. If we want to distinguish between them we need to get more/cleaner data or different data to compare to different predictions. What is wrong with that? >"You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis." This is not what I am suggesting to do, see above. >"Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true." What purpose does this serve? EDIT: Here is a venn diagram of what I mean, with four possibilities: https://image.ibb.co/c432OT/venn_Science.png https://image.ibb.co/c432OT/venn_Science.png You are saying for some reason it is important check for !B which equals regions A - (A or B) + !(A or B). Why? Why not just check which corresponds to the results? Is it model A only, model B only, model A and model B, or neither model A nor model B?

I'm a high school science teacher. Could you point me to the typical methods you use to calculate the level of confidence of results at CERN?

by drsopp 8y ago

I'm a high school science teacher. Could you point me to the typical methods you use to calculate the level of confidence of results at CERN?

cshimmin 8y ago

Sure thing! In broad strokes, the most formal way (in our field's typically frequentist paradigm) to calculate confidence/significance is to build a robust statistical model for your entire experiment (including the random effect that various uncertainties, etc will have on the outcome), and then to randomly synthesize many many outcomes of your experiment with respect to a given hypothesis. The outcome of each "pseudoexperiment" is boiled down to a single number, known as a test-statistic, and this way we can generate a distribution of that test statistic that gives you an idea of the probability of any given outcome. Then, when we do the experiment for real and get a single outcome, we can evaluate the compatibility of that outcome with the distribution of possible outcomes sampled from the statistical model (e.g. by calculating a p-value). Generally the Profile Likelihood Ratio is chosen as the test statistic at LHC experiments. This is possibly the most highly-cited statistics paper in our field: https://arxiv.org/abs/1007.1727 https://arxiv.org/abs/1007.1727 It provides asymptotic formulae that allow us to estimate these p-values without having to simulate our experiment tens of thousands of times. But it also gives a nice overview of the different test statistics one might use. "LHC Statistics for Pedestrians" is short and informative, but a bit haphazard and probably assumes you're familiar with a lot of jargon: https://cds.cern.ch/record/1099994/files/p205.pdf https://cds.cern.ch/record/1099994/files/p205.pdf "Practical Statistics for the LHC" is a much more complete and thorough introduction, but it's quite long and technical: https://arxiv.org/abs/1503.07622 https://arxiv.org/abs/1503.07622 "Statistics for searches at the LHC": https://arxiv.org/abs/1307.2487 https://arxiv.org/abs/1307.2487 Similar to the previous one, but maybe more words and less math. Perhaps one of these will resonate with you more than the others! Lastly, here is a public note that was created jointly by the ATLAS and CMS collaborations prior to the Higgs boson discovery. Basically, we got together and came up with a consistent way to present new results from the LHC, and this note documents those protocols: https://cds.cern.ch/record/1379837 https://cds.cern.ch/record/1379837

drsopp 8y ago
Wow! This is amazing. Thank you!
nonbel 8y ago
>"Question number one would be: Did I or did I not establish a discovery? Question number two would be: How well does my alternate model describe this discovery?" https://cds.cern.ch/record/1099994/files/p205.pdf https://cds.cern.ch/record/1099994/files/p205.pdf I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data. EDIT: For example, with the famous "sun's gravity bending starlight" example they compared the observations with predictions due to Newtonian gravity x with predictions due to Relativity 2x, there was no need to check for zero deflection since there was no theory predicting that. However, if some value (including zero) inconsistent with both was observed a new theory would need to be developed. Then that theory would need to be tested on some other phenomenon.
- beojan 8y ago
  > I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data. You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis. Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true. For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model.
  - nonbel 8y ago
    >"For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model." The data is consistent with the predictions of both models in this case. If we want to distinguish between them we need to get more/cleaner data or different data to compare to different predictions. What is wrong with that? >"You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis." This is not what I am suggesting to do, see above. >"Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true." What purpose does this serve? EDIT: Here is a venn diagram of what I mean, with four possibilities: https://image.ibb.co/c432OT/venn_Science.png https://image.ibb.co/c432OT/venn_Science.png You are saying for some reason it is important check for !B which equals regions A - (A or B) + !(A or B). Why? Why not just check which corresponds to the results? Is it model A only, model B only, model A and model B, or neither model A nor model B?