11 ms·
> I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data.
You don'
> I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data.
You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis. Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true.
For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model.
>"For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model."
The data is consistent with the predictions of both models in this case. If we want to distinguish between them we need to get more/cleaner data or different data to compare to different predictions. What is wrong with that?
>"You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis."
This is not what I am suggesting to do, see above.
>"Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true."
What purpose does this serve?
EDIT:
Here is a venn diagram of what I mean, with four possibilities:
https://image.ibb.co/c432OT/venn_Science.png https://image.ibb.co/c432OT/venn_Science.png
You are saying for some reason it is important check for !B which equals regions A - (A or B) + !(A or B). Why?
Why not just check which corresponds to the results? Is it model A only, model B only, model A and model B, or neither model A nor model B?
If I understand your question right, you're suggesting model A (the Standard Model, say) and model B (your own pet theory) should be evaluated on an equal footing.
In principle that's fair enough, but in practice, the Standard Model has already passed lots of tests -- it fits the results of many, many experiments -- so we already have a high confidence that it's correct (or rather, a very close approximation to the truth for all the experiments we're been able to try). So brand new model B does not start on an equal footing. The null hypothesis is "model A is correct".
If there's some new experimental result that does not fit with the Standard Model -- and this usually not a clear-cut thing, but a statistical estimate of likelihood -- but does fit model B, that's a useful new result. If the error bars are such that it fits both models, you stick with the tried and tested model A, unless B has some compelling new advantage (like being much simpler than the Standard Model, say) and it also fits all other existing experimental data.
In that case report: "these results are consistent with model A and model B, since model A has been studied much more so we will stick with that for now". There is no reason to include these additional (usually elaborate) steps.
Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence.
And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?
Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence.
In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error.
And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?
Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?
I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what? I don't think you can easily say "the Standard Model is incorrect" -- we know it's incomplete, hence incorrect, but it also fits experimental data incredibly well, so it must be a close approximation to some level of reality at the very least.
(edit to clarify: what I mean is, I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)
>"In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error."
I was talking about all the other types of observations used to support the standard model. Afaik the existence of sterile neutrinos would not change the predictions.
>"Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?"
It isn't really a matter of frequentist vs bayesian statistics. Frequentist stats looks like what you see in Figure 1 here (my venn diagram is just a simpler version of it): http://rsta.royalsocietypublishing.org/content/231/694-706/289 http://rsta.royalsocietypublishing.org/content/231/694-706/2...
However, it is put forward that prior information should be involved in interpreting the results of the significance test. This idea has been rejected by all the giants and founders of modern stats (Fisher, Neyman, Pearson, etc). Its just some ad hoc thing come up with by stats 101 authors.
>"I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what?"
We're supposed to be comparing the predictions derived from our theories/models to data collected after the prediction was made and use these predictions to come up with useful tech and perform amazing feats.
>"I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)"
The current solution seems to be performing an elaborate ritual that obscures the true method of using collective opinion.
> "1. If you're using a data driven background, you're background model isn't the Standard Model but that there isn't something special about your signal region."
Sure, it means you assume that during the signal detection everything was working exactly as it was during background data collection. There are going to be other implementation specific assumptions like what thresholds to use regarding environment noise triggers, accounting for sensor drift, etc. These data-driven models are approximations, I doubt the people actually coming up with them really believe they are 100% true. If you give a real life example I will point out exactly where there are issues.
> "If you are using the SM as your background, then yes you do believe it to be 100% true"
I don't know what this is supposed to mean. Of course you are assuming your background is 100% true. Do particle physicists actually believe this though? Not from my reading:
"So although the Standard Model accurately describes the phenomena within its domain, it is still incomplete."
https://home.cern/about/physics/standard-model https://home.cern/about/physics/standard-model
Yes, I already said the background estimation can be wrong. We can check it by defining validation regions outside the signal region and making sure it correctly describes the data there (where there is no signal) and assign a conservative systematic uncertainty to cover any differences, but yes it can be wrong. This is a background estimation problem though, not a flaw with hypothesis testing. By the way, the signal region data and background data are collected simultaneously, so there can be no systematic difference in experiment conditions between the two.
As for the SM, we are looking for those places where it is incomplete, and to do this we must assume it is complete and look for strong evidence that we are wrong. To date, it doesn't seem like we see though, as far as collider physics is concerned.
> No, if you knew the null model was flawed to begin with nothing has been learned.
We don't know that the null is flawed to begin with. Not in a way that matters to the analysis anyway, otherwise you've picked the wrong null. No one runs analyses to discover the pion for the millionth time.
For BSM work, the way it generally works is this (ignoring limit setting):
- Pick a signature predicted by one or more BSM models (or even none), that either isn't predicted by the SM or is exceedingly rare in the SM. For example, you might predict a new particle that decays in some predetermined way.
- Define a signal region in some combination of variables where you expect to find a signal corresponding to this signature. This region is blinded, and you don't look at it until after fixing your entire procedure.
- Estimate the background in this signal region with a combination of Monte Carlo simulations and extrapolation from outside this region (data-driven backgrounds). This obviously has both statistical and systematic uncertainties, and a lot of the hard work is in getting this right. For instance, you might use independent Monte Carlo generators, or define control regions to check your estimation in.
- "Open the box" and look at the signal region data. If you see an excess over the estimated background, calculate the significance. Even here, if you were looking for a new particle, for instance, and the excess isn't a localized bump, this would be an indication that the background estimation may be flawed.
- If the significance is over 5 sigma, you have a discovery.
As you can see, you aren't using a null hypothesis that you already know is flawed. Your null is specifically "no signal exists". A positive deviation that isn't a statistical fluctuation is by definition a discovery of a signal.
Things like malfunctioning equipment go into the uncertainties if they occur in a way that the researchers have considered (which often means uncertainties are set conservatively if the equipment behaviour is poorly understood). If they occur in a way that no one considered, there's no statistical trickery that's every going to compensate for that. If we just went straight to setting limits, we would still see a deviation from what's expected there if equipment malfunctioned in a manner that faked a signal.
> In summary: If there is anything wrong with the null/background model at all, the p-value will converge on zero, it is just a matter of collecting enough data.
In summary: Assuming the background estimation is correct, the p-value will only converge to zero if a true signal exists. If the background estimation is wrong, then the background estimation is wrong and this is a problem with the background estimation, not hypothesis testing.
I'm by no means claiming the system is perfect, but it isn't systematically flawed the way you seem to be claiming.