7 ms·

>"Question number one would be: Did I or did I not establish a discovery? Question number two would be: How well does my alternate model describe this discovery

• beojan 8y ago
  > I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data. You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis. Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true. For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model.
  • nonbel 8y ago
    >"For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model." The data is consistent with the predictions of both models in this case. If we want to distinguish between them we need to get more/cleaner data or different data to compare to different predictions. What is wrong with that? >"You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis." This is not what I am suggesting to do, see above. >"Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true." What purpose does this serve? EDIT: Here is a venn diagram of what I mean, with four possibilities: https://image.ibb.co/c432OT/venn_Science.png https://image.ibb.co/c432OT/venn_Science.png You are saying for some reason it is important check for !B which equals regions A - (A or B) + !(A or B). Why? Why not just check which corresponds to the results? Is it model A only, model B only, model A and model B, or neither model A nor model B?
    • iainmerrick 8y ago
      If I understand your question right, you're suggesting model A (the Standard Model, say) and model B (your own pet theory) should be evaluated on an equal footing. In principle that's fair enough, but in practice, the Standard Model has already passed lots of tests -- it fits the results of many, many experiments -- so we already have a high confidence that it's correct (or rather, a very close approximation to the truth for all the experiments we're been able to try). So brand new model B does not start on an equal footing. The null hypothesis is "model A is correct". If there's some new experimental result that does not fit with the Standard Model -- and this usually not a clear-cut thing, but a statistical estimate of likelihood -- but does fit model B, that's a useful new result. If the error bars are such that it fits both models, you stick with the tried and tested model A, unless B has some compelling new advantage (like being much simpler than the Standard Model, say) and it also fits all other existing experimental data.
      • nonbel 8y ago
        In that case report: "these results are consistent with model A and model B, since model A has been studied much more so we will stick with that for now". There is no reason to include these additional (usually elaborate) steps. Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?
        • iainmerrick 8y ago
          Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose? Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics? I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what? I don't think you can easily say "the Standard Model is incorrect" -- we know it's incomplete, hence incorrect, but it also fits experimental data incredibly well, so it must be a close approximation to some level of reality at the very least. (edit to clarify: what I mean is, I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)