6 ms·

If I understand your question right, you're suggesting model A (the Standard Model, say) and model B (your own pet theory) should be evaluated on an equal footi

If I understand your question right, you're suggesting model A (the Standard Model, say) and model B (your own pet theory) should be evaluated on an equal footing. In principle that's fair enough, but in practice, the Standard Model has already passed lots of tests -- it fits the results of many, many experiments -- so we already have a high confidence that it's correct (or rather, a very close approximation to the truth for all the experiments we're been able to try). So brand new model B does not start on an equal footing. The null hypothesis is "model A is correct". If there's some new experimental result that does not fit with the Standard Model -- and this usually not a clear-cut thing, but a statistical estimate of likelihood -- but does fit model B, that's a useful new result. If the error bars are such that it fits both models, you stick with the tried and tested model A, unless B has some compelling new advantage (like being much simpler than the Standard Model, say) and it also fits all other existing experimental data.

nonbel 8y ago
In that case report: "these results are consistent with model A and model B, since model A has been studied much more so we will stick with that for now". There is no reason to include these additional (usually elaborate) steps. Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?
- iainmerrick 8y ago
  Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose? Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics? I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what? I don't think you can easily say "the Standard Model is incorrect" -- we know it's incomplete, hence incorrect, but it also fits experimental data incredibly well, so it must be a close approximation to some level of reality at the very least. (edit to clarify: what I mean is, I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)
  - nonbel 8y ago
    >"In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error." I was talking about all the other types of observations used to support the standard model. Afaik the existence of sterile neutrinos would not change the predictions. >"Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?" It isn't really a matter of frequentist vs bayesian statistics. Frequentist stats looks like what you see in Figure 1 here (my venn diagram is just a simpler version of it): http://rsta.royalsocietypublishing.org/content/231/694-706/289 http://rsta.royalsocietypublishing.org/content/231/694-706/2... However, it is put forward that prior information should be involved in interpreting the results of the significance test. This idea has been rejected by all the giants and founders of modern stats (Fisher, Neyman, Pearson, etc). Its just some ad hoc thing come up with by stats 101 authors. >"I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what?" We're supposed to be comparing the predictions derived from our theories/models to data collected after the prediction was made and use these predictions to come up with useful tech and perform amazing feats. >"I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)" The current solution seems to be performing an elaborate ritual that obscures the true method of using collective opinion.
- beojan 8y ago
  The method described is the extremely standard Hypothesis Testing method. The null hypothesis is that there is no new physics, while the alternative is that there is some new physics. The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance. For instance, the Higgs is part of the Standard Model but for the Higgs discovery the null hypothesis was that the Higgs does not exist.
  - graycat 8y ago
    Nice.
  - nonbel 8y ago
    >"The method described is the extremely standard Hypothesis Testing method....The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance." Yes, I know. This isn't at all a positive thing as you seem to think...
    - beojan 8y ago
      What isn't? Hypothesis Testing or the 5σ standard?
      - nonbel 8y ago
        A more stringent significance level just means it costs more money or the data is much cheaper to collect. You can look up "NHST controversy", "NHST pseudoscience", etc to find many people discussing the problems for over 50 years. Here is a decent start: https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nh... I've been lately seeing that this practice is infecting particle physics, which is not a good sign. I do not think they are overwhelmed yet though. Here is how it has gone in every field so far (education research, psychology, medicine, etc): Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?
graycat 8y ago
Nice, really nice, introduction to statistical hypothesis testing, nicer, sorry 'bout that, than is common in teaching of statistics. Nice work by physics.
- nonbel 8y ago
  What are you saying is nice?
  - graycat 8y ago
    Some unusual clarity on how hypothesis testing works or should work. For one more point, we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true. So, what null hypotheses we can consider are both enabled and constrained by the assumptions to calculate the probabilities given the data.
    - nonbel 8y ago
      >"we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true." That is an awful reason to choose a null hypothesis. You should choose a null hypothesis you derived from your theory so you can see if your theory is capable of making accurate predictions.
      - beojan 8y ago
        Your null hypothesis shouldn't have anything to do with your theory. It should always be "nothing interesting (or new) is happening". Your alternative hypothesis is derived from your theory.
        nonbel 8y ago
        Yep, that's the "switcheroo" that reverses the long successful logic of science. I'd suggest starting with Meehl 1967: >"In the physical sciences, the usual result of an improvement in experimental design, instrumentation, or numerical mass of data, is to increase the difficulty of the “observational hurdle” which the physical theory of interest must successfully surmount; whereas, in psychology and some of the allied behavior sciences, the usual effect of such improvement in experimental precision is to provide an easier hurdle for the theory to surmount." https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...

nonbel 8y ago

In that case report: "these results are consistent with model A and model B, since model A has been studied much more so we will stick with that for now". There is no reason to include these additional (usually elaborate) steps. Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?

iainmerrick 8y ago
Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose? Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics? I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what? I don't think you can easily say "the Standard Model is incorrect" -- we know it's incomplete, hence incorrect, but it also fits experimental data incredibly well, so it must be a close approximation to some level of reality at the very least. (edit to clarify: what I mean is, I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)
- nonbel 8y ago
  >"In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error." I was talking about all the other types of observations used to support the standard model. Afaik the existence of sterile neutrinos would not change the predictions. >"Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?" It isn't really a matter of frequentist vs bayesian statistics. Frequentist stats looks like what you see in Figure 1 here (my venn diagram is just a simpler version of it): http://rsta.royalsocietypublishing.org/content/231/694-706/289 http://rsta.royalsocietypublishing.org/content/231/694-706/2... However, it is put forward that prior information should be involved in interpreting the results of the significance test. This idea has been rejected by all the giants and founders of modern stats (Fisher, Neyman, Pearson, etc). Its just some ad hoc thing come up with by stats 101 authors. >"I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what?" We're supposed to be comparing the predictions derived from our theories/models to data collected after the prediction was made and use these predictions to come up with useful tech and perform amazing feats. >"I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)" The current solution seems to be performing an elaborate ritual that obscures the true method of using collective opinion.
beojan 8y ago
The method described is the extremely standard Hypothesis Testing method. The null hypothesis is that there is no new physics, while the alternative is that there is some new physics. The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance. For instance, the Higgs is part of the Standard Model but for the Higgs discovery the null hypothesis was that the Higgs does not exist.
- graycat 8y ago
  Nice.
- nonbel 8y ago
  >"The method described is the extremely standard Hypothesis Testing method....The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance." Yes, I know. This isn't at all a positive thing as you seem to think...
  - beojan 8y ago
    What isn't? Hypothesis Testing or the 5σ standard?
    - nonbel 8y ago
      A more stringent significance level just means it costs more money or the data is much cheaper to collect. You can look up "NHST controversy", "NHST pseudoscience", etc to find many people discussing the problems for over 50 years. Here is a decent start: https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nh... I've been lately seeing that this practice is infecting particle physics, which is not a good sign. I do not think they are overwhelmed yet though. Here is how it has gone in every field so far (education research, psychology, medicine, etc): Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?

graycat 8y ago

Nice, really nice, introduction to statistical hypothesis testing, nicer, sorry 'bout that, than is common in teaching of statistics. Nice work by physics.

nonbel 8y ago
What are you saying is nice?
- graycat 8y ago
  Some unusual clarity on how hypothesis testing works or should work. For one more point, we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true. So, what null hypotheses we can consider are both enabled and constrained by the assumptions to calculate the probabilities given the data.
  - nonbel 8y ago
    >"we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true." That is an awful reason to choose a null hypothesis. You should choose a null hypothesis you derived from your theory so you can see if your theory is capable of making accurate predictions.
    - beojan 8y ago
      Your null hypothesis shouldn't have anything to do with your theory. It should always be "nothing interesting (or new) is happening". Your alternative hypothesis is derived from your theory.
      - nonbel 8y ago
        Yep, that's the "switcheroo" that reverses the long successful logic of science. I'd suggest starting with Meehl 1967: >"In the physical sciences, the usual result of an improvement in experimental design, instrumentation, or numerical mass of data, is to increase the difficulty of the “observational hurdle” which the physical theory of interest must successfully surmount; whereas, in psychology and some of the allied behavior sciences, the usual effect of such improvement in experimental precision is to provide an easier hurdle for the theory to surmount." https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theorytestingparadox.pdf https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...