5 ms·

In that case report: "these results are consistent with model A and model B, since model A has been studied much more so we will stick with that for now". There

by nonbel 8y ago

In that case report: "these results are consistent with model A and model B, since model A has been studied much more so we will stick with that for now". There is no reason to include these additional (usually elaborate) steps. Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?

iainmerrick 8y ago
Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose? Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics? I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what? I don't think you can easily say "the Standard Model is incorrect" -- we know it's incomplete, hence incorrect, but it also fits experimental data incredibly well, so it must be a close approximation to some level of reality at the very least. (edit to clarify: what I mean is, I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)
- nonbel 8y ago
  >"In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error." I was talking about all the other types of observations used to support the standard model. Afaik the existence of sterile neutrinos would not change the predictions. >"Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?" It isn't really a matter of frequentist vs bayesian statistics. Frequentist stats looks like what you see in Figure 1 here (my venn diagram is just a simpler version of it): http://rsta.royalsocietypublishing.org/content/231/694-706/289 http://rsta.royalsocietypublishing.org/content/231/694-706/2... However, it is put forward that prior information should be involved in interpreting the results of the significance test. This idea has been rejected by all the giants and founders of modern stats (Fisher, Neyman, Pearson, etc). Its just some ad hoc thing come up with by stats 101 authors. >"I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what?" We're supposed to be comparing the predictions derived from our theories/models to data collected after the prediction was made and use these predictions to come up with useful tech and perform amazing feats. >"I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)" The current solution seems to be performing an elaborate ritual that obscures the true method of using collective opinion.
beojan 8y ago
The method described is the extremely standard Hypothesis Testing method. The null hypothesis is that there is no new physics, while the alternative is that there is some new physics. The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance. For instance, the Higgs is part of the Standard Model but for the Higgs discovery the null hypothesis was that the Higgs does not exist.
- graycat 8y ago
  Nice.
- nonbel 8y ago
  >"The method described is the extremely standard Hypothesis Testing method....The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance." Yes, I know. This isn't at all a positive thing as you seem to think...
  - beojan 8y ago
    What isn't? Hypothesis Testing or the 5σ standard?
    - nonbel 8y ago
      A more stringent significance level just means it costs more money or the data is much cheaper to collect. You can look up "NHST controversy", "NHST pseudoscience", etc to find many people discussing the problems for over 50 years. Here is a decent start: https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nh... I've been lately seeing that this practice is infecting particle physics, which is not a good sign. I do not think they are overwhelmed yet though. Here is how it has gone in every field so far (education research, psychology, medicine, etc): Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?

by nonbel 8y ago

iainmerrick 8y ago

Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence. In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error. And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose? Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics? I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what? I don't think you can easily say "the Standard Model is incorrect" -- we know it's incomplete, hence incorrect, but it also fits experimental data incredibly well, so it must be a close approximation to some level of reality at the very least. (edit to clarify: what I mean is, I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)

nonbel 8y ago
>"In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error." I was talking about all the other types of observations used to support the standard model. Afaik the existence of sterile neutrinos would not change the predictions. >"Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?" It isn't really a matter of frequentist vs bayesian statistics. Frequentist stats looks like what you see in Figure 1 here (my venn diagram is just a simpler version of it): http://rsta.royalsocietypublishing.org/content/231/694-706/289 http://rsta.royalsocietypublishing.org/content/231/694-706/2... However, it is put forward that prior information should be involved in interpreting the results of the significance test. This idea has been rejected by all the giants and founders of modern stats (Fisher, Neyman, Pearson, etc). Its just some ad hoc thing come up with by stats 101 authors. >"I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what?" We're supposed to be comparing the predictions derived from our theories/models to data collected after the prediction was made and use these predictions to come up with useful tech and perform amazing feats. >"I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)" The current solution seems to be performing an elaborate ritual that obscures the true method of using collective opinion.

beojan 8y ago

The method described is the extremely standard Hypothesis Testing method. The null hypothesis is that there is no new physics, while the alternative is that there is some new physics. The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance. For instance, the Higgs is part of the Standard Model but for the Higgs discovery the null hypothesis was that the Higgs does not exist.

graycat 8y ago
Nice.
nonbel 8y ago
>"The method described is the extremely standard Hypothesis Testing method....The only difference is that in particle physics the standard isn't p < 0.05 but a 5σ significance." Yes, I know. This isn't at all a positive thing as you seem to think...
- beojan 8y ago
  What isn't? Hypothesis Testing or the 5σ standard?
  - nonbel 8y ago
    A more stringent significance level just means it costs more money or the data is much cheaper to collect. You can look up "NHST controversy", "NHST pseudoscience", etc to find many people discussing the problems for over 50 years. Here is a decent start: https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nh... I've been lately seeing that this practice is infecting particle physics, which is not a good sign. I do not think they are overwhelmed yet though. Here is how it has gone in every field so far (education research, psychology, medicine, etc): Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct. The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?