5 ms·
I think you're right that that's one of the chief concerns, but I think you paint it incorrectly as a technical problem. ("Multiple comparisons" as you call it
by jessriedel 11y ago
I think you're right that that's one of the chief concerns, but I think you paint it incorrectly as a technical problem. ("Multiple comparisons" as you call it. Particle physicists call it the "look-elsewhere effect": https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect )
The easy, though tedious, part of the look-elsewhere effect is to calculate a probabilisitic correction for all the various combinatorics of X objects in Y orbits over a time period of length Z that one might have seen. The hard, and borderline philosophical, part is to identify the proper reference class, i.e., what qualitatively different sorts of observations might have also considered evidence for planet nine, and how many other astrophysical searches are we running that might have given us weird results? This is really where frequentist approaches strain to be useful, and it's hard to avoid resorting to a Bayesian prior.
Analogously for particle physics, it's easy to compute the chance of getting a 4-sigma deviation for two different experiments in any gamma-gamma energy bins, so the after-the-fact corrections to the significant of a fluctuation in a particular bin are easy to compute. Much more difficult is to consider all the hundreds of parallel searches going on, of greatly varying levels of motivation, and correcting based on that.
- jerf 11y agoI'd also observe that in terms of prior probability, "There is a ninth planet in the solar system that we have not yet detected" is probably hovering at least in the low-integer percentages, and I imagine some scientists would place it higher than that. The general idea is not a wild, far-out one by any stretch of the imagination.
- XorNot 11y agoThat isn't how probability works though. You have a Bayesian priors problem - you can't establish what the initial probabilities are.
- jerf 11y agoI can put some bounds on them, which is all I tried to do. It is certainly not reasonable to act as if the idea there is a ninth undetected planet is some sort of one-in-a-trillion possibility. I'm not even convinced you're correct anyhow; since I tend to accept the interpretation of probability as a statement of our knowledge of the world as good enough most of the time, we can establish a reasonable probability statement that expresses that uncertainty. We don't know everything about the universe or even our solar system but there is something about this topic that makes people start grossly overstating our ignorance.