5 ms·

> Batygin and Brown inferred its presence from the peculiar clustering of six previously known objects that orbit beyond Neptune. This raised a big red flag in

> Batygin and Brown inferred its presence from the peculiar clustering of six previously known objects that orbit beyond Neptune. This raised a big red flag in my mind. This must produce a literally astronomical multiple comparisons problem. Yes they reported sigma = 3.8, but if they didn't do their multiple comparisons correction right (which I am in no position to determine), they're basically reading tea leaves. If you're not familiar with multiple comparisons, it's kind of like [this](https://www.goodreads.com/quotes/649893-you-know-the-most-amazing-thing-happened-to-me-tonight https://www.goodreads.com/quotes/649893-you-know-the-most-am...) or [this](https://xkcd.com/882/ https://xkcd.com/882/). If you look at enough extra-neptunian bodies, some of them are going to be in an odd looking cluster.

jessriedel 11y ago
I think you're right that that's one of the chief concerns, but I think you paint it incorrectly as a technical problem. ("Multiple comparisons" as you call it. Particle physicists call it the "look-elsewhere effect": https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect ) The easy, though tedious, part of the look-elsewhere effect is to calculate a probabilisitic correction for all the various combinatorics of X objects in Y orbits over a time period of length Z that one might have seen. The hard, and borderline philosophical, part is to identify the proper reference class, i.e., what qualitatively different sorts of observations might have also considered evidence for planet nine, and how many other astrophysical searches are we running that might have given us weird results? This is really where frequentist approaches strain to be useful, and it's hard to avoid resorting to a Bayesian prior. Analogously for particle physics, it's easy to compute the chance of getting a 4-sigma deviation for two different experiments in any gamma-gamma energy bins, so the after-the-fact corrections to the significant of a fluctuation in a particular bin are easy to compute. Much more difficult is to consider all the hundreds of parallel searches going on, of greatly varying levels of motivation, and correcting based on that.
- jerf 11y ago
  I'd also observe that in terms of prior probability, "There is a ninth planet in the solar system that we have not yet detected" is probably hovering at least in the low-integer percentages, and I imagine some scientists would place it higher than that. The general idea is not a wild, far-out one by any stretch of the imagination.
  - XorNot 11y ago
    That isn't how probability works though. You have a Bayesian priors problem - you can't establish what the initial probabilities are.
    - jerf 11y ago
      I can put some bounds on them, which is all I tried to do. It is certainly not reasonable to act as if the idea there is a ninth undetected planet is some sort of one-in-a-trillion possibility. I'm not even convinced you're correct anyhow; since I tend to accept the interpretation of probability as a statement of our knowledge of the world as good enough most of the time, we can establish a reasonable probability statement that expresses that uncertainty. We don't know everything about the universe or even our solar system but there is something about this topic that makes people start grossly overstating our ignorance.
politician 11y ago
Keep reading, their simulation explains several additional previously unrelated trajectories.
stdbrouw 11y ago
The anomaly itself does not seem to be disputed. It's true that when looking for arbitrary clusters and unusual patterns you can pretty much always find something -- though it's more of a Texas Sharpshooter fallacy [1] than the kind of multiple comparisons you see in e.g. medical research -- but the paper in question [2] mentions a whole bunch of alternative theories for why the particular clustering of a bunch of rocks in the Kuiper belt is what it is, and it doesn't seem very likely to me that none of those researchers ever thought "well, gee, maybe this is all just a fluke." In fact the competing hypotheses would be my main source of skepticism – maybe it's a planet, maybe it's an entirely different phenomenon. [1] https://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy https://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy [2] http://iopscience.iop.org/article/10.3847/0004-6256/151/2/22;jsessionid=9DAB98EED9CB30448604A2F4CA0F8752.c5.iopscience.cld.iop.org http://iopscience.iop.org/article/10.3847/0004-6256/151/2/22...
logicallee 11y ago
So I might not have a great enough understanding but isn't 3.8 sigma not high for this kind of announcement (with the current title)? if someone wants to report finding a planet in the solar system ten times the size of Earth wouldn't they do so at more than @3.8 sigma. In particular, the prior is quite low because it would affect a bunch of other astronomical calculations, wouldn't it? So you would want the confidence that much higher.... http://www.dummies.com/how-to/content/how-to-calculate-six-sigma-quality.html http://www.dummies.com/how-to/content/how-to-calculate-six-s... What I mean is that in human terms, Google says the average man is 5'9" with standard deviation 2.9 inches, then 3.8 sigma (the chances of this happening by chance) are the same as the chances of finding someone 6'8". It doesn't seem unbelievable - https://answers.yahoo.com/question/index?qid=20101208183308AA4LYym https://answers.yahoo.com/question/index?qid=20101208183308A... If a woman is set up on a blind date and told the guy is 6'8" would she find it like not credible? My understnading of statistics then is that the idea that there just so happens to be a planet ten times the size of Earth in our solar system is pretty unbelievable. So I'd want the chances of it being an experimental fluke to be lower than seeing a 6'8" guy, anywhere, at any time, in any context. Because I would think that's what happens when you look at enough data. - Aren't the chances of this being completely spurious literally exactly the same as a researcher saying "I saw a 6'8" guy somewhere", meaning anywhere, in any context? Wouldn't you want higher evidence before stating "evidence found for ninth planet in our solar system, ten times the mass of Earth"?
- umanwizard 11y ago
  I wish people wouldn't downvote you without explaining why you're wrong. I have zero education in statistics and your comment makes intuitive sense to me so I'd like to learn more.
  - logicallee 11y ago
    Before your comment, I edited it to be much more civil and less presumptuous, my original phrasing had included "3.8 sigma is fuck-all", I was briefer, more resolute, and had just included my first link. (You can see evidence of my original phrasing if you remove the hedging I added.) But the downvotes meant I'm probably wrong - rather than delete I edited it to be much nicer so I could find out why I'm wrong.
Steuard 11y ago
I haven't seen the original research myself to judge, but it sounds like they're being careful about these effects. Looking at their discussion on the main page of http://www.findplanetnine.com/ http://www.findplanetnine.com/, the claim seems to be that this isn't "we spotted six out of the solar system's zillions of objects that formed a cluster" but rather "the unique six known objects within a certain range of orbital parameters all form a cluster in a different parameter". More specifically, quoting from that page, "all Kuiper belt objects with orbits that do not veer into inter-planetary space and spend longer than approximately 2000 years to complete a single revolution around the Sun... trace out elliptical paths that point into approximately the same direction in physical space, and lie in approximately the same plane." That could be just an observational coincidence, and that's where they claim the statistical figure that you've cited. I have no idea to what extent those figures account for the multiple comparisons issue, but the range of orbital parameters that I've quoted here doesn't look very fine-tuned to me. (The upper bound certainly isn't, anyway!) So it's just a question of how the lower bound was chosen. And on some level, I'm not even sure that that's an issue: an observation that's based on "the longest N bound orbits all show this pattern" feels meaningful to me even if I'm not sure in advance how big N is going to be.
scott_s 11y ago
The authors are not relying just on the improbability of that arrangement, but also on its stability. From one of the authors, on http://www.findplanetnine.com/ http://www.findplanetnine.com/: "Mike and I were genuinely perplexed. Could the confinement of the orbits be due to an observational bias or a mere coincidence (after all, we are talking about six objects here - not exactly “big data”)? Thankfully, the probability of the observed alignment being fortuitous can be assessed in a statistically rigorous manner, and clocks in at right around 0.007%. Not a great gamble. Moreover, application of simple perturbation theory (or direct numerical integration) demonstrates that if allowed to evolve under the gravitational influence of Jupiter, Saturn, Uranus and Neptune, the orbits would become randomly oriented on timescales much shorter than than the multi-billion year lifetime of the solar system. So the dynamical origin of the peculiar structure of the Kuiper belt cannot be outsourced to the distant past - something is holding the orbits together right now."
gaur 11y ago
> but if they didn't do their multiple comparisons correction right (which I am in no position to determine), they're basically reading tea leaves. Please don't spread FUD about other people's research and then try to soften it with a disclaimer about how you're "in no position to determine" anything about the research. If you want to offer criticism of the research, you should do the legwork to find out what the researchers did and did not do.
- deleted 11y ago
  [deleted]
- dr_zoidberg 11y ago
  Well said. There are people reviewing the papers, who are in a position to determine if the work is correct, who are enthusiastic about this.
- tdaltonc 11y ago
  I've though a lot about your comment. I don't think that they're dummies. This is one of the big unsolved problems in statistics and empiricism.[1] There is no general solution to this problem, and the practical solutions to this problem are all very particular to their field. The geneticists have certain ways for controlling for certain known problems; the neuro imagers have certain ways for controlling for certain known problems; same goes for ML, particle physics, economics, ect. I'm personally familiar with a few of these, and I know that they are very particular to their field and that they each have their own history (which is why I'm hesitant to present myself as an expert on statistic empiricism in the context of trans-neptunian planet hunting). And those histories (the ones I'm familiar with) are a parade of over confidence followed by the community establishing new things that need to be controlled for. The red flag in my mind isn't that they are following the established rigor for trans-neptunian planet hunting (I'm sure that they are, because peer-review), or that they aren't following a theoretically valid methodology (I'm sure that they are not, because it doesn't exist). The red flag in my head is that trans-neptunian planet hunting, as a field, looks like the kind of place that there would be a lot of false positives (few historical true-positives, relatively small field, relatively novel methods). I don't doubt this experiment the way a expert in trans-neptunian planet hunting would: "They should control for X, Y, and Z, but didn't!" I don't have the expertise to do that. I'm criticizing it the way a philosopher of science might: "This sort of question in this sort of field is likely to have such-and-such problems and not yet have good ways to dealing with it yet." And I do think that know enough to log that criticism. Love to know what you think of that explanation. [1] https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_statistics https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_s...
Balgair 11y ago
I knew Chaz personally in undergrad and a bit since then. Though I have not read the article yet (I'm in a state of shock really) I can attest to the man. He is so kind, funny, personable, and one a heck of a guitar player. Based on his stellar work in undergrad in modeling the orbital path futures of our solar system on the Gyr timescale, I am sure he is sure of the work. (Only non humanities major to ever win the Dean's Award at UCSC, it was great work) The man would not ever publish this unless he was sure of it. Science and time may tell him wrong, but I'm one to believe him as a friend. Both he and Olga must be on cloud nine right now! (oh wow, nine, wow!) I mean WOW! Like, a PLANET! A frickin PLANET! None of this dwarf stuff, a full PLANET! (Not to malign all those folk's hard work in finding the dwarves either). Jesus tapdancing christ gravy! What a day! EDIT: Upon further reading, it may still be a dwarf planet as it has not 'cleared' it's orbit. Still, my lord, so happy for them both right now!
- mark-r 11y ago
  "Stellar work"?
  - Balgair 11y ago
    har har
- dr_zoidberg 11y ago
  As for the "full" planet/dwarf planet debate, a more precise explanation that has been seen recently in many talks is "gravitational dominance of the orbit". This arises from the case of Jupiter and its trojans. The criteria is "it dominates the gravitational interactions in its neighbourhood", which such an object (affecting practically everything in the Kuiper Belt) does.

jessriedel 11y ago

I think you're right that that's one of the chief concerns, but I think you paint it incorrectly as a technical problem. ("Multiple comparisons" as you call it. Particle physicists call it the "look-elsewhere effect": https://en.wikipedia.org/wiki/Look-elsewhere_effect https://en.wikipedia.org/wiki/Look-elsewhere_effect ) The easy, though tedious, part of the look-elsewhere effect is to calculate a probabilisitic correction for all the various combinatorics of X objects in Y orbits over a time period of length Z that one might have seen. The hard, and borderline philosophical, part is to identify the proper reference class, i.e., what qualitatively different sorts of observations might have also considered evidence for planet nine, and how many other astrophysical searches are we running that might have given us weird results? This is really where frequentist approaches strain to be useful, and it's hard to avoid resorting to a Bayesian prior. Analogously for particle physics, it's easy to compute the chance of getting a 4-sigma deviation for two different experiments in any gamma-gamma energy bins, so the after-the-fact corrections to the significant of a fluctuation in a particular bin are easy to compute. Much more difficult is to consider all the hundreds of parallel searches going on, of greatly varying levels of motivation, and correcting based on that.

jerf 11y ago
I'd also observe that in terms of prior probability, "There is a ninth planet in the solar system that we have not yet detected" is probably hovering at least in the low-integer percentages, and I imagine some scientists would place it higher than that. The general idea is not a wild, far-out one by any stretch of the imagination.
- XorNot 11y ago
  That isn't how probability works though. You have a Bayesian priors problem - you can't establish what the initial probabilities are.
  - jerf 11y ago
    I can put some bounds on them, which is all I tried to do. It is certainly not reasonable to act as if the idea there is a ninth undetected planet is some sort of one-in-a-trillion possibility. I'm not even convinced you're correct anyhow; since I tend to accept the interpretation of probability as a statement of our knowledge of the world as good enough most of the time, we can establish a reasonable probability statement that expresses that uncertainty. We don't know everything about the universe or even our solar system but there is something about this topic that makes people start grossly overstating our ignorance.

politician 11y ago

Keep reading, their simulation explains several additional previously unrelated trajectories.

stdbrouw 11y ago

The anomaly itself does not seem to be disputed. It's true that when looking for arbitrary clusters and unusual patterns you can pretty much always find something -- though it's more of a Texas Sharpshooter fallacy [1] than the kind of multiple comparisons you see in e.g. medical research -- but the paper in question [2] mentions a whole bunch of alternative theories for why the particular clustering of a bunch of rocks in the Kuiper belt is what it is, and it doesn't seem very likely to me that none of those researchers ever thought "well, gee, maybe this is all just a fluke." In fact the competing hypotheses would be my main source of skepticism – maybe it's a planet, maybe it's an entirely different phenomenon. [1] https://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy https://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy [2] http://iopscience.iop.org/article/10.3847/0004-6256/151/2/22;jsessionid=9DAB98EED9CB30448604A2F4CA0F8752.c5.iopscience.cld.iop.org http://iopscience.iop.org/article/10.3847/0004-6256/151/2/22...

logicallee 11y ago

So I might not have a great enough understanding but isn't 3.8 sigma not high for this kind of announcement (with the current title)? if someone wants to report finding a planet in the solar system ten times the size of Earth wouldn't they do so at more than @3.8 sigma. In particular, the prior is quite low because it would affect a bunch of other astronomical calculations, wouldn't it? So you would want the confidence that much higher.... http://www.dummies.com/how-to/content/how-to-calculate-six-sigma-quality.html http://www.dummies.com/how-to/content/how-to-calculate-six-s... What I mean is that in human terms, Google says the average man is 5'9" with standard deviation 2.9 inches, then 3.8 sigma (the chances of this happening by chance) are the same as the chances of finding someone 6'8". It doesn't seem unbelievable - https://answers.yahoo.com/question/index?qid=20101208183308AA4LYym https://answers.yahoo.com/question/index?qid=20101208183308A... If a woman is set up on a blind date and told the guy is 6'8" would she find it like not credible? My understnading of statistics then is that the idea that there just so happens to be a planet ten times the size of Earth in our solar system is pretty unbelievable. So I'd want the chances of it being an experimental fluke to be lower than seeing a 6'8" guy, anywhere, at any time, in any context. Because I would think that's what happens when you look at enough data. - Aren't the chances of this being completely spurious literally exactly the same as a researcher saying "I saw a 6'8" guy somewhere", meaning anywhere, in any context? Wouldn't you want higher evidence before stating "evidence found for ninth planet in our solar system, ten times the mass of Earth"?

umanwizard 11y ago
I wish people wouldn't downvote you without explaining why you're wrong. I have zero education in statistics and your comment makes intuitive sense to me so I'd like to learn more.
- logicallee 11y ago
  Before your comment, I edited it to be much more civil and less presumptuous, my original phrasing had included "3.8 sigma is fuck-all", I was briefer, more resolute, and had just included my first link. (You can see evidence of my original phrasing if you remove the hedging I added.) But the downvotes meant I'm probably wrong - rather than delete I edited it to be much nicer so I could find out why I'm wrong.

Steuard 11y ago

I haven't seen the original research myself to judge, but it sounds like they're being careful about these effects. Looking at their discussion on the main page of http://www.findplanetnine.com/ http://www.findplanetnine.com/, the claim seems to be that this isn't "we spotted six out of the solar system's zillions of objects that formed a cluster" but rather "the unique six known objects within a certain range of orbital parameters all form a cluster in a different parameter". More specifically, quoting from that page, "all Kuiper belt objects with orbits that do not veer into inter-planetary space and spend longer than approximately 2000 years to complete a single revolution around the Sun... trace out elliptical paths that point into approximately the same direction in physical space, and lie in approximately the same plane." That could be just an observational coincidence, and that's where they claim the statistical figure that you've cited. I have no idea to what extent those figures account for the multiple comparisons issue, but the range of orbital parameters that I've quoted here doesn't look very fine-tuned to me. (The upper bound certainly isn't, anyway!) So it's just a question of how the lower bound was chosen. And on some level, I'm not even sure that that's an issue: an observation that's based on "the longest N bound orbits all show this pattern" feels meaningful to me even if I'm not sure in advance how big N is going to be.

scott_s 11y ago

The authors are not relying just on the improbability of that arrangement, but also on its stability. From one of the authors, on http://www.findplanetnine.com/ http://www.findplanetnine.com/: "Mike and I were genuinely perplexed. Could the confinement of the orbits be due to an observational bias or a mere coincidence (after all, we are talking about six objects here - not exactly “big data”)? Thankfully, the probability of the observed alignment being fortuitous can be assessed in a statistically rigorous manner, and clocks in at right around 0.007%. Not a great gamble. Moreover, application of simple perturbation theory (or direct numerical integration) demonstrates that if allowed to evolve under the gravitational influence of Jupiter, Saturn, Uranus and Neptune, the orbits would become randomly oriented on timescales much shorter than than the multi-billion year lifetime of the solar system. So the dynamical origin of the peculiar structure of the Kuiper belt cannot be outsourced to the distant past - something is holding the orbits together right now."

gaur 11y ago

> but if they didn't do their multiple comparisons correction right (which I am in no position to determine), they're basically reading tea leaves. Please don't spread FUD about other people's research and then try to soften it with a disclaimer about how you're "in no position to determine" anything about the research. If you want to offer criticism of the research, you should do the legwork to find out what the researchers did and did not do.

deleted 11y ago
[deleted]
dr_zoidberg 11y ago
Well said. There are people reviewing the papers, who are in a position to determine if the work is correct, who are enthusiastic about this.
tdaltonc 11y ago
I've though a lot about your comment. I don't think that they're dummies. This is one of the big unsolved problems in statistics and empiricism.[1] There is no general solution to this problem, and the practical solutions to this problem are all very particular to their field. The geneticists have certain ways for controlling for certain known problems; the neuro imagers have certain ways for controlling for certain known problems; same goes for ML, particle physics, economics, ect. I'm personally familiar with a few of these, and I know that they are very particular to their field and that they each have their own history (which is why I'm hesitant to present myself as an expert on statistic empiricism in the context of trans-neptunian planet hunting). And those histories (the ones I'm familiar with) are a parade of over confidence followed by the community establishing new things that need to be controlled for. The red flag in my mind isn't that they are following the established rigor for trans-neptunian planet hunting (I'm sure that they are, because peer-review), or that they aren't following a theoretically valid methodology (I'm sure that they are not, because it doesn't exist). The red flag in my head is that trans-neptunian planet hunting, as a field, looks like the kind of place that there would be a lot of false positives (few historical true-positives, relatively small field, relatively novel methods). I don't doubt this experiment the way a expert in trans-neptunian planet hunting would: "They should control for X, Y, and Z, but didn't!" I don't have the expertise to do that. I'm criticizing it the way a philosopher of science might: "This sort of question in this sort of field is likely to have such-and-such problems and not yet have good ways to dealing with it yet." And I do think that know enough to log that criticism. Love to know what you think of that explanation. [1] https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_statistics https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_s...

Balgair 11y ago

I knew Chaz personally in undergrad and a bit since then. Though I have not read the article yet (I'm in a state of shock really) I can attest to the man. He is so kind, funny, personable, and one a heck of a guitar player. Based on his stellar work in undergrad in modeling the orbital path futures of our solar system on the Gyr timescale, I am sure he is sure of the work. (Only non humanities major to ever win the Dean's Award at UCSC, it was great work) The man would not ever publish this unless he was sure of it. Science and time may tell him wrong, but I'm one to believe him as a friend. Both he and Olga must be on cloud nine right now! (oh wow, nine, wow!) I mean WOW! Like, a PLANET! A frickin PLANET! None of this dwarf stuff, a full PLANET! (Not to malign all those folk's hard work in finding the dwarves either). Jesus tapdancing christ gravy! What a day! EDIT: Upon further reading, it may still be a dwarf planet as it has not 'cleared' it's orbit. Still, my lord, so happy for them both right now!

mark-r 11y ago
"Stellar work"?
- Balgair 11y ago
  har har
dr_zoidberg 11y ago
As for the "full" planet/dwarf planet debate, a more precise explanation that has been seen recently in many talks is "gravitational dominance of the orbit". This arises from the case of Jupiter and its trojans. The criteria is "it dominates the gravitational interactions in its neighbourhood", which such an object (affecting practically everything in the Kuiper Belt) does.