7 ms·

Because they're statistical artifacts, basically. P=0.05 means you're got a 1/20 chance of it happening on any given random test set where the bill hypothesis i

Because they're statistical artifacts, basically. P=0.05 means you're got a 1/20 chance of it happening on any given random test set where the bill hypothesis is true, after all. Physics uses far higher confidence levels, but also runs (effectively) massive numbers of tests. That's part of the danger of picking theories based on the data - each degree of freedom in interpretation multiplies your chances of having spurious results crop up.

kmm 7y ago
Given the high statistical confidence they have, 7.2 sigma or 1 in 2 trillion, it's more likely to be some sort of systemic issue with the experimental setup.
air7 7y ago
This is well known and corrected for in properly conducted experiments. https://en.m.wikipedia.org/wiki/Multiple_comparisons_problem https://en.m.wikipedia.org/wiki/Multiple_comparisons_problem
gus_massa 7y ago
I almost agree. This is the more probable explanation of the other ~39 anomalies that are at 2 or 3 sigma. As a rule of thumb, don't get too attached to a 2 or 3 sisma because there is a high chance that it is only a statistical fluke and the apparent result will disappear with more data. But this cases has too many sigmas, 7!! And the group has a history of dubious discoveries, like the 5 sigma result in 2015 that was never reproduced. So I'm more inclined for a bad measurement method until an independent group can replicate it.
- peteradio 7y ago
  > like the 5 sigma result in 2015 Isn't that just an earlier version of the same experiment? Doesn't seem too surprising that a broad distribution has a mean shift with a different experimental setup. I'd count this as a repeat of the same experiment. That of course doesn't rule out some flaw in their experimental setup that happens in roughly the same way in each repeat.
  - gus_massa 7y ago
    The mass of the result of 2015 is close to the current claim, so I have to correct may comment and say that it has not disappeared (yet). But they have a 13.45+-.30 MeV result in 2012 with a significance of 3 sigmas [1] and a 12.0+-2.5 MeV result in 2008 (I can't find the significance, it looks like a 2 or 3 sigma) [2]. If it possible to get a false result by chance like 1/20 or 1/100 of the times, but they already got 2 of them. So I'm very skeptical of the new third claim. (Also the story about the protophobic particle is strange. There have been more strange things, but this is strange anyway. And it's also not clear why the protophobicity hides the particle in all the previous electron-positron collision experiments in many other laboratories.) [1] https://www.lnf.infn.it/sis/frascatiseries/Volume56/Krasznahorkay.pdf https://www.lnf.infn.it/sis/frascatiseries/Volume56/Krasznah... [2] http://www.actaphys.uj.edu.pl/fulltext?series=Reg&vol=39&page=483 http://www.actaphys.uj.edu.pl/fulltext?series=Reg&vol=39&pag...
    - peteradio 7y ago
      Just to play advocate more, if you look at the broad excess in the angular plot in all those papers it shouldn't be surprising that this results in a broad and changing best estimates for mass. So without crunching the numbers, again its possible all these are measuring the same thing with a fluctuating best guess for mass. I think we shouldn't dismiss the latest result due to the previous results. I think we should wonder what could be the systematic issue that could cause this effect without it being new physics. I think the excess in all the experiments will likely have the same root, be it new physics or something else that is not yet understood. So don't count those previous papers against them, those are all measuring the same thing and the convention of physics to match to some theory is causing the precise conclusion to jump around. If you look at the actual raw data the effect is not changing too much.
      - gus_massa 7y ago
        The relevant graph are in the Figure 6 of the first paper and Figure 2b of the second. The idea is to match a simulation of the results you would expect if the new particle has different masses and compare it with the experimental data and see how good is the fit. The horizontal axis is the mass of the hypothetical particle in the simulation, and the vertical value is how good is the fit (lower is better). In both the 17MeV region is out of the interesting region. So if there is an effect, they did a very bad analysis before. I think it's better to wait until another group can independently reproduce the results. It doesn't look too complicated, for someone that has a similar lab. (It's not like the results of the LHC, where the reproduction steps start with: "First dig a 17mi long circle and then call for further instructions." :) Note that to try to avoid false positives, they have two almost independent groups in oposite points of the circle.)

kmm 7y ago

Given the high statistical confidence they have, 7.2 sigma or 1 in 2 trillion, it's more likely to be some sort of systemic issue with the experimental setup.

air7 7y ago

This is well known and corrected for in properly conducted experiments. https://en.m.wikipedia.org/wiki/Multiple_comparisons_problem https://en.m.wikipedia.org/wiki/Multiple_comparisons_problem

gus_massa 7y ago

I almost agree. This is the more probable explanation of the other ~39 anomalies that are at 2 or 3 sigma. As a rule of thumb, don't get too attached to a 2 or 3 sisma because there is a high chance that it is only a statistical fluke and the apparent result will disappear with more data. But this cases has too many sigmas, 7!! And the group has a history of dubious discoveries, like the 5 sigma result in 2015 that was never reproduced. So I'm more inclined for a bad measurement method until an independent group can replicate it.

peteradio 7y ago
> like the 5 sigma result in 2015 Isn't that just an earlier version of the same experiment? Doesn't seem too surprising that a broad distribution has a mean shift with a different experimental setup. I'd count this as a repeat of the same experiment. That of course doesn't rule out some flaw in their experimental setup that happens in roughly the same way in each repeat.
- gus_massa 7y ago
  The mass of the result of 2015 is close to the current claim, so I have to correct may comment and say that it has not disappeared (yet). But they have a 13.45+-.30 MeV result in 2012 with a significance of 3 sigmas [1] and a 12.0+-2.5 MeV result in 2008 (I can't find the significance, it looks like a 2 or 3 sigma) [2]. If it possible to get a false result by chance like 1/20 or 1/100 of the times, but they already got 2 of them. So I'm very skeptical of the new third claim. (Also the story about the protophobic particle is strange. There have been more strange things, but this is strange anyway. And it's also not clear why the protophobicity hides the particle in all the previous electron-positron collision experiments in many other laboratories.) [1] https://www.lnf.infn.it/sis/frascatiseries/Volume56/Krasznahorkay.pdf https://www.lnf.infn.it/sis/frascatiseries/Volume56/Krasznah... [2] http://www.actaphys.uj.edu.pl/fulltext?series=Reg&vol=39&page=483 http://www.actaphys.uj.edu.pl/fulltext?series=Reg&vol=39&pag...
  - peteradio 7y ago
    Just to play advocate more, if you look at the broad excess in the angular plot in all those papers it shouldn't be surprising that this results in a broad and changing best estimates for mass. So without crunching the numbers, again its possible all these are measuring the same thing with a fluctuating best guess for mass. I think we shouldn't dismiss the latest result due to the previous results. I think we should wonder what could be the systematic issue that could cause this effect without it being new physics. I think the excess in all the experiments will likely have the same root, be it new physics or something else that is not yet understood. So don't count those previous papers against them, those are all measuring the same thing and the convention of physics to match to some theory is causing the precise conclusion to jump around. If you look at the actual raw data the effect is not changing too much.
    - gus_massa 7y ago
      The relevant graph are in the Figure 6 of the first paper and Figure 2b of the second. The idea is to match a simulation of the results you would expect if the new particle has different masses and compare it with the experimental data and see how good is the fit. The horizontal axis is the mass of the hypothetical particle in the simulation, and the vertical value is how good is the fit (lower is better). In both the 17MeV region is out of the interesting region. So if there is an effect, they did a very bad analysis before. I think it's better to wait until another group can independently reproduce the results. It doesn't look too complicated, for someone that has a similar lab. (It's not like the results of the LHC, where the reproduction steps start with: "First dig a 17mi long circle and then call for further instructions." :) Note that to try to avoid false positives, they have two almost independent groups in oposite points of the circle.)