5 ms·

High-energy experiment is generally statistically unsound

29 points by scottlocklin 7y ago

buboard 7y ago
This is the gist i believe: > In 2015, CERN’s LHCb found pentaquarks in their data with 9 sigma significance!!! This either proved that CERN can find whatever it wants in its data or it proved that the experiments which showed the pentaquark did not exist were wrong. I suspect that the former is true because I wonder what the expected probability of seeing the particle was in their application of Bayes’ theorem. I 'm not sure that's an indictment of LHC itself or rather of the sloppy way with which science attribute importance to whatever is statistically significant. It's probably true that there is too much data and not enough good models.
- amadsen 7y ago
  > the sloppy way with which science attribute importance to whatever is statistically significant. But we don't. Just because an effect is statistically significant doesn't mean it could not also be explained by some systematic effect and physicists are very aware of that. The famous five sigma rule prevents a claim of discovery without statistically significant evidence. It does NOT mean that we automatically accept every five sigma deviation from the background model as new physics.
ivalm 7y ago
Seems like a misguided rant. The whole beginning of "but what about applications?" makes no sense. The complaints about what is essentially p-hacking might apply to CERN (but probably not), but doesnt work with LIGO where many events were detected by their model-free pipeline.
- EdwardDiego 7y ago
  IIRC the author doesn't believe black holes exist, so not surprised.
  - ivalm 7y ago
    Sure, but there are two possible ways to "not believe" in black holes. One can say "there is an alternative not yet known explanation to gravity and this explanation will not have singularities/math artifacts." While GR is one of the most verified theories in physics I have some modicum of sympathy for that view. What is disingenuous is her view "black holes dont exist because GR measurements are bad." Because they really are good and statistically significant. For LIGO results you can reject the nature of the compact bodies (although the fits are very good), but you cant reject the event of the detection.
- sprash 7y ago
  Indeed, but if you ignore much of the crack-pottery and the lack of understanding of fundamental statistics you could find some valid criticism of current physics in this piece (but more like needles in a hay stack). Physics is now entering a new era of "exploration". This means lots of data and little results. The career of the average scientist will be a series of papers where they prove that they found nothing. I can see how this is frustrating or boring to many people but it is the only way forward. The low hanging fruit has been harvested decades ago. The methodologies of measurements have to adjust accordingly and the purpose of experiments has to be advertised correctly.
  - ivalm 7y ago
    I mean I left physics after doing my PhD precisely because I felt it wasnt exciting, but I disagree with the criticism of "big data." I know analytic results are romanticized and even now physicists are looking to express things in but a few equations, but it might also be that the world is complicated and to learn the details you really need to crunch a lot. Kind of like in avalanches, lots of things can be modeled by power laws but the actual details are incredibly messy and non-integrable.
  - vanderZwan 7y ago
    > Indeed, but if you ignore much of the crack-pottery She claims to have studied physics for twenty years and have a PhD in accelerator physics. If true (and she seems too well-informed to be lying), her crack-pottery is at the very least well-informed enough to likely contain many useful insights and views worth considering. Especially regarding physicist culture. There is a page where she shares a few of her novel[0], and speaking as someone who dropped out of physics myself (just a lot earlier than she did) I certainly recognize more than a few themes in the feedback contained therein. [0] https://kirstenhacker.wordpress.com/2019/07/30/reviews/ https://kirstenhacker.wordpress.com/2019/07/30/reviews/
    - blix 7y ago
      It's kinda neat that one of her major complaints with physicist culture is pernicious groupthink, and that some of the primary arguments against her are prima facie dismissals for not subscribing to consensus.
deleted 7y ago
[deleted]
Roritharr 7y ago
Reading this felt like a breath of fresh air. Her blog is really worthwhile to read, also this article about the first picture of a black hole: https://kirstenhacker.wordpress.com/2019/08/24/signs-of-the-times/ https://kirstenhacker.wordpress.com/2019/08/24/signs-of-the-... I'm not qualified to judge either way, but I like the style of her reasoning very much.
- amadsen 7y ago
  The whole thing is just a big straw man argument. Well, actually the piece is pretty incoherent and I found it hard to find a red thread through the musings on such diverse topics as WIMPs, pentaquarks and the functioning of drift chambers. But the core of it seems to be at she thinks that scientists are now sitting on massive piles of data, mine through it looking for any oddities and then jump on the chance to publish a discovery of a new physical phenomenon as soon as one is found. I can't judge whether she is just ignorant or deliberately misleading (she claims to have worked in physics), but this is just not how experiments are done. You almost always start with a theoretical prediction by a new theory, and there is no lack of theories out there so it has to be a pretty good one that has plenty of justification to merit attention enough to invest the 10+ person-years of effort it typically takes to conduct a rigorous analysis of a chunk of data from a modern large scale scientific instrument. You carefully demonstrate in various side bands your ability to correctly model every background process and instrumental effect that could influence the measurement. After that, if the data very clearly favors the new prediction over the null hypothesis, you may publish the discovery in order to invite other scientists to confirm or refute it. This does not mean that you or your peers accept that the new theory is correct. And this is all supposedly in support of the thesis that large experimental facilities are a waste of money, which is patently nonsensical. They have been estimated to pay for themselves in direct returns alone (training of students and researchers, spin-off applications to silicon sensors, cryo technology, vacuum technology, lasers, accelerators, computing, etc). And the long term importance of investments in fundamental research are incalculable. For perspective, remember that the electron was considered a "useless" discovery back in 1897.
  - blix 7y ago
koalala 7y ago
>Let’s put aside the fact that one cannot directly observe what is inside of a proton or a neutron and that it is quite a logical leap to assume that the extremely unstable particles that come out of a collision exist in a stable form within the non-collided particles. This is nonsense, deep inelastic scattering experiments have for example directly shown that protons have the predicted internal structure [1]. [1] https://en.wikipedia.org/wiki/Deep_inelastic_scattering https://en.wikipedia.org/wiki/Deep_inelastic_scattering
qtplatypus 7y ago
“A question which does not get asked in physics circles is: ‘do impossible to measure things objectively exist?’” The reason that the question isn’t asked in physics circles is because the question is a philosophical question not a physics question.
- vanderZwan 7y ago
  Her larger point is that there are still assumed answers to these questions anyway that physics is building on.
  - qtplatypus 7y ago
    I am not sure that physists do. Many of the physics I know seem to subscribe to the “Shut up and calculate” school of physics where they say all of these things are a model that gives results that correspond to the experiments.
    - vanderZwan 7y ago
      Yes, and they say this at the same time as they say share plenty of stories about alternative formulations that work as well but rejected because they lack "elegance" or whatever, and somehow they do not see any conflict in this. It's a weird kind of denial of making first principle assumptions.
      - qtplatypus 7y ago
        True but if you have two alternates that are of equal predictive power then the choice between them can be totally arbitrary.
        nixtaken 7y ago
        With the contrast she made between WIMPs+tiredlight and Higgs+bigbang, I think she suggested that when both languages have equal predictive power, one should choose the language that is consistent with existing definitions. If a new language confuses definitions and creates the appearance of paradoxes, it is not a good language. This is drawn out in some of her other articles.

29 points by scottlocklin 7y ago

This is the gist i believe: > In 2015, CERN’s LHCb found pentaquarks in their data with 9 sigma significance!!! This either proved that CERN can find whatever it wants in its data or it proved that the experiments which showed the pentaquark did not exist were wrong. I suspect that the former is true because I wonder what the expected probability of seeing the particle was in their application of Bayes’ theorem. I 'm not sure that's an indictment of LHC itself or rather of the sloppy way with which science attribute importance to whatever is statistically significant. It's probably true that there is too much data and not enough good models.

amadsen 7y ago
> the sloppy way with which science attribute importance to whatever is statistically significant. But we don't. Just because an effect is statistically significant doesn't mean it could not also be explained by some systematic effect and physicists are very aware of that. The famous five sigma rule prevents a claim of discovery without statistically significant evidence. It does NOT mean that we automatically accept every five sigma deviation from the background model as new physics.

ivalm 7y ago

Seems like a misguided rant. The whole beginning of "but what about applications?" makes no sense. The complaints about what is essentially p-hacking might apply to CERN (but probably not), but doesnt work with LIGO where many events were detected by their model-free pipeline.

EdwardDiego 7y ago
IIRC the author doesn't believe black holes exist, so not surprised.
- ivalm 7y ago
  Sure, but there are two possible ways to "not believe" in black holes. One can say "there is an alternative not yet known explanation to gravity and this explanation will not have singularities/math artifacts." While GR is one of the most verified theories in physics I have some modicum of sympathy for that view. What is disingenuous is her view "black holes dont exist because GR measurements are bad." Because they really are good and statistically significant. For LIGO results you can reject the nature of the compact bodies (although the fits are very good), but you cant reject the event of the detection.
sprash 7y ago
Indeed, but if you ignore much of the crack-pottery and the lack of understanding of fundamental statistics you could find some valid criticism of current physics in this piece (but more like needles in a hay stack). Physics is now entering a new era of "exploration". This means lots of data and little results. The career of the average scientist will be a series of papers where they prove that they found nothing. I can see how this is frustrating or boring to many people but it is the only way forward. The low hanging fruit has been harvested decades ago. The methodologies of measurements have to adjust accordingly and the purpose of experiments has to be advertised correctly.
- ivalm 7y ago
  I mean I left physics after doing my PhD precisely because I felt it wasnt exciting, but I disagree with the criticism of "big data." I know analytic results are romanticized and even now physicists are looking to express things in but a few equations, but it might also be that the world is complicated and to learn the details you really need to crunch a lot. Kind of like in avalanches, lots of things can be modeled by power laws but the actual details are incredibly messy and non-integrable.
- vanderZwan 7y ago
  > Indeed, but if you ignore much of the crack-pottery She claims to have studied physics for twenty years and have a PhD in accelerator physics. If true (and she seems too well-informed to be lying), her crack-pottery is at the very least well-informed enough to likely contain many useful insights and views worth considering. Especially regarding physicist culture. There is a page where she shares a few of her novel[0], and speaking as someone who dropped out of physics myself (just a lot earlier than she did) I certainly recognize more than a few themes in the feedback contained therein. [0] https://kirstenhacker.wordpress.com/2019/07/30/reviews/ https://kirstenhacker.wordpress.com/2019/07/30/reviews/
  - blix 7y ago
    It's kinda neat that one of her major complaints with physicist culture is pernicious groupthink, and that some of the primary arguments against her are prima facie dismissals for not subscribing to consensus.

deleted 7y ago

[deleted]

Roritharr 7y ago

Reading this felt like a breath of fresh air. Her blog is really worthwhile to read, also this article about the first picture of a black hole: https://kirstenhacker.wordpress.com/2019/08/24/signs-of-the-times/ https://kirstenhacker.wordpress.com/2019/08/24/signs-of-the-... I'm not qualified to judge either way, but I like the style of her reasoning very much.

amadsen 7y ago
The whole thing is just a big straw man argument. Well, actually the piece is pretty incoherent and I found it hard to find a red thread through the musings on such diverse topics as WIMPs, pentaquarks and the functioning of drift chambers. But the core of it seems to be at she thinks that scientists are now sitting on massive piles of data, mine through it looking for any oddities and then jump on the chance to publish a discovery of a new physical phenomenon as soon as one is found. I can't judge whether she is just ignorant or deliberately misleading (she claims to have worked in physics), but this is just not how experiments are done. You almost always start with a theoretical prediction by a new theory, and there is no lack of theories out there so it has to be a pretty good one that has plenty of justification to merit attention enough to invest the 10+ person-years of effort it typically takes to conduct a rigorous analysis of a chunk of data from a modern large scale scientific instrument. You carefully demonstrate in various side bands your ability to correctly model every background process and instrumental effect that could influence the measurement. After that, if the data very clearly favors the new prediction over the null hypothesis, you may publish the discovery in order to invite other scientists to confirm or refute it. This does not mean that you or your peers accept that the new theory is correct. And this is all supposedly in support of the thesis that large experimental facilities are a waste of money, which is patently nonsensical. They have been estimated to pay for themselves in direct returns alone (training of students and researchers, spin-off applications to silicon sensors, cryo technology, vacuum technology, lasers, accelerators, computing, etc). And the long term importance of investments in fundamental research are incalculable. For perspective, remember that the electron was considered a "useless" discovery back in 1897.
- blix 7y ago

koalala 7y ago

>Let’s put aside the fact that one cannot directly observe what is inside of a proton or a neutron and that it is quite a logical leap to assume that the extremely unstable particles that come out of a collision exist in a stable form within the non-collided particles. This is nonsense, deep inelastic scattering experiments have for example directly shown that protons have the predicted internal structure [1]. [1] https://en.wikipedia.org/wiki/Deep_inelastic_scattering https://en.wikipedia.org/wiki/Deep_inelastic_scattering

qtplatypus 7y ago

“A question which does not get asked in physics circles is: ‘do impossible to measure things objectively exist?’” The reason that the question isn’t asked in physics circles is because the question is a philosophical question not a physics question.

vanderZwan 7y ago
Her larger point is that there are still assumed answers to these questions anyway that physics is building on.
- qtplatypus 7y ago
  I am not sure that physists do. Many of the physics I know seem to subscribe to the “Shut up and calculate” school of physics where they say all of these things are a model that gives results that correspond to the experiments.
  - vanderZwan 7y ago
    Yes, and they say this at the same time as they say share plenty of stories about alternative formulations that work as well but rejected because they lack "elegance" or whatever, and somehow they do not see any conflict in this. It's a weird kind of denial of making first principle assumptions.
    - qtplatypus 7y ago
      True but if you have two alternates that are of equal predictive power then the choice between them can be totally arbitrary.
      - nixtaken 7y ago
        With the contrast she made between WIMPs+tiredlight and Higgs+bigbang, I think she suggested that when both languages have equal predictive power, one should choose the language that is consistent with existing definitions. If a new language confuses definitions and creates the appearance of paradoxes, it is not a good language. This is drawn out in some of her other articles.