5 ms·

Along with #1, a culture shift to promoting the publishing of null-results would also help quite a bit. Journal acceptance and funding is now incredibly biased

Along with #1, a culture shift to promoting the publishing of null-results would also help quite a bit. Journal acceptance and funding is now incredibly biased towards finding a significant result. Every researcher will find p<.05 on average after 20 studies even if those studies are literally measuring random noise. We need to have some incentive to at least share null-results and replications.

kragen 5y ago
Agreed! Aside from the spurious results produced by publication bias, the amount of effort spent on trying things that somebody already knows don't work is staggering. The knowledge of what doesn't work is mostly passed on tacitly through apprenticeship rather than published, which means it can easily be lost. As a simple example, we've known how to make transparent glass for 1900 years, but it took Ben Krasnow two months to achieve it himself despite having money, libraries, and glass experts at his disposal, and being generally competent at making things, having built, for example, his own electron microscope: https://www.youtube.com/watch?v=mUcUy7SqdS0 https://www.youtube.com/watch?v=mUcUy7SqdS0
dhosek 5y ago
This is true in all fields. I was playing with an idea that factors of Fermat non-primes (2^{2^n}+1 for n > 5) could be expressed in terms of products of the complex factors of the polynomial x^{2^n}+1 with x=1 with the idea that it might lead to a proof that for all n > 5, F_n is composite. I spent a bunch of time working out the exact expressions for these roots and playing around with their products (which ended up having a connection to Chebyshev polynomials), and then it occurred to me to try exhaustively trying the possible products of the complex roots of x^{64}+1 with x=1 to see if it turned up the factors of F_6 and it turned out it was a dead end. A compilation of false hypotheses in different fields of mathematics could make for interesting reading (and perhaps lead to eliminating multiple people getting stuck in the same cul de sac).
- noslenwerdna 5y ago
  Interestingly, this isn't true in particle physics! Probably most papers published by ATLAS or CMS are a negative result.
generationP 5y ago
As the Surgisphere affair has showed, publishing null results opens its own can of worms: No one will bother replicating them. Surprising positives at least have a kind of target drawn on them, but a negative result that matches people's expectations is really not something anyone disinterested would want to replicate -- there is no fame in that. Fraudulent non-results will be blocking fruitful directions. I wish I had a panacea for these things...
- zozbot234 5y ago
  The Surgisphere papers were not "null" results. They found significantly higher risk of death after HCQ treatment, based on what turned out to be highly flawed data. The problem was with the original dataset. (Not to be outdone, the Surgisphere folks also contributed their data to a preprint which purported to show horse dewormer^H^H^H^H^H^H^H^H^H^H^H^H^H^H ivermectin could be a successful SARS-CoV2 treatment).
  - sampo 5y ago
    > horse dewormer^H^H^H^H^H^H^H^H^H^H^H^H^H^H ivermectin You are not being truthful, labeling ivermectin only a "horse dewormer". Ivermectin is used as an antiparasitic drug also for people. And not only as a dewormer, but also against lice and mites. https://en.wikipedia.org/wiki/Ivermectin https://en.wikipedia.org/wiki/Ivermectin
    - kragen 5y ago
      Indeed. One of the most disheartening things about the covid pandemic has been seeing how many people are willing to propagate the most egregious health-related misinformation, such as that ivermectin is only or primarily a horse dewormer or that it is dangerous, as long as it identifies them with the desired faction. Such depravity should not have come as a surprise to me, but it did. I've taken ivermectin—for parasites, not for covid. I wonder how many people will die of parasitic infections or more toxic antihelminthics and acaricides in the next few years because they or their neighbors refused to take ivermectin after they heard it was a dangerous veterinary-only drug from unscrupulous liars during the covid pandemic.

kragen 5y ago

Agreed! Aside from the spurious results produced by publication bias, the amount of effort spent on trying things that somebody already knows don't work is staggering. The knowledge of what doesn't work is mostly passed on tacitly through apprenticeship rather than published, which means it can easily be lost. As a simple example, we've known how to make transparent glass for 1900 years, but it took Ben Krasnow two months to achieve it himself despite having money, libraries, and glass experts at his disposal, and being generally competent at making things, having built, for example, his own electron microscope: https://www.youtube.com/watch?v=mUcUy7SqdS0 https://www.youtube.com/watch?v=mUcUy7SqdS0

dhosek 5y ago

This is true in all fields. I was playing with an idea that factors of Fermat non-primes (2^{2^n}+1 for n > 5) could be expressed in terms of products of the complex factors of the polynomial x^{2^n}+1 with x=1 with the idea that it might lead to a proof that for all n > 5, F_n is composite. I spent a bunch of time working out the exact expressions for these roots and playing around with their products (which ended up having a connection to Chebyshev polynomials), and then it occurred to me to try exhaustively trying the possible products of the complex roots of x^{64}+1 with x=1 to see if it turned up the factors of F_6 and it turned out it was a dead end. A compilation of false hypotheses in different fields of mathematics could make for interesting reading (and perhaps lead to eliminating multiple people getting stuck in the same cul de sac).

noslenwerdna 5y ago
Interestingly, this isn't true in particle physics! Probably most papers published by ATLAS or CMS are a negative result.

generationP 5y ago

As the Surgisphere affair has showed, publishing null results opens its own can of worms: No one will bother replicating them. Surprising positives at least have a kind of target drawn on them, but a negative result that matches people's expectations is really not something anyone disinterested would want to replicate -- there is no fame in that. Fraudulent non-results will be blocking fruitful directions. I wish I had a panacea for these things...

zozbot234 5y ago
The Surgisphere papers were not "null" results. They found significantly higher risk of death after HCQ treatment, based on what turned out to be highly flawed data. The problem was with the original dataset. (Not to be outdone, the Surgisphere folks also contributed their data to a preprint which purported to show horse dewormer^H^H^H^H^H^H^H^H^H^H^H^H^H^H ivermectin could be a successful SARS-CoV2 treatment).
- sampo 5y ago
  > horse dewormer^H^H^H^H^H^H^H^H^H^H^H^H^H^H ivermectin You are not being truthful, labeling ivermectin only a "horse dewormer". Ivermectin is used as an antiparasitic drug also for people. And not only as a dewormer, but also against lice and mites. https://en.wikipedia.org/wiki/Ivermectin https://en.wikipedia.org/wiki/Ivermectin
  - kragen 5y ago
    Indeed. One of the most disheartening things about the covid pandemic has been seeing how many people are willing to propagate the most egregious health-related misinformation, such as that ivermectin is only or primarily a horse dewormer or that it is dangerous, as long as it identifies them with the desired faction. Such depravity should not have come as a surprise to me, but it did. I've taken ivermectin—for parasites, not for covid. I wonder how many people will die of parasitic infections or more toxic antihelminthics and acaricides in the next few years because they or their neighbors refused to take ivermectin after they heard it was a dangerous veterinary-only drug from unscrupulous liars during the covid pandemic.