5 ms·

Highlight negative results to improve science

93 points by headalgorithm 7y ago

dekhn 7y ago
Reason #37 that I am no longer a scientist: most of my time as a scientist was spent attempting to replicate the positive results of other scientists and failing, because the work was not replicable. I actually trust negative results way more than positive results now- it's a proxy that you can trust the investigator. I wish we had a Journal of Negative Results.
- cheez 7y ago
  Don't let your dreams be dreams. Everyone would like this.
- rjyoungling 7y ago
  Such a fascinating take. When I was young I used to think that scientists and the scientific process were perfectly objective. As you get older you learn that human beings have human being problems. Those don't magically disappear once you become a scientist. You see this even in highly objective fields such as physics, where much progress comes from the previous generation simply dying. Our own ego's, as well as the factors mentioned in the article, create problems much more than you might expect when you're young and naive.
  - dekhn 7y ago
    Is physics truly objective? I see people still arguing about quantum mechanics, and it looks like the safest conclusion is that we don't actually live in a straightforward, temporally causal world (prediction: https://en.wikipedia.org/wiki/Wheeler%27s_delayed-choice_experiment https://en.wikipedia.org/wiki/Wheeler%27s_delayed-choice_exp..., experimental support: https://en.wikipedia.org/wiki/Delayed-choice_quantum_eraser https://en.wikipedia.org/wiki/Delayed-choice_quantum_eraser) Nobody's making progress because people are dying (we still see many vestiges of the Cophenhagen interpretation!)
    - rjyoungling 7y ago
      Yeah, I see your POV. Think it depends where you're coming from. Physics is objective if by objective you mean we can predict how the wavefunction will evolve according to the Schrodinger equation. (Depending on who you ask) it might be less objective if you ask: ''But what's REALLY going on?'' Then you start getting into the interpretations of qm and different scientists will have different views. I think it's at least likely that we'll be able to answer that question in a few decades (or at least move the ball forward). The view of Everettians, for example, is falsifiable and there are experiments going on as we speak trying to achieve that.
- denzil_correa 7y ago
  There are a few journals of negative results. For example, https://jnrbm.biomedcentral.com/ https://jnrbm.biomedcentral.com/
  - dekhn 7y ago
    "Journal of Negative Results in Biomedicine (JNRBM) ceased to be published by BioMed Central as of 1st September 2017"
- zachguo 7y ago
  That's also why I gave up pursuing a Ph.D. in psychology and shifted my focus from experiment design to computational analysis & modeling. I failed to replicate a few cornerstone experiments with modern equipment and a more diverse subject group, no matter how hard I tried. https://slate.com/technology/2013/05/weird-psychology-social-science-researchers-rely-too-much-on-western-college-students.html https://slate.com/technology/2013/05/weird-psychology-social...
- deleted 7y ago
  [deleted]
- hannob 7y ago
  There are Journals of Negative Results. This isn't really a fix, because people will still not publish negative results unless there's an incentive to do so. And there rarely is, because most negative results will be considered as boring (unless they disprove an existing, popular result). What we really need is to decouple the publishing decision from the outcome. A way to do this is to use registered reports.
- ablekh 7y ago
  I'm not sure about journals of negative results (though I ran across this effort: https://www.negative-results.org https://www.negative-results.org). However, I'm aware of, at least, one workshop on negative results. That was the 2015 ERROR workshop in e-science domain. Please see https://press3.mcs.anl.gov/errorworkshop https://press3.mcs.anl.gov/errorworkshop for details.
slowhand09 7y ago
relevant https://news.ycombinator.com/item?id=20997978 https://news.ycombinator.com/item?id=20997978
6gvONxR4sf7o 7y ago
There's another issue at play here that's tightly connected to this one. Underpowered studies. Here's the process. I look elsewhere (e.g. similar studies) to make an educated guess about the size of effect I'm going to look for. Let's say I land on 2 units. Then I ask how many samples I need to detect a 2 unit effect with some low false positive procedure. I want to detect my 2 unit effect with high probability if it's there. So you end up with three numbers. The probability you find something given that there's nothing, often 5%. The probability you find nothing given that there's something (something being a 2 unit effect), often 20%. The size of effect you're using in these calculations (2 units). The smaller effect size you're looking for, the more data you need. The lower error rates you're looking to get, the more data you need. So you make sacrifices and accept higher error rates. You make optimistic effect size estimates (making your true error rates even higher). Using p < 5% as a positive result publishing threshold is bad enough. Using power > 80% as a negative result publishing threshold is even crazier. That's a 20% error rate for publishing negatives. Higher if you used optimistic effect estimates. What this all means is yes we should publish negatives, but a negative worth listening to is going to be expensive. In a bayesian sense, it's the difference between me spending little money to learn that the effect of coffee on mood is probably between -1 and 1 (a bad negative result) vs spending a lot of money to learn that it's probably between -0.001 and 0.001 (a more useful negative result).
- SubiculumCode 7y ago
  Thank you for summarizing this. Overly powered studies are rare, and this is the primary reason that null results are not given the status as positive results do. They should go into a journal of negative results and boldly state a power analysis.
Alex3917 7y ago
If studies with negative results are as valuable as studies with positive results, do the sales curves for reprints look similar for the 100 best-selling studies with positive results and the 100 best-selling studies for negative results?
- celeritascelery 7y ago
  It’s not monetarily as valuable, but scientifically.
- warlog 7y ago
  The incentive to consider is that positive results get published, get funded, get you academic job s, get you more money, get you tenure. Negative results? Maybe you get to be a contrarian voice on a panel during the afternoon of the last day of a conference.
bryanmgreen 7y ago
Applicable across mostly everything, right? Relationships, business, Amazon product reviews.....
hannob 7y ago
So how many negative results has Nature published lately?

93 points by headalgorithm 7y ago

Reason #37 that I am no longer a scientist: most of my time as a scientist was spent attempting to replicate the positive results of other scientists and failing, because the work was not replicable. I actually trust negative results way more than positive results now- it's a proxy that you can trust the investigator. I wish we had a Journal of Negative Results.

cheez 7y ago
Don't let your dreams be dreams. Everyone would like this.
rjyoungling 7y ago
Such a fascinating take. When I was young I used to think that scientists and the scientific process were perfectly objective. As you get older you learn that human beings have human being problems. Those don't magically disappear once you become a scientist. You see this even in highly objective fields such as physics, where much progress comes from the previous generation simply dying. Our own ego's, as well as the factors mentioned in the article, create problems much more than you might expect when you're young and naive.
- dekhn 7y ago
  Is physics truly objective? I see people still arguing about quantum mechanics, and it looks like the safest conclusion is that we don't actually live in a straightforward, temporally causal world (prediction: https://en.wikipedia.org/wiki/Wheeler%27s_delayed-choice_experiment https://en.wikipedia.org/wiki/Wheeler%27s_delayed-choice_exp..., experimental support: https://en.wikipedia.org/wiki/Delayed-choice_quantum_eraser https://en.wikipedia.org/wiki/Delayed-choice_quantum_eraser) Nobody's making progress because people are dying (we still see many vestiges of the Cophenhagen interpretation!)
  - rjyoungling 7y ago
    Yeah, I see your POV. Think it depends where you're coming from. Physics is objective if by objective you mean we can predict how the wavefunction will evolve according to the Schrodinger equation. (Depending on who you ask) it might be less objective if you ask: ''But what's REALLY going on?'' Then you start getting into the interpretations of qm and different scientists will have different views. I think it's at least likely that we'll be able to answer that question in a few decades (or at least move the ball forward). The view of Everettians, for example, is falsifiable and there are experiments going on as we speak trying to achieve that.
denzil_correa 7y ago
There are a few journals of negative results. For example, https://jnrbm.biomedcentral.com/ https://jnrbm.biomedcentral.com/
- dekhn 7y ago
  "Journal of Negative Results in Biomedicine (JNRBM) ceased to be published by BioMed Central as of 1st September 2017"
zachguo 7y ago
That's also why I gave up pursuing a Ph.D. in psychology and shifted my focus from experiment design to computational analysis & modeling. I failed to replicate a few cornerstone experiments with modern equipment and a more diverse subject group, no matter how hard I tried. https://slate.com/technology/2013/05/weird-psychology-social-science-researchers-rely-too-much-on-western-college-students.html https://slate.com/technology/2013/05/weird-psychology-social...
deleted 7y ago
[deleted]
hannob 7y ago
There are Journals of Negative Results. This isn't really a fix, because people will still not publish negative results unless there's an incentive to do so. And there rarely is, because most negative results will be considered as boring (unless they disprove an existing, popular result). What we really need is to decouple the publishing decision from the outcome. A way to do this is to use registered reports.
ablekh 7y ago
I'm not sure about journals of negative results (though I ran across this effort: https://www.negative-results.org https://www.negative-results.org). However, I'm aware of, at least, one workshop on negative results. That was the 2015 ERROR workshop in e-science domain. Please see https://press3.mcs.anl.gov/errorworkshop https://press3.mcs.anl.gov/errorworkshop for details.

slowhand09 7y ago

relevant https://news.ycombinator.com/item?id=20997978 https://news.ycombinator.com/item?id=20997978

6gvONxR4sf7o 7y ago

There's another issue at play here that's tightly connected to this one. Underpowered studies. Here's the process. I look elsewhere (e.g. similar studies) to make an educated guess about the size of effect I'm going to look for. Let's say I land on 2 units. Then I ask how many samples I need to detect a 2 unit effect with some low false positive procedure. I want to detect my 2 unit effect with high probability if it's there. So you end up with three numbers. The probability you find something given that there's nothing, often 5%. The probability you find nothing given that there's something (something being a 2 unit effect), often 20%. The size of effect you're using in these calculations (2 units). The smaller effect size you're looking for, the more data you need. The lower error rates you're looking to get, the more data you need. So you make sacrifices and accept higher error rates. You make optimistic effect size estimates (making your true error rates even higher). Using p < 5% as a positive result publishing threshold is bad enough. Using power > 80% as a negative result publishing threshold is even crazier. That's a 20% error rate for publishing negatives. Higher if you used optimistic effect estimates. What this all means is yes we should publish negatives, but a negative worth listening to is going to be expensive. In a bayesian sense, it's the difference between me spending little money to learn that the effect of coffee on mood is probably between -1 and 1 (a bad negative result) vs spending a lot of money to learn that it's probably between -0.001 and 0.001 (a more useful negative result).

SubiculumCode 7y ago
Thank you for summarizing this. Overly powered studies are rare, and this is the primary reason that null results are not given the status as positive results do. They should go into a journal of negative results and boldly state a power analysis.

Alex3917 7y ago

If studies with negative results are as valuable as studies with positive results, do the sales curves for reprints look similar for the 100 best-selling studies with positive results and the 100 best-selling studies for negative results?

celeritascelery 7y ago
It’s not monetarily as valuable, but scientifically.
warlog 7y ago
The incentive to consider is that positive results get published, get funded, get you academic job s, get you more money, get you tenure. Negative results? Maybe you get to be a contrarian voice on a panel during the afternoon of the last day of a conference.

bryanmgreen 7y ago

Applicable across mostly everything, right? Relationships, business, Amazon product reviews.....

hannob 7y ago

So how many negative results has Nature published lately?