4 ms·

The obsession with positive findings is the most absurd thing when you think about it right? Like, I've read anecdotes from people saying that the editors or w

The obsession with positive findings is the most absurd thing when you think about it right? Like, I've read anecdotes from people saying that the editors or whomever at some of these journals might turn down publishing null findings. Think about that for a second. What does that tell you about their mindset? What possible reason would you have for not publishing null findings? Editor: "We gotta move these journals johnny! They need those spicy findings. If the findings aren't spicy, this stuff won't sell." That's hyperbole, but that's essentially the only reasoning I can think of, and it's absurd. Like, what professionals reading journals are going to be like "Whelp, the findings in this journal haven't been spicy. I can't dab on this nonsense. I'm going to start reading the other journal." said no researcher ever; neither literally or in essence.

SquishyPanda23 8y ago
Null findings are often not as useful as you might think. A point null hypothesis for a continuous variable is literally always false. Especially for the softer science, I've even seen studies mocked for having too many data points, since it's known that with enough data null hypotheses are false. The story is better if your null hypothesis is an interval, but then you're really just obliquely using the interval to bound something you could be measuring more directly anyway. What I'd like to see is moving away from null hypothesis testing altogether and focusing on measuring things. For example, focusing on measuring effect sizes, or the probability that a hypothesis is true.
- thiagotomei 8y ago
  But what does "measuring more directly anyway" mean if you're trying to measure something that may or may not exist? For instance, in searches for new phenomena in high-energy physics, one usually puts an upper limit on deviations from the expectation of "known physics" (i.e., standard model). That essentially translates to statements like "if this particle exists, its mass should be higher than X TeV, or else we would have seen it already in our data". Of course, in reality, the particle probably does not exist, so you cannot really measure its mass!
  - SquishyPanda23 8y ago
    Sorry this is so late, but you can measure the probability that the particle exists. Null hypothesis tests basically try to calculate the probability of a data set given the null hypothesis. What you really want is the probability of a hypothesis given the data set. So in that case, you want to estimate the probability of theories of physics, such as those that include the particle and those that don't.
- wnoise 8y ago
  s/probability/odds ratio/
Retra 8y ago
Nobody ever won a Nobel prize for a null result... So it's not entirely without reason that people would prefer to get positive results.
- mbb70 8y ago
  While of course true, there have been a few groundbreaking null results such as the Michelson–Morley experiment.
  - evanb 8y ago
    Michelson did win the Nobel prize, though the citation reads "for his optical precision instruments and the spectroscopic and metrological investigations carried out with their aid", so not explicitly for the null result.
ubershmekel 8y ago
You damn well know you can more easily dab on positive findings. There should probably be a Nobel Prize for null findings to fix the incentive structure. But how do you grade and compare the different null findings? By effort? The ramifications of a null finding are likely more limited.
- pgeorgi 8y ago
  Since Nobel Prizes are rather detached from the discovery at hands, it might be possible to evaluate the impact of null findings on later positive findings. Essentially: when awarding a Nobel Prize, map out which null findings narrowed the path sufficiently to support the Nobel Prize worthy work, and give them recognition as "supporting acts".
TangoTrotFox 8y ago
There's a much more simple reason. A null result can often just indicate a bad hypothesis, and there are lots of bad hypotheses. On the other hand I think something that does support your point is that there are also a lot of bad hypotheses being published, increasingly even in reputable journals, after what is clearly extensive p-hacking. 'So, yeah we took these 29 variables measured in arbitrary, yet extremely specific, fashion, and lo and behold - our hypothesis is affirmed!' It's hard to see how these papers get published outside of the 'spiciness'.
SubiculumCode 8y ago
Null results often don't mean anything other than insufficient power, poor study design, failure to control for relevant variables, etc. The reasoning against publishing null results is that one cannot prove a null, and therefore there is no finding, AND that the null finding could be due to uninteresting reasons (e.g. sample size). With per-registered studies, there is greater focus on sample size and study design, since the study will get published either way. This balances out the concerns somewhat.
08-15 8y ago
The obsession with p values is the most absurd thing. You write as if there are "positive findings" and "negative findings". This isn't true in orthodox statistics. There are only "negative findings" (the null hypothesis is rejected) and "null findings" (nothing is rejected, nothing is confirmed). Only the negative findings get published. What doesn't exist is "positive findings". Nothing is ever confirmed: not the null (it's assumed to be true) and not the alternative (it's not even tested). Now who wants to print a journal in which 95 out of 100 articles say "we learned nothing" and the other 5 can't be reproduced? Much better to print a journal in which every articles claims a results, even if none of them can be reproduced.
- pbhjpbhj 8y ago
  All 100 are results. The studies saying the five can't be reproduced, where are they. If I'm designing an experiment to attempt to confirm a theoretical model then finding similarities in the 10 prior attempts that failed could give me clues as to what to try. Certainly if 10 respected labs have done things in exactly the way I was going to try then it's worth questioning long and hard whether I really need to repeat that procedure. Why did these all fail to reject the null hypothesis. That's a powerful question.
  - timr 8y ago
    "Why did these all fail to reject the null hypothesis. That's a powerful question." Because that's nearly always the outcome. By conventional statistical metrics, the null hypothesis isn't rejected >=95% of the time. "The studies saying the five can't be reproduced, where are they." They don't get published, because of the aforementioned statistical problem. The bias toward positive results isn't irrational; it's a natural response to the fact that the vast majority of what any scientist produces will be a "negative" result. The way you learn what not to try is by studying under experienced scientists, and talking to other current practitioners. For any field, there's a vast shared experience that guides experimentation. As a new researcher, a good place to find this kind of information is in review articles and book chapters. But mostly you get it by working with experienced people.

SquishyPanda23 8y ago

Null findings are often not as useful as you might think. A point null hypothesis for a continuous variable is literally always false. Especially for the softer science, I've even seen studies mocked for having too many data points, since it's known that with enough data null hypotheses are false. The story is better if your null hypothesis is an interval, but then you're really just obliquely using the interval to bound something you could be measuring more directly anyway. What I'd like to see is moving away from null hypothesis testing altogether and focusing on measuring things. For example, focusing on measuring effect sizes, or the probability that a hypothesis is true.

thiagotomei 8y ago
But what does "measuring more directly anyway" mean if you're trying to measure something that may or may not exist? For instance, in searches for new phenomena in high-energy physics, one usually puts an upper limit on deviations from the expectation of "known physics" (i.e., standard model). That essentially translates to statements like "if this particle exists, its mass should be higher than X TeV, or else we would have seen it already in our data". Of course, in reality, the particle probably does not exist, so you cannot really measure its mass!
- SquishyPanda23 8y ago
  Sorry this is so late, but you can measure the probability that the particle exists. Null hypothesis tests basically try to calculate the probability of a data set given the null hypothesis. What you really want is the probability of a hypothesis given the data set. So in that case, you want to estimate the probability of theories of physics, such as those that include the particle and those that don't.
wnoise 8y ago
s/probability/odds ratio/

Retra 8y ago

Nobody ever won a Nobel prize for a null result... So it's not entirely without reason that people would prefer to get positive results.

mbb70 8y ago
While of course true, there have been a few groundbreaking null results such as the Michelson–Morley experiment.
- evanb 8y ago
  Michelson did win the Nobel prize, though the citation reads "for his optical precision instruments and the spectroscopic and metrological investigations carried out with their aid", so not explicitly for the null result.

ubershmekel 8y ago

You damn well know you can more easily dab on positive findings. There should probably be a Nobel Prize for null findings to fix the incentive structure. But how do you grade and compare the different null findings? By effort? The ramifications of a null finding are likely more limited.

pgeorgi 8y ago
Since Nobel Prizes are rather detached from the discovery at hands, it might be possible to evaluate the impact of null findings on later positive findings. Essentially: when awarding a Nobel Prize, map out which null findings narrowed the path sufficiently to support the Nobel Prize worthy work, and give them recognition as "supporting acts".

TangoTrotFox 8y ago

There's a much more simple reason. A null result can often just indicate a bad hypothesis, and there are lots of bad hypotheses. On the other hand I think something that does support your point is that there are also a lot of bad hypotheses being published, increasingly even in reputable journals, after what is clearly extensive p-hacking. 'So, yeah we took these 29 variables measured in arbitrary, yet extremely specific, fashion, and lo and behold - our hypothesis is affirmed!' It's hard to see how these papers get published outside of the 'spiciness'.

SubiculumCode 8y ago

Null results often don't mean anything other than insufficient power, poor study design, failure to control for relevant variables, etc. The reasoning against publishing null results is that one cannot prove a null, and therefore there is no finding, AND that the null finding could be due to uninteresting reasons (e.g. sample size). With per-registered studies, there is greater focus on sample size and study design, since the study will get published either way. This balances out the concerns somewhat.

08-15 8y ago

The obsession with p values is the most absurd thing. You write as if there are "positive findings" and "negative findings". This isn't true in orthodox statistics. There are only "negative findings" (the null hypothesis is rejected) and "null findings" (nothing is rejected, nothing is confirmed). Only the negative findings get published. What doesn't exist is "positive findings". Nothing is ever confirmed: not the null (it's assumed to be true) and not the alternative (it's not even tested). Now who wants to print a journal in which 95 out of 100 articles say "we learned nothing" and the other 5 can't be reproduced? Much better to print a journal in which every articles claims a results, even if none of them can be reproduced.

pbhjpbhj 8y ago
All 100 are results. The studies saying the five can't be reproduced, where are they. If I'm designing an experiment to attempt to confirm a theoretical model then finding similarities in the 10 prior attempts that failed could give me clues as to what to try. Certainly if 10 respected labs have done things in exactly the way I was going to try then it's worth questioning long and hard whether I really need to repeat that procedure. Why did these all fail to reject the null hypothesis. That's a powerful question.
- timr 8y ago
  "Why did these all fail to reject the null hypothesis. That's a powerful question." Because that's nearly always the outcome. By conventional statistical metrics, the null hypothesis isn't rejected >=95% of the time. "The studies saying the five can't be reproduced, where are they." They don't get published, because of the aforementioned statistical problem. The bias toward positive results isn't irrational; it's a natural response to the fact that the vast majority of what any scientist produces will be a "negative" result. The way you learn what not to try is by studying under experienced scientists, and talking to other current practitioners. For any field, there's a vast shared experience that guides experimentation. As a new researcher, a good place to find this kind of information is in review articles and book chapters. But mostly you get it by working with experienced people.