4 ms·

> The researchers also surveyed the scientists involved to find out why so many animals were missing. The most common reasons they gave were that the studies di

> The researchers also surveyed the scientists involved to find out why so many animals were missing. The most common reasons they gave were that the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication I do not come from academia so I may be mistaken, but why would you publish a non statistically significant result? There seems to me there's a very distinct difference between "this is most likely true" and "this might be true, we can't say yet". The first is useful, the second adds to the chatter needlessly. Some may say that it promotes further research, but what if a false weakly proven result misleads instead?

GirkovArpa 6y ago
Imagine if a drug company ran a study 100 times before finding a "significant" result that supported the claims they wanted to make about their product. Without the other 99 studies, the single study is highly misleading.
- pixl97 6y ago
  Depends what methodologies were changed between studies. If people with a particular gene were effected, but said expression was very rare then you might have a huge number of studies that show no effect, but were highly effective in that one case.
  - rebuilder 6y ago
    Exactly, so if you only published the one study where this rare gene was present in a participant, you'd get a wildly distorted picture of the general effectiveness of your intervention.
    - pixl97 6y ago
      The issue comes in, especially in future medicines that only target particular genes, that you only care about the effectiveness in that limited case.
      - rebuilder 6y ago
        In that case, wouldn't you screen the trial participants for that gene?
- PeterisP 6y ago
  That's why you should implement (and require) pre-registration of clinical studies - however, that still just means that everyone knows that there are the other 99 unsuccessful studies, not that they will get written up in detail and published.
MattGaiser 6y ago
Because say that there are 20 studies trying to prove that the Earth is flat. The current system only rewards the 1 study that "proved" that. The 19 other studies which were inconclusive or demonstrated otherwise were not published. Now, if you had 19 studies which were inconclusive and 1 study that claimed to have conclusions, you would look at it very differently than one study that claimed to have conclusions.
- refurb 6y ago
  This is true, but at least in drug development (where I have some experience), no two studies are alike. Even when a drug company runs two phase 3 trials (like they are supposed to in order to secure FDA approval), the design or execution varies enough that if one fails, it doesn't necessarily suggest that the other one is a false positive.
Reelin 6y ago
> > the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication > why would you publish a non statistically significant result There's actually two separate things going on here. The first is that researches (at least in the biomedical sciences) often decide not to waste their time writing up results that are either weak, inconclusive, or negative. This is only mildly controversial - everyone involved recognizes the inherent time availability constraint, but knowing what didn't work can help to inform the field at large in it's own way. The second is that the term "statistically significant" itself has become _highly_ controversial in recent years. The issue is that when you perform statistical tests, they generally (oversimplifying) spit out a number telling you what the odds of your results being correct are. In concrete terms, say you have some graphs that show a slight improvement in some condition when a drug is used. Is the drug actually effective, or is the "improvement" actually just random measurement errors? So you run a statistical test on your data and it gives you a score of 80%. It's telling you that there's an 80% chance that your results were due to an actual improvement and a 20% chance that they were due to random chance. If you had unlimited time and money you could just keep collecting results endlessly. Eventually, that score would either go towards 100% (it definitely works) or 0% (it definitely doesn't work). But this is the real world, where we don't have unlimited time and money (particularly academic researchers). So when is your result worth publishing? How do you decide when to throw in the towel? And if you're reviewing papers for a journal, how low a score is acceptable before you vote to reject the paper on the grounds that the conclusions are unreliable and not worth looking at? Enter the term "statistical significance". At some point, people started classifying scores that were above some arbitrary threshold as "significant" and those below it as "not significant". But there's an obvious problem here - some measure of probability flipping from (say) 89.999% to 90.000% doesn't do anything magical! Worse, what a future reader intends to do with the results will determine how important any given score is in that particular case. Clearly, results and their associated score need to be interpreted in context instead of blindly. Using a term such as "statistical significance" flies in the face of that by actively encouraging lazy thinking. So the controversy being referred to in that specific sentence you quoted isn't the decision not to publish but rather the usage of the term itself. (Which is confusing, because the article at large is addressing the controversy surrounding not publishing.)
Munksgaard 6y ago
Obligatory xkcd: https://xkcd.com/882/ https://xkcd.com/882/

GirkovArpa 6y ago

Imagine if a drug company ran a study 100 times before finding a "significant" result that supported the claims they wanted to make about their product. Without the other 99 studies, the single study is highly misleading.

pixl97 6y ago
Depends what methodologies were changed between studies. If people with a particular gene were effected, but said expression was very rare then you might have a huge number of studies that show no effect, but were highly effective in that one case.
- rebuilder 6y ago
  Exactly, so if you only published the one study where this rare gene was present in a participant, you'd get a wildly distorted picture of the general effectiveness of your intervention.
  - pixl97 6y ago
    The issue comes in, especially in future medicines that only target particular genes, that you only care about the effectiveness in that limited case.
    - rebuilder 6y ago
      In that case, wouldn't you screen the trial participants for that gene?
PeterisP 6y ago
That's why you should implement (and require) pre-registration of clinical studies - however, that still just means that everyone knows that there are the other 99 unsuccessful studies, not that they will get written up in detail and published.

MattGaiser 6y ago

Because say that there are 20 studies trying to prove that the Earth is flat. The current system only rewards the 1 study that "proved" that. The 19 other studies which were inconclusive or demonstrated otherwise were not published. Now, if you had 19 studies which were inconclusive and 1 study that claimed to have conclusions, you would look at it very differently than one study that claimed to have conclusions.

refurb 6y ago
This is true, but at least in drug development (where I have some experience), no two studies are alike. Even when a drug company runs two phase 3 trials (like they are supposed to in order to secure FDA approval), the design or execution varies enough that if one fails, it doesn't necessarily suggest that the other one is a false positive.

Reelin 6y ago

> > the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication > why would you publish a non statistically significant result There's actually two separate things going on here. The first is that researches (at least in the biomedical sciences) often decide not to waste their time writing up results that are either weak, inconclusive, or negative. This is only mildly controversial - everyone involved recognizes the inherent time availability constraint, but knowing what didn't work can help to inform the field at large in it's own way. The second is that the term "statistically significant" itself has become _highly_ controversial in recent years. The issue is that when you perform statistical tests, they generally (oversimplifying) spit out a number telling you what the odds of your results being correct are. In concrete terms, say you have some graphs that show a slight improvement in some condition when a drug is used. Is the drug actually effective, or is the "improvement" actually just random measurement errors? So you run a statistical test on your data and it gives you a score of 80%. It's telling you that there's an 80% chance that your results were due to an actual improvement and a 20% chance that they were due to random chance. If you had unlimited time and money you could just keep collecting results endlessly. Eventually, that score would either go towards 100% (it definitely works) or 0% (it definitely doesn't work). But this is the real world, where we don't have unlimited time and money (particularly academic researchers). So when is your result worth publishing? How do you decide when to throw in the towel? And if you're reviewing papers for a journal, how low a score is acceptable before you vote to reject the paper on the grounds that the conclusions are unreliable and not worth looking at? Enter the term "statistical significance". At some point, people started classifying scores that were above some arbitrary threshold as "significant" and those below it as "not significant". But there's an obvious problem here - some measure of probability flipping from (say) 89.999% to 90.000% doesn't do anything magical! Worse, what a future reader intends to do with the results will determine how important any given score is in that particular case. Clearly, results and their associated score need to be interpreted in context instead of blindly. Using a term such as "statistical significance" flies in the face of that by actively encouraging lazy thinking. So the controversy being referred to in that specific sentence you quoted isn't the decision not to publish but rather the usage of the term itself. (Which is confusing, because the article at large is addressing the controversy surrounding not publishing.)

Munksgaard 6y ago

Obligatory xkcd: https://xkcd.com/882/ https://xkcd.com/882/