13 ms·

First analysis of ‘pre-registered’ studies shows sharp rise in null findings

415 points by danso 8y ago

xupybd 8y ago
This would be a huge leap forward. I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. A null finding might not be great for the researcher but it's a win for science.
- hn_throwaway_99 8y ago
  Agreed - this should help "level the playing field" so scientists can at least be somewhat more evaluated on the quality of their hypotheses and scientific protocols as opposed to the (largely governed by luck) outcome of their studies.
  - forapurpose 8y ago
    > the (largely governed by luck) outcome of their studies I've never heard that. Is there some evidence that it's largely luck? Does the theory imply that Einstein and Newton were extremely lucky?
    - mettamage 8y ago
      Even if it doesn't, there's a whole world outside of theoretical physics.
      - forapurpose 8y ago
        So Darwin was very lucky? The questions remain for any field: Is there evidence, and is the implication that the leading scientists are consider very lucky?
        drb91 8y ago
        Nah, I think you’re missing the point that you can google “bullshit psych studies” and get pages of results. This isn’t an indictment of science, this is an indictment of journals and academia today. Had this registration been in effect 300 years ago, Newton/Darwin would still be fine.
        pjc50 8y ago
        Yes, he was the lucky person chosen to be the field geologist on the Beagle
        goto11 8y ago
        Darwin was lucky that Wallace came to the same theory independently but contacted Darwin for feedback. This forced Darwin to publish his own results. If Wallace hadn't contacted Darwin before publishing, Darwin would probably be an unknown today.
    - drb91 8y ago
      They have the advantage of being in fairly trivially replicatable fields, so we can say with confidence they found something true. As to luck, who can say?
    - xupybd 8y ago
      Not the best examples. They both came up with analytical models and tested them. They were also amazing outliers. In that that achieved so much. For your everyday empirical research there is a lot of luck. You have to disprove a lot of possibilities to get to the true data. And as we only reward the positive findings your lucky if you’re test that one truth. If you’re sifting through drugs to find there possible other uses you’re going to have a lot of null results. But those are good they’re still extra knowledge.
    - buboard 8y ago
      You re almost creating a straw man. There have probably been scientists as intelligent or more intelligent than einstein, but worked on fields that proved fruitless, or didn't manage to finish their work within a lifetime. There is a large element of luck in any research, which has to align with capability. The matter at hand is how to keep those promising people in a science career and not the just the ones that publish a lot. Publishing negative but honest results is a step. Though i hope it doesn't end up becoming gamed as well.
      - whatshisface 8y ago
        Einstien did his Nobel prize winning work (the photoelectric effect) very early in life, and then went on to continually make big successes. If he had started out spinning his wheels he would have had plenty of time to switch.
        buboard 8y ago
        that doesnt mean he was immune to unsuccessful investigations, in which he spent a large part of his later life https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_investigations https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_inve...
    - hn_throwaway_99 8y ago
      I'm not 100% sure about Newton but Einstein didn't do any empirical research, which is the topic of this discussion.
- wallace_f 8y ago
  >I can't blame researchers for tailoring their research to help their own careers What about a lowly restaurant-owner who skimps on food quality or sanitary costs to "help their career?" A psychopath or "can't blame them?" Erroneous research can also have human costs. White collar and blue collar workers should both be scrutinized equally. I'm not sure where this acceptance of moral depravity among more-privileged peoples comes from, but I really don't like it.
  - jfz 8y ago
    There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. Researchers are currently under no obligation to write up their null findings, and it would be hard to get null findings in prestigious journals. This should be fixed at an institutional level, like the pre-registration of studies in the article, not as a mandate to each individual researcher to write up and publish every study.
    - wallace_f 8y ago
      >There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. This is the most-downvoted comment I've ever made on HN, and I think your explanation is in summary what people disagree with. However, the problems of bias and integrity in scientific research can and do have costs in terms of harm to human life. It's just that the connection between just following incentives and bad scientific research is much more abstracted, and therefore is not clearly intentional negligence, as the case with something like food safety.
      - btrettel 8y ago
        There's a great recent blog post on the basic point you're making: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
        8y ago
- deleted 8y ago
  [deleted]
- cabalamat 8y ago
  > A null finding might not be great for the researcher The fix for that would be to change science funding so a null finding doesn't hurt the researcher's career.
  - dmichulke 8y ago
    It already doesn't, null findings are simply not published. Unless of course you want to reward null findings which might, depending on the circumstances, open another can of worms.
    - thiagotomei 8y ago
      At least in my field (high energy physics in the LHC), null findings are routinely published. Everybody who is searching for physics beyond the standard model publishes their null findings left and right.
      - whatshisface 8y ago
        Above, a commenter makes a distinction between negative and null result, saying that null results are those without any statistically significant conclusion (neither negative nor positive). Those don't get published, the BSM papers usually rule out parts of parameter space which makes them negative results.
  - taurath 8y ago
    The interests of humanity and the interests of the organizations that have the capital to fund science projects are not frequently aligned.
    - justinjlynn 8y ago
      This must change.
    - 08-15 8y ago
      Exactly! Stop government funded research! Today, most science is funded by government agencies, like the NIH. This means, bureaucrats who know nothing about science allocate other people's money to research they don't care about. Since they don't understand the research, they need objective measures of "research quality", and they picked number of publications and impact factor of the journals as measures. What you measure becomes your objective, and we got "publish or perish" and all sorts of scientific misconduct. Science needs a free market, too.
- hannob 8y ago
  > I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. While I agree that mechanisms like registered reports are the way to fix these things at the core, I do think that these mentalities ("it's just the system, I can't do anything but cheat with my scientific publications, my career!") are a big part of the problem. Good read on the issue: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
  - deleted 8y ago
    [deleted]
  - Devils_Avocado 8y ago
    So, do 'scientists' not have dopamine pathways and an innate biological drive to maximise their personal longevity & reproductive success by securing control of as many resources as possible for as little risk as possible? "It's not the system; we only consistently get the same negative result that's different from what we want & expect because we have the wrong people staffing it. Don't change the status quo, change human neurological architecture".
  - gaius 8y ago
    It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. Likewise. The BBC reports that 2/3 of scientific research cannot be reproduced. Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778
    - lancebeet 8y ago
      If we want to give people the benefit of the doubt, it could also mean that scientific research is difficult and even competent individuals often make mistakes.
    - SubiculumCode 8y ago
- PeterisP 8y ago
  One thing that this achieves is that it makes null findings publishable. At least in my field, I pretty much have to discard null results because no reasonable venue will publish them; it's not worth the time to write them up if they'll be rejected anyway - at best, no-improvement results can get a mention, a line in a table, or possibly a paragraph in a published paper about something that actually does make a difference.
m3kw9 8y ago
Negative findings are just as good to publish so people don’t repeat mistake twice
- roywiggins 8y ago
  Negative findings don't prove that the idea was a mistake- it's just failure to find sufficient evidence. You might have just been unlucky.
  - nneonneo 8y ago
    Meta-analyses based on lots of studies, though, benefit from having unbiased publication of study results.
    - jfz 8y ago
      This is critical - even if no individual study has positive results, the data from each study can be aggregated and that meta-analysis can find results much more easily. I remember reading (but can't find the link to) an article about a medicine that took many trials to have one trial with statistical significance, but a meta-analysis of the first five or so trials would have revealed the efficacy of the drug under test much sooner and at a much lower cost.
      - tmalsburg2 8y ago
        Meta-analyses suffer from publication bias, though. Studies with one outcome are more likely to be published than those with some other outcome, and therefore the input for the meta-anslysis is biased.
- gus_massa 8y ago
  The problem is that it is too easy to have wrong ideas. If there only metric is how many studies you (or your group) have published, some will try to game it.
  - 8y ago
deleted 8y ago
[deleted]
RandomInteger4 8y ago
The obsession with positive findings is the most absurd thing when you think about it right? Like, I've read anecdotes from people saying that the editors or whomever at some of these journals might turn down publishing null findings. Think about that for a second. What does that tell you about their mindset? What possible reason would you have for not publishing null findings? Editor: "We gotta move these journals johnny! They need those spicy findings. If the findings aren't spicy, this stuff won't sell." That's hyperbole, but that's essentially the only reasoning I can think of, and it's absurd. Like, what professionals reading journals are going to be like "Whelp, the findings in this journal haven't been spicy. I can't dab on this nonsense. I'm going to start reading the other journal." said no researcher ever; neither literally or in essence.
- SquishyPanda23 8y ago
  Null findings are often not as useful as you might think. A point null hypothesis for a continuous variable is literally always false. Especially for the softer science, I've even seen studies mocked for having too many data points, since it's known that with enough data null hypotheses are false. The story is better if your null hypothesis is an interval, but then you're really just obliquely using the interval to bound something you could be measuring more directly anyway. What I'd like to see is moving away from null hypothesis testing altogether and focusing on measuring things. For example, focusing on measuring effect sizes, or the probability that a hypothesis is true.
  - thiagotomei 8y ago
    But what does "measuring more directly anyway" mean if you're trying to measure something that may or may not exist? For instance, in searches for new phenomena in high-energy physics, one usually puts an upper limit on deviations from the expectation of "known physics" (i.e., standard model). That essentially translates to statements like "if this particle exists, its mass should be higher than X TeV, or else we would have seen it already in our data". Of course, in reality, the particle probably does not exist, so you cannot really measure its mass!
salty_biscuits 8y ago
Do you think they pre registered their study into the effect of pre registered studies on null hypothesis rate?
- hn_throwaway_99 8y ago
  This was a meta-analysis, not a study of its own.
  - jeremyjh 8y ago
    What difference does that make?
    - laGrenouille 8y ago
      The point, I think, is that it means something slightly different to per-register an hypothesis that depends on retrospective data. When you register to hypothesis, the data already exist and one could (in theory) have already started looking at it. One would hope that researchers would not explicitly cheat like this, but it would be hard for experts in the fields to not have some sense of the trends seen within their field prior to running the meta analysis.
      - escherplex 8y ago
        Yeah but of course hypotheses don't arise ex nihilo but from a catalog of focused perceptions catalyzing conceptions which may suggest hypotheses encompassing novel ordering functions, innovations, ameliorations (or whatnot) depending on the field of study. If an individual considers a hypothesis of sufficient potential pragmatic value to research and document then, objectively, if negative results manifest these too need be reported as negative data germane to other researchers in the field. Problem is current academic culture is predisposed to reward only success. If your beautiful theory is murdered by an ugly fact (Huxley) then you don't get your doctorate or your departmental status diminishes since only positive results are reinforced. From this arises the temptation to be disingenuous or outright prevaricate, if you can get away with it. Does seem a cultural change is in order. Registering hypotheses which guarantees publication regardless of results may be useful.
monktastic1 8y ago
If I'm understanding correctly, the novel studies had a higher hypothesis success rate than the replicated studies. I imagine this is because there were many (successful) "debunking" attempts?
- PeterisP 8y ago
  There's the aspect of why you would choose to replicate a particular study instead of doing something else - there's always more ideas than your ability to execute them. At least for me, there are two cases why I'd bother to do that and why I could write in a grant proposal that this work is necessary - either I want to build on that study; in which case most likely I wouldn't publish just a pure replication but rather a comparison of my changes with my replication of the original study, and this paper would be counted as a novel study. Alternatively, I'm doing a replication because I'm not certain whether the original study is actually true, because I have some solid reason to believe that it's wrong.
  - pbhjpbhj 8y ago
    All studies need directly replicating, surely.
    - PeterisP 8y ago
      Possibly yes, someday, if they're still relevant. I mean, not every paper even deserves to be read, there's a lot of garbage published somewhere with an imitation of peer review. There's a lot of publications that have never been cited and likely won't ever be, much less replicated. As I said before, at any point of time in any scale of research the next research steps that "need to be done" vastly outnumber the resources to do it, so obviously not all of these things can be done. The majority of reasonable grant proposals get rejected, so that research doesn't get done - it's all a matter of prioritization; unless there's solid argument that this research task is within, say, the top 20% of the important research tasks, it won't get done. And most studies are not so important to justify repeating the effort; perhaps it was justified to expend X resources to get to that result, it doesn't necessarily mean that it's worth to spend 2X resources to get to a slightly more certain result after replication. It only needs replicating if lots of people are going to build their research on top of these results, and that simply doesn't happen for most studies.
forapurpose 8y ago
I wonder how much of the bias for publishing positive results is a holdover from the days of having only paper journals: On the Internet, we have unlimited room for publishing results; there's no reason not to publish the negative ones. When research was published only in paper journals, there was scarce space even for positive results; possibly it would have been considered a waste to use that precious space for negative results. Also, people reading those journals had limited time and wanted the most important results; they may have expected what we would call a 'curated' collection of studies. These days the studies are published in databases and nobody expects them all.
- Jolter 8y ago
  The Internet may have unlimited space for publishing, but it does not have unlimited bandwidth in peer review. So, the increased publication of null results will still mean slightly fewer published positive results.
  - forapurpose 8y ago
    Will the negative results be peer reviewed?
forapurpose 8y ago
Here's one way to think of the value of positive or negative results: Everyone talks about the value of negative results, but will we ever see one of those studies on the front page of HN?
cbkeller 8y ago
Geologist here. This is super-important for hypothesis science. At the same time, I think one of the biggest inaccuracies in the public perception of science (and one of my pet peeves) is the idea that all science is hypothesis science. It turns out there's also still plenty of discovery science to be done -- and while it's less common than it was 100 or 200 years ago, it's quite important! In geology, this often quite literally takes the form of a blank space on the map -- there are plenty of unmapped quadrangles on the geologic map of the world at 1:24,000 and finer scales, and the USGS will pay you just to learn how to fill them in [1]. This is one of the few subjects on which the misinformation is so pervasive that even the wikipedia article [2] is substantially inaccurate (I blame the overly-simplified formulation of the scientific method that most of us are first exposed to in elementary school). The one part that you can correctly infer from the wp article though, is that discovery science is having a bit of a resurgence recently thanks to the proliferation and reuse of large datasets. [1] https://ncgmp.usgs.gov/about/edmap.html https://ncgmp.usgs.gov/about/edmap.html [2] https://en.wikipedia.org/wiki/Discovery_science https://en.wikipedia.org/wiki/Discovery_science One of the simplest distinguishing characteristics is that there's no such thing as a negative result in discovery science: if you're mapping a blank area, whatever you find will be something we didn't know before.
- SubiculumCode 8y ago
  I think neuroscience is much like exploration. You can make hypotheses, but we barely know what is going on, so the hypotheses aren't really well grounded. Find hats there, publish the result, build the map, so to speak, so that future scientists can make more informed hypotheses.
  - cbkeller 8y ago
    Yeah. In areas where we know so little that it's hard to make informed hypotheses, there's probably a substantial component of discovery science. It's a bit of a yin-yang situation IMO: discovery science often leads to testable hypotheses for future work, and things we learn from hypothesis-testing can open up new avenues for discovery
  - tgb 8y ago
bo1024 8y ago
Hard to disentangle two possible mechanisms: (1) null findings that would normally get tossed are being submitted and accepted at higher rates (i.e. preregistration increases acceptability of null findings), (2) null findings that would normally get p-hacked into positive findings and published are not (preregistration working as intended). Both good.
lolc 8y ago
> Allen adds that their analysis is exploratory, and that there could be other explanations for the findings. I like how they're clear how their findings are dogfood at this point. They should pre-register a proper meta-study now :-) In a more serious vein: Do they actually intend to pre-register meta-studies too?
honestoHeminway 8y ago
Imagine if null-results would be retroactively registerd. It would be horrifying, seeing all those students doing the same work over and over again, not publishing it.
chiefalchemist 8y ago
But wouldn't be beneficial to have these published in public somewhere? Else, I would imagine, there could be plenty of "re-invention" going on. Also, at the very least, if I had a general idea, these "nulls" would help me to further refine what I might want to poke at. The way null is used here is wrong and misleading.
- 0xffff2 8y ago
  Null results are beneficial to the scientific community at large, but detrimental to the researcher(s) publishing the null result. This results from a system in which researchers are judged on a metric approximated by the percentage of positive results they publish.
  - chiefalchemist 8y ago
    Not so sound snarky (I know that's not welcome here) but that doesn't sound very scientific to me. If ultimately science is a method-based process to find truth, then hiding facts feels hypocritical. No doubt, I agree with you. We all do. My rub is that I see repeated calls from science that the public bow to its mastery. Unfortunately, a institutional lack of transparency is hardly grounds for trust. Perhaps it's time for science to hold itself up to it's own standards?
Fuxy 8y ago
Not to mention even a study that didn't pan out still has a lot of data that may be used for something else in the future. Even a failed creation has parts that can be re used in other projects after all.

415 points by danso 8y ago

This would be a huge leap forward. I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. A null finding might not be great for the researcher but it's a win for science.

hn_throwaway_99 8y ago
Agreed - this should help "level the playing field" so scientists can at least be somewhat more evaluated on the quality of their hypotheses and scientific protocols as opposed to the (largely governed by luck) outcome of their studies.
- forapurpose 8y ago
  > the (largely governed by luck) outcome of their studies I've never heard that. Is there some evidence that it's largely luck? Does the theory imply that Einstein and Newton were extremely lucky?
  - mettamage 8y ago
    Even if it doesn't, there's a whole world outside of theoretical physics.
    - forapurpose 8y ago
      So Darwin was very lucky? The questions remain for any field: Is there evidence, and is the implication that the leading scientists are consider very lucky?
      - drb91 8y ago
        Nah, I think you’re missing the point that you can google “bullshit psych studies” and get pages of results. This isn’t an indictment of science, this is an indictment of journals and academia today. Had this registration been in effect 300 years ago, Newton/Darwin would still be fine.
      - pjc50 8y ago
        Yes, he was the lucky person chosen to be the field geologist on the Beagle
      - goto11 8y ago
        Darwin was lucky that Wallace came to the same theory independently but contacted Darwin for feedback. This forced Darwin to publish his own results. If Wallace hadn't contacted Darwin before publishing, Darwin would probably be an unknown today.
  - drb91 8y ago
    They have the advantage of being in fairly trivially replicatable fields, so we can say with confidence they found something true. As to luck, who can say?
  - xupybd 8y ago
    Not the best examples. They both came up with analytical models and tested them. They were also amazing outliers. In that that achieved so much. For your everyday empirical research there is a lot of luck. You have to disprove a lot of possibilities to get to the true data. And as we only reward the positive findings your lucky if you’re test that one truth. If you’re sifting through drugs to find there possible other uses you’re going to have a lot of null results. But those are good they’re still extra knowledge.
  - buboard 8y ago
    You re almost creating a straw man. There have probably been scientists as intelligent or more intelligent than einstein, but worked on fields that proved fruitless, or didn't manage to finish their work within a lifetime. There is a large element of luck in any research, which has to align with capability. The matter at hand is how to keep those promising people in a science career and not the just the ones that publish a lot. Publishing negative but honest results is a step. Though i hope it doesn't end up becoming gamed as well.
    - whatshisface 8y ago
      Einstien did his Nobel prize winning work (the photoelectric effect) very early in life, and then went on to continually make big successes. If he had started out spinning his wheels he would have had plenty of time to switch.
      - buboard 8y ago
        that doesnt mean he was immune to unsuccessful investigations, in which he spent a large part of his later life https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_investigations https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_inve...
  - hn_throwaway_99 8y ago
    I'm not 100% sure about Newton but Einstein didn't do any empirical research, which is the topic of this discussion.
wallace_f 8y ago
>I can't blame researchers for tailoring their research to help their own careers What about a lowly restaurant-owner who skimps on food quality or sanitary costs to "help their career?" A psychopath or "can't blame them?" Erroneous research can also have human costs. White collar and blue collar workers should both be scrutinized equally. I'm not sure where this acceptance of moral depravity among more-privileged peoples comes from, but I really don't like it.
- jfz 8y ago
  There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. Researchers are currently under no obligation to write up their null findings, and it would be hard to get null findings in prestigious journals. This should be fixed at an institutional level, like the pre-registration of studies in the article, not as a mandate to each individual researcher to write up and publish every study.
  - wallace_f 8y ago
    >There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. This is the most-downvoted comment I've ever made on HN, and I think your explanation is in summary what people disagree with. However, the problems of bias and integrity in scientific research can and do have costs in terms of harm to human life. It's just that the connection between just following incentives and bad scientific research is much more abstracted, and therefore is not clearly intentional negligence, as the case with something like food safety.
    - btrettel 8y ago
      There's a great recent blog post on the basic point you're making: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
      - 8y ago
deleted 8y ago
[deleted]
cabalamat 8y ago
> A null finding might not be great for the researcher The fix for that would be to change science funding so a null finding doesn't hurt the researcher's career.
- dmichulke 8y ago
  It already doesn't, null findings are simply not published. Unless of course you want to reward null findings which might, depending on the circumstances, open another can of worms.
  - thiagotomei 8y ago
    At least in my field (high energy physics in the LHC), null findings are routinely published. Everybody who is searching for physics beyond the standard model publishes their null findings left and right.
    - whatshisface 8y ago
      Above, a commenter makes a distinction between negative and null result, saying that null results are those without any statistically significant conclusion (neither negative nor positive). Those don't get published, the BSM papers usually rule out parts of parameter space which makes them negative results.
- taurath 8y ago
  The interests of humanity and the interests of the organizations that have the capital to fund science projects are not frequently aligned.
  - justinjlynn 8y ago
    This must change.
  - 08-15 8y ago
    Exactly! Stop government funded research! Today, most science is funded by government agencies, like the NIH. This means, bureaucrats who know nothing about science allocate other people's money to research they don't care about. Since they don't understand the research, they need objective measures of "research quality", and they picked number of publications and impact factor of the journals as measures. What you measure becomes your objective, and we got "publish or perish" and all sorts of scientific misconduct. Science needs a free market, too.
hannob 8y ago
> I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. While I agree that mechanisms like registered reports are the way to fix these things at the core, I do think that these mentalities ("it's just the system, I can't do anything but cheat with my scientific publications, my career!") are a big part of the problem. Good read on the issue: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
- deleted 8y ago
  [deleted]
- Devils_Avocado 8y ago
  So, do 'scientists' not have dopamine pathways and an innate biological drive to maximise their personal longevity & reproductive success by securing control of as many resources as possible for as little risk as possible? "It's not the system; we only consistently get the same negative result that's different from what we want & expect because we have the wrong people staffing it. Don't change the status quo, change human neurological architecture".
- gaius 8y ago
  It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. Likewise. The BBC reports that 2/3 of scientific research cannot be reproduced. Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778
  - lancebeet 8y ago
    If we want to give people the benefit of the doubt, it could also mean that scientific research is difficult and even competent individuals often make mistakes.
  - SubiculumCode 8y ago
PeterisP 8y ago
One thing that this achieves is that it makes null findings publishable. At least in my field, I pretty much have to discard null results because no reasonable venue will publish them; it's not worth the time to write them up if they'll be rejected anyway - at best, no-improvement results can get a mention, a line in a table, or possibly a paragraph in a published paper about something that actually does make a difference.

m3kw9 8y ago

Negative findings are just as good to publish so people don’t repeat mistake twice

roywiggins 8y ago
Negative findings don't prove that the idea was a mistake- it's just failure to find sufficient evidence. You might have just been unlucky.
- nneonneo 8y ago
  Meta-analyses based on lots of studies, though, benefit from having unbiased publication of study results.
  - jfz 8y ago
    This is critical - even if no individual study has positive results, the data from each study can be aggregated and that meta-analysis can find results much more easily. I remember reading (but can't find the link to) an article about a medicine that took many trials to have one trial with statistical significance, but a meta-analysis of the first five or so trials would have revealed the efficacy of the drug under test much sooner and at a much lower cost.
    - tmalsburg2 8y ago
      Meta-analyses suffer from publication bias, though. Studies with one outcome are more likely to be published than those with some other outcome, and therefore the input for the meta-anslysis is biased.
gus_massa 8y ago
The problem is that it is too easy to have wrong ideas. If there only metric is how many studies you (or your group) have published, some will try to game it.
- 8y ago

deleted 8y ago

[deleted]

RandomInteger4 8y ago

The obsession with positive findings is the most absurd thing when you think about it right? Like, I've read anecdotes from people saying that the editors or whomever at some of these journals might turn down publishing null findings. Think about that for a second. What does that tell you about their mindset? What possible reason would you have for not publishing null findings? Editor: "We gotta move these journals johnny! They need those spicy findings. If the findings aren't spicy, this stuff won't sell." That's hyperbole, but that's essentially the only reasoning I can think of, and it's absurd. Like, what professionals reading journals are going to be like "Whelp, the findings in this journal haven't been spicy. I can't dab on this nonsense. I'm going to start reading the other journal." said no researcher ever; neither literally or in essence.

SquishyPanda23 8y ago
Null findings are often not as useful as you might think. A point null hypothesis for a continuous variable is literally always false. Especially for the softer science, I've even seen studies mocked for having too many data points, since it's known that with enough data null hypotheses are false. The story is better if your null hypothesis is an interval, but then you're really just obliquely using the interval to bound something you could be measuring more directly anyway. What I'd like to see is moving away from null hypothesis testing altogether and focusing on measuring things. For example, focusing on measuring effect sizes, or the probability that a hypothesis is true.
- thiagotomei 8y ago
  But what does "measuring more directly anyway" mean if you're trying to measure something that may or may not exist? For instance, in searches for new phenomena in high-energy physics, one usually puts an upper limit on deviations from the expectation of "known physics" (i.e., standard model). That essentially translates to statements like "if this particle exists, its mass should be higher than X TeV, or else we would have seen it already in our data". Of course, in reality, the particle probably does not exist, so you cannot really measure its mass!

salty_biscuits 8y ago

Do you think they pre registered their study into the effect of pre registered studies on null hypothesis rate?

hn_throwaway_99 8y ago
This was a meta-analysis, not a study of its own.
- jeremyjh 8y ago
  What difference does that make?
  - laGrenouille 8y ago
    The point, I think, is that it means something slightly different to per-register an hypothesis that depends on retrospective data. When you register to hypothesis, the data already exist and one could (in theory) have already started looking at it. One would hope that researchers would not explicitly cheat like this, but it would be hard for experts in the fields to not have some sense of the trends seen within their field prior to running the meta analysis.
    - escherplex 8y ago
      Yeah but of course hypotheses don't arise ex nihilo but from a catalog of focused perceptions catalyzing conceptions which may suggest hypotheses encompassing novel ordering functions, innovations, ameliorations (or whatnot) depending on the field of study. If an individual considers a hypothesis of sufficient potential pragmatic value to research and document then, objectively, if negative results manifest these too need be reported as negative data germane to other researchers in the field. Problem is current academic culture is predisposed to reward only success. If your beautiful theory is murdered by an ugly fact (Huxley) then you don't get your doctorate or your departmental status diminishes since only positive results are reinforced. From this arises the temptation to be disingenuous or outright prevaricate, if you can get away with it. Does seem a cultural change is in order. Registering hypotheses which guarantees publication regardless of results may be useful.

monktastic1 8y ago

If I'm understanding correctly, the novel studies had a higher hypothesis success rate than the replicated studies. I imagine this is because there were many (successful) "debunking" attempts?

PeterisP 8y ago
There's the aspect of why you would choose to replicate a particular study instead of doing something else - there's always more ideas than your ability to execute them. At least for me, there are two cases why I'd bother to do that and why I could write in a grant proposal that this work is necessary - either I want to build on that study; in which case most likely I wouldn't publish just a pure replication but rather a comparison of my changes with my replication of the original study, and this paper would be counted as a novel study. Alternatively, I'm doing a replication because I'm not certain whether the original study is actually true, because I have some solid reason to believe that it's wrong.
- pbhjpbhj 8y ago
  All studies need directly replicating, surely.
  - PeterisP 8y ago
    Possibly yes, someday, if they're still relevant. I mean, not every paper even deserves to be read, there's a lot of garbage published somewhere with an imitation of peer review. There's a lot of publications that have never been cited and likely won't ever be, much less replicated. As I said before, at any point of time in any scale of research the next research steps that "need to be done" vastly outnumber the resources to do it, so obviously not all of these things can be done. The majority of reasonable grant proposals get rejected, so that research doesn't get done - it's all a matter of prioritization; unless there's solid argument that this research task is within, say, the top 20% of the important research tasks, it won't get done. And most studies are not so important to justify repeating the effort; perhaps it was justified to expend X resources to get to that result, it doesn't necessarily mean that it's worth to spend 2X resources to get to a slightly more certain result after replication. It only needs replicating if lots of people are going to build their research on top of these results, and that simply doesn't happen for most studies.

forapurpose 8y ago

I wonder how much of the bias for publishing positive results is a holdover from the days of having only paper journals: On the Internet, we have unlimited room for publishing results; there's no reason not to publish the negative ones. When research was published only in paper journals, there was scarce space even for positive results; possibly it would have been considered a waste to use that precious space for negative results. Also, people reading those journals had limited time and wanted the most important results; they may have expected what we would call a 'curated' collection of studies. These days the studies are published in databases and nobody expects them all.

Jolter 8y ago
The Internet may have unlimited space for publishing, but it does not have unlimited bandwidth in peer review. So, the increased publication of null results will still mean slightly fewer published positive results.
- forapurpose 8y ago
  Will the negative results be peer reviewed?

forapurpose 8y ago

Here's one way to think of the value of positive or negative results: Everyone talks about the value of negative results, but will we ever see one of those studies on the front page of HN?

cbkeller 8y ago

Geologist here. This is super-important for hypothesis science. At the same time, I think one of the biggest inaccuracies in the public perception of science (and one of my pet peeves) is the idea that all science is hypothesis science. It turns out there's also still plenty of discovery science to be done -- and while it's less common than it was 100 or 200 years ago, it's quite important! In geology, this often quite literally takes the form of a blank space on the map -- there are plenty of unmapped quadrangles on the geologic map of the world at 1:24,000 and finer scales, and the USGS will pay you just to learn how to fill them in [1]. This is one of the few subjects on which the misinformation is so pervasive that even the wikipedia article [2] is substantially inaccurate (I blame the overly-simplified formulation of the scientific method that most of us are first exposed to in elementary school). The one part that you can correctly infer from the wp article though, is that discovery science is having a bit of a resurgence recently thanks to the proliferation and reuse of large datasets. [1] https://ncgmp.usgs.gov/about/edmap.html https://ncgmp.usgs.gov/about/edmap.html [2] https://en.wikipedia.org/wiki/Discovery_science https://en.wikipedia.org/wiki/Discovery_science One of the simplest distinguishing characteristics is that there's no such thing as a negative result in discovery science: if you're mapping a blank area, whatever you find will be something we didn't know before.

SubiculumCode 8y ago
I think neuroscience is much like exploration. You can make hypotheses, but we barely know what is going on, so the hypotheses aren't really well grounded. Find hats there, publish the result, build the map, so to speak, so that future scientists can make more informed hypotheses.
- cbkeller 8y ago
  Yeah. In areas where we know so little that it's hard to make informed hypotheses, there's probably a substantial component of discovery science. It's a bit of a yin-yang situation IMO: discovery science often leads to testable hypotheses for future work, and things we learn from hypothesis-testing can open up new avenues for discovery
- tgb 8y ago

bo1024 8y ago

Hard to disentangle two possible mechanisms: (1) null findings that would normally get tossed are being submitted and accepted at higher rates (i.e. preregistration increases acceptability of null findings), (2) null findings that would normally get p-hacked into positive findings and published are not (preregistration working as intended). Both good.

lolc 8y ago

> Allen adds that their analysis is exploratory, and that there could be other explanations for the findings. I like how they're clear how their findings are dogfood at this point. They should pre-register a proper meta-study now :-) In a more serious vein: Do they actually intend to pre-register meta-studies too?

honestoHeminway 8y ago

Imagine if null-results would be retroactively registerd. It would be horrifying, seeing all those students doing the same work over and over again, not publishing it.

chiefalchemist 8y ago

But wouldn't be beneficial to have these published in public somewhere? Else, I would imagine, there could be plenty of "re-invention" going on. Also, at the very least, if I had a general idea, these "nulls" would help me to further refine what I might want to poke at. The way null is used here is wrong and misleading.

0xffff2 8y ago
Null results are beneficial to the scientific community at large, but detrimental to the researcher(s) publishing the null result. This results from a system in which researchers are judged on a metric approximated by the percentage of positive results they publish.
- chiefalchemist 8y ago
  Not so sound snarky (I know that's not welcome here) but that doesn't sound very scientific to me. If ultimately science is a method-based process to find truth, then hiding facts feels hypocritical. No doubt, I agree with you. We all do. My rub is that I see repeated calls from science that the public bow to its mastery. Unfortunately, a institutional lack of transparency is hardly grounds for trust. Perhaps it's time for science to hold itself up to it's own standards?

Fuxy 8y ago

Not to mention even a study that didn't pan out still has a lot of data that may be used for something else in the future. Even a failed creation has parts that can be re used in other projects after all.