7 ms·

Researchers value null results, but struggle to publish them

144 points by Bluestein 1y ago

BrenBarn 1y ago
They avoid mentioning the elephant in the room: jobs and tenure. When you can get hired for a tenure-track job based on your null-result publications, and can get tenure for your null-result publications, then people will publish null results. Until then, they won't hit the mainstream.
- antithesizer 1y ago
  It's fascinating how utterly dominated science is by economics. Even truth itself needs an angle.
  - youainti 1y ago
    Imagine being an economist... You can't get away from it.
  - Bluestein 1y ago
    > Even truth itself needs an angle. "When even truth itself needs an angle ... ... every lie looks like a viable alternative".-
  - pixl97 1y ago
    I mean, science has always been in one way or another. All the 'scientists' of olde were either wealthy or given some kind of grant by those that were. Science itself won't be exempt from the freeloader problem either. Not that I'm saying all science has to economic purposes.
  - autoexec 1y ago
    The influence of money really does hold back scientific progress and is often specifically used to prevent some truths from being known or to reduce the confidence we have in those truths. Obviously it takes money to do pretty much anything in our society but it does seem like it has way more influence that is necessary. Greed seems to corrupt everything, and even though we can identify areas where things can be improved nobody seems to be wiling or able to change course.
    - derektank 1y ago
      I don't see any way to get around the fact that science is expensive. Researching the boundaries of human knowledge requires access to rare or novel materials, which requires people to invest time and effort in producing them, which of course requires money. If your research requires access to vats of liquid oxygen, or high powered lasers, or ultra pure chemicals, or 500 pure bred rats, you'll need people working full time to produce those things for you, and they need to be able to put food on their table.
      - BrenBarn 1y ago
        There's plenty of scientific research being done outside the narrow physical-sciences type of fields you mention. Not all science is expensive in that way.
  - ytpete 1y ago
    I would think a large part of it is simply the universities want to offer tenure to the most competent researchers. And it's harder to judge whether a researcher who mostly gets null results is doing a good job. Perhaps this means it really does have to start with journal publications though. If journals value null results, peer reviewers will sharpen their ability to distinguish null but well-run experiments from ones that failed simply due to poor execution. Then employers can use published null results as a positive signal that a researcher is indeed doing good quality work.
- MITSardine 1y ago
  If people are so interested, they'd presumably read and cite null-result publications, and their authors would get the same boons as if having published a positive result. There's some issues, though. Firstly, how do you enforce citing negative results? In the case of positive results, reviewers can ask that work be cited if it had already introduced things present in the article. This is because a publication is a claim to originality. But how do you define originality in not following a given approach? Anyone can not have the idea of doing something. You can't well cite all the paths not followed in your work, considering you might not even be aware of a negative result publication regarding these ideas you discarded or didn't have. Bibliography is time consuming enough as it is, without having to also cite all things irrelevant. Another issue is that the effort to write an article and get it published and, on the other side, to review it, makes it hard to justify publishing negative results. I'd say an issue is rather that many positive results are already not getting published... There's a lot of informal knowledge, as people don't have time to write 100 page papers with all the tricks and details regularly, nor reviewers to read them. Also, I could see a larger acceptance of negative result publications bringing perverse incentives. Currently, you have to get somewhere eventually. If negative results become legitimate publications, what would e.g. PhD theses become? Oh, we tried to reinvent everything but nothing worked, here's 200 pages of negative results no-one would have reasonably tried anyways. While the current state of affairs favours incremental research, I think that is still better than no serious research at all.
  - michaelt 1y ago
    > If people are so interested, they'd presumably read and cite null-result publications, The thing is, people mostly cite work they're building upon, and it's often difficult to build much on a null result. If I'm an old-timey scientist trying to invent the first lightbulb, and I try a brass filament and it doesn't work, then I try a steel filament and it doesn't work, then I try an aluminium filament and it doesn't work - will anyone be interested in that? On the other hand, if I tested platinum or carbonised paper or something else that actually works? Well, there's a lot more to build on there, and a lot more to be interested in.
    - 1y ago
- throwawaymaths 1y ago
  This. And the incentives can be even more perverse: If you find a null result you might not want to let your competitors know, because they'll get stuck in the same sand trap.
- kurthr 1y ago
  There are a few different realities here. First, it's not really whether you can get tenure with the publications, because almost none of the major respected journals accept simple null/negative results for publication. It's too "boring". Now, they do occasionally publish "surprising" null/negative results, but that's usually do to rivalry or scandal. The counter example to some extent is medical/drug control trials, but those are pharma driven, and gov published though an academic could be on the paper, and it might find its way onto a tenure review. Second, in the beginning there is funding. If you don't have a grant for it, you don't do the research. Most grants are for "discoveries" and those only come about from "positive" scientific results. So the first path to this is to pay people to run the experiments (that nobody wants to see "fail"). Then, you have to trust that the people running them don't screw up the actual experiment, because there are an almost infinite number of ways to do things wrong, and only experts can even make things work at all for difficult modern science. Then you hope that the statistics are done well and not skewed, and hope a major journal publishes. Third, could a Journal of Negative Results that only published well run experiments, by respected experts, with good statistics and minimal bias be profitable? I don't know, a few exist, but I think it would take government or charity to get it off the ground, and a few big names to get people reading it for prestige. Otherwise, we're just talking about something on par with arXiv.org. It can't just be a journal that publishes every negative result or somehow reviewers have to experts in everything, since properly reviewing negative results from many fields is a HUGE challenge. My experience writing, and getting grants/research funded, is that there's a lot of bootstrapping where you use some initial funding to do research on some interesting topics and get some initial results, before you then propose to "do" that research (which you have high confidence will succeed) so that you can get funding to finish the next phase of research (and confirm the original work) to get the next grant. It's a cycle, and you don't dare break it, because if you "fail" to get "good" results from your research, and you don't get published, then your proposals for the next set of grants will be viewed very negatively!
j7ake 1y ago
There is zero incentive for the researcher personally to publish null results. Null results are the foundations on which “glossy” results are produced. Researchers would be wasting time giving away their competitive advantage by publishing null results.
- zeroCalories 1y ago
  What do you mean glossy results? Wouldn't it be to your advantage to take down another researcher? Or do you mean they use null results to construct a better theory for more credit?
  - j7ake 1y ago
    Glossy results are memeable stories that editors of journals would like to have in their next edition. There is very little incentive in publicly criticising a paper, there is incentive to show why others are wrong and why your new technically superior method solves the problem and finds new insights to the field.
bkanuka 1y ago
I studied physics in university, and found it challenging to find null-result publications to cite, which can be useful when proposing a new experiment or as background info for a non-null paper. I promised myself if I became ultra-wealthy I would start a "Journal of Null Science" to collect these publications. (this journal still doesn't exist)
- esafak 1y ago
  https://journal.trialanderror.org/ https://journal.trialanderror.org/ http://arjournals.com/ http://arjournals.com/
- jonah-archive 1y ago
  https://www.jasnh.com https://www.jasnh.com , introduced in 2002: https://web.archive.org/web/20020601214717/https://www.apa.org/monitor/newjournal.html https://web.archive.org/web/20020601214717/https://www.apa.o...
  - Bluestein 1y ago
    This is really really so necessary ... If really pro science, some non-profit should really fund this sort of research.- PS. Heck, if nothing else, it'd give synthetic intellection systems somewhere to not go with their research, and their agency and such ...
    - timr 1y ago
      Before tackling that, a non-profit should fund well-designed randomized controlled trials in areas where none exist. Which is most of them. Commit to funding and publishing the trial, regardless of outcome, once a cross-disciplinary panel of disinterested experts on trial statistics approve the pre-registered methodology. If there are too many qualified studies to fund, choose randomly. This alone would probably kill off a lot of fraudulent science in areas like nutrition and psychology. It's what the government should be doing with NIH and NSF funding, but is not. If you manage to get a good RCT through execution & publication, that should make your career, regardless of outcome.
andix 1y ago
Journals could fix that. They could create a category null results and dedicate it a fixed amount of pages (like 20%). Researchers want to be in journals, if this category doesn’t have a lot of submissions it would be much easier to get published.
- Bluestein 1y ago
  This is great idea.- Heck, "we tried, and did not get there but ..." should be a category unto itself.-
- vouaobrasil 1y ago
  Journals are mainly interested in profit, not fixing anything.
  - spragl 1y ago
    Surely publishing a result is not in itself costly. But I guess the peer review is. So journals could have a section (the grey pages?) for "unsellable results" that they didnt give a peer review. They would of course need to assess them in some other way, to ensure a reasonable level of quality.
  - mattkrause 1y ago
    That should be only sort of true. Some of the very high-profile journals are run by non-profits, including: Science (American Association for the Advancement of Science), PNAS, (National Academy of Sciences), eLife (HHMI/Max Planck/Wellcome Trust). A slew of more specialized journals are run by societies too. In theory, they should be willing to lead the charge. In practice, I think they are largely dependent on income from the journals for a lot of their operations and so are reluctant to rock the boat.
pacbard 1y ago
Not all null results are created equal. There are interesting null results that get published and are well known. For example, Card & Kruger (1994) was a null result paper showing that increasing the minimum wage has a null effect on employment rates. This result went against the common assumption that increasing wages will decrease employment at the time. Other null results are either dirty (e.g., big standard errors) or due to process problems (e.g., experimental failure). These are more difficult to publish because it's difficult to learn anything new from these results. The challenge is that researchers do not know if they are going to get a "good" null or a "bad" one. Most of the time, you have to invest significant effort and time into a project, only to get a null result at the end. These results are difficult to publish in most cases and can lead to the end of careers if someone is pre-tenure or lead to funding problems for anyone.
- kazinator 1y ago
  What matters for publication is a surprising result, not whether it confirms the main hypothesis or the null one. The "psychological null hypothesis" is that which follows the common assumption, whether that assumption states that there is a relationship between the variables or that there is not.
- MITSardine 1y ago
  Is that actually a null result though? That sounds like a standard positive result: "We managed to show that minimum wage has no effect on employment rate". A null result would have been: "We tried to apply Famous Theory to showing that minimum wage has no effect on employment rate but we failed because this and that".
  - Analemma_ 1y ago
    No, because in theory a minimum wage increase could decrease the unemployment rate. If it does neither, that’s a null result.
    - mitthrowaway2 1y ago
      That's sort of like saying "I measured acceleration due to gravity by dropping a bowling ball off the Tower of Pisa, and got a null result: the ball simply hovered in midair." If the result is very surprising and contradicts established theory, I wouldn't consider it a null result, even if some parameter numerically measures as zero.
simpaticoder 1y ago
You could publish them as a listicle "10 falsehoods organic chemists believe!" Because behind most every null result was an hypothesis that sounded like it was probably true. Most likely, it would sound probably true to most people in the field, so publishing the result is of real value to others. The problem arises that null results are cheap and easy to "find" for things no-one thinks sound plausible, and therefore a trivial way to game the publish or perish system. I suspect that this alone explains the bias against publishing null results.
physarum_salad 1y ago
[dead]
qcnguy 1y ago
They could just publish them on arxiv or their own websites. Publishing is easy. What they mean is, they struggle to get attention to them in a way that boosts their own career by pumping the metrics their administrators care about. Not the same thing at all.
chris_armstrong 1y ago
Onlynulls, the subscription-only service you’d be too ashamed to have your supervisor find you on.
janalsncm 1y ago
Never worked in academia, but in industry null results are really valuable: “we tried that, it didn’t work, don’t waste your time”. Sometimes very intuitive things just don’t work. Sometimes people are less inclined to share those things more publicly. In a perfect world, there’s still a forcing function to get researchers to publish null results. Maybe the head of a department publishes the research they tried but didn’t work out. I wonder how much money has been lost on repeatedly trying the same approaches that don’t work.
sumanthvepa 1y ago
The Michelson-Morely experiment is probably one of the most famous null result in all of science. https://en.wikipedia.org/wiki/Michelson%E2%80%93Morley_experiment https://en.wikipedia.org/wiki/Michelson%E2%80%93Morley_exper...
- dr_dshiv 1y ago
  It would have been a LOT more famous if it hadn’t been null. But, yes — they provided giant shoulders for Einstein to stand on.
Nevermark 1y ago
We just need a site for posting null result reports. With really good keyword, search functionality. Much less formal requirements than regular papers, make it easy. But with some common sense guidelines. And public upvotes, and commenting, so contributors get some feedback love, and failures can attract potential helpful turnaround ideas. And of course, annual rewards. For humor & not so serious (because, why so serious?) street cred, but with the serous mission of raising consciousness about how negative results are not some narrow bit of information, but that attempts and results, bad or not, are rich sources of new ideas.
alex-moon 1y ago
It's amazing to see this on the front page of HN as it came up in a discussion with my partner early in our relationship. I was saying something about how a lot of people don't understand the robustness of peer review and replication. I was gushing about how it's the most perfect system of knowledge advancement and she replied, "I mean, it's not perfect though," and then said, pretty much verbatim, the title of this article. The problem specifically isn't so much that null results don't get published, it's that they get published as a positive result in something the researchers weren't studying - they have to change their hypothesis retroactively to make it look like a positive result. Worse, this leads to studies that are designed to attempt to study as many things as possible, to hedge their bets. These studies suffer from quality problems because of course you can't really study something deeply if you're controlling a lot of variables.
- Bluestein 1y ago
  That's crazy, the serendipity.- PS. Good point about the "shotgun" studies.-
- SubiculumCode 1y ago
  I won't say it doesn't happen. I'll just say that I haven't seen it happen in my science career on any of the projects I have worked on.
bhk 1y ago
Ironic. Isn't Nature the worst offender?
more_corn 1y ago
There really should be a journal dedicated to this.
danjl 1y ago
Why not have a standard section in every successful paper that covers all of the null results that led up to the successful conclusion? There are lots of standard sections. Seems easy to add a requirement for the failures that led to the success. Many papers already include tips, tricks and gotchas, perhaps we just need to formalize this a bit more. Seems much easier than trying to find a way to get people to publish full papers on null results.

144 points by Bluestein 1y ago

They avoid mentioning the elephant in the room: jobs and tenure. When you can get hired for a tenure-track job based on your null-result publications, and can get tenure for your null-result publications, then people will publish null results. Until then, they won't hit the mainstream.

antithesizer 1y ago
It's fascinating how utterly dominated science is by economics. Even truth itself needs an angle.
- youainti 1y ago
  Imagine being an economist... You can't get away from it.
- Bluestein 1y ago
  > Even truth itself needs an angle. "When even truth itself needs an angle ... ... every lie looks like a viable alternative".-
- pixl97 1y ago
  I mean, science has always been in one way or another. All the 'scientists' of olde were either wealthy or given some kind of grant by those that were. Science itself won't be exempt from the freeloader problem either. Not that I'm saying all science has to economic purposes.
- autoexec 1y ago
  The influence of money really does hold back scientific progress and is often specifically used to prevent some truths from being known or to reduce the confidence we have in those truths. Obviously it takes money to do pretty much anything in our society but it does seem like it has way more influence that is necessary. Greed seems to corrupt everything, and even though we can identify areas where things can be improved nobody seems to be wiling or able to change course.
  - derektank 1y ago
    I don't see any way to get around the fact that science is expensive. Researching the boundaries of human knowledge requires access to rare or novel materials, which requires people to invest time and effort in producing them, which of course requires money. If your research requires access to vats of liquid oxygen, or high powered lasers, or ultra pure chemicals, or 500 pure bred rats, you'll need people working full time to produce those things for you, and they need to be able to put food on their table.
    - BrenBarn 1y ago
      There's plenty of scientific research being done outside the narrow physical-sciences type of fields you mention. Not all science is expensive in that way.
- ytpete 1y ago
  I would think a large part of it is simply the universities want to offer tenure to the most competent researchers. And it's harder to judge whether a researcher who mostly gets null results is doing a good job. Perhaps this means it really does have to start with journal publications though. If journals value null results, peer reviewers will sharpen their ability to distinguish null but well-run experiments from ones that failed simply due to poor execution. Then employers can use published null results as a positive signal that a researcher is indeed doing good quality work.
MITSardine 1y ago
If people are so interested, they'd presumably read and cite null-result publications, and their authors would get the same boons as if having published a positive result. There's some issues, though. Firstly, how do you enforce citing negative results? In the case of positive results, reviewers can ask that work be cited if it had already introduced things present in the article. This is because a publication is a claim to originality. But how do you define originality in not following a given approach? Anyone can not have the idea of doing something. You can't well cite all the paths not followed in your work, considering you might not even be aware of a negative result publication regarding these ideas you discarded or didn't have. Bibliography is time consuming enough as it is, without having to also cite all things irrelevant. Another issue is that the effort to write an article and get it published and, on the other side, to review it, makes it hard to justify publishing negative results. I'd say an issue is rather that many positive results are already not getting published... There's a lot of informal knowledge, as people don't have time to write 100 page papers with all the tricks and details regularly, nor reviewers to read them. Also, I could see a larger acceptance of negative result publications bringing perverse incentives. Currently, you have to get somewhere eventually. If negative results become legitimate publications, what would e.g. PhD theses become? Oh, we tried to reinvent everything but nothing worked, here's 200 pages of negative results no-one would have reasonably tried anyways. While the current state of affairs favours incremental research, I think that is still better than no serious research at all.
- michaelt 1y ago
  > If people are so interested, they'd presumably read and cite null-result publications, The thing is, people mostly cite work they're building upon, and it's often difficult to build much on a null result. If I'm an old-timey scientist trying to invent the first lightbulb, and I try a brass filament and it doesn't work, then I try a steel filament and it doesn't work, then I try an aluminium filament and it doesn't work - will anyone be interested in that? On the other hand, if I tested platinum or carbonised paper or something else that actually works? Well, there's a lot more to build on there, and a lot more to be interested in.
  - 1y ago
throwawaymaths 1y ago
This. And the incentives can be even more perverse: If you find a null result you might not want to let your competitors know, because they'll get stuck in the same sand trap.
kurthr 1y ago
There are a few different realities here. First, it's not really whether you can get tenure with the publications, because almost none of the major respected journals accept simple null/negative results for publication. It's too "boring". Now, they do occasionally publish "surprising" null/negative results, but that's usually do to rivalry or scandal. The counter example to some extent is medical/drug control trials, but those are pharma driven, and gov published though an academic could be on the paper, and it might find its way onto a tenure review. Second, in the beginning there is funding. If you don't have a grant for it, you don't do the research. Most grants are for "discoveries" and those only come about from "positive" scientific results. So the first path to this is to pay people to run the experiments (that nobody wants to see "fail"). Then, you have to trust that the people running them don't screw up the actual experiment, because there are an almost infinite number of ways to do things wrong, and only experts can even make things work at all for difficult modern science. Then you hope that the statistics are done well and not skewed, and hope a major journal publishes. Third, could a Journal of Negative Results that only published well run experiments, by respected experts, with good statistics and minimal bias be profitable? I don't know, a few exist, but I think it would take government or charity to get it off the ground, and a few big names to get people reading it for prestige. Otherwise, we're just talking about something on par with arXiv.org. It can't just be a journal that publishes every negative result or somehow reviewers have to experts in everything, since properly reviewing negative results from many fields is a HUGE challenge. My experience writing, and getting grants/research funded, is that there's a lot of bootstrapping where you use some initial funding to do research on some interesting topics and get some initial results, before you then propose to "do" that research (which you have high confidence will succeed) so that you can get funding to finish the next phase of research (and confirm the original work) to get the next grant. It's a cycle, and you don't dare break it, because if you "fail" to get "good" results from your research, and you don't get published, then your proposals for the next set of grants will be viewed very negatively!

j7ake 1y ago

There is zero incentive for the researcher personally to publish null results. Null results are the foundations on which “glossy” results are produced. Researchers would be wasting time giving away their competitive advantage by publishing null results.

zeroCalories 1y ago
What do you mean glossy results? Wouldn't it be to your advantage to take down another researcher? Or do you mean they use null results to construct a better theory for more credit?
- j7ake 1y ago
  Glossy results are memeable stories that editors of journals would like to have in their next edition. There is very little incentive in publicly criticising a paper, there is incentive to show why others are wrong and why your new technically superior method solves the problem and finds new insights to the field.

bkanuka 1y ago

I studied physics in university, and found it challenging to find null-result publications to cite, which can be useful when proposing a new experiment or as background info for a non-null paper. I promised myself if I became ultra-wealthy I would start a "Journal of Null Science" to collect these publications. (this journal still doesn't exist)

esafak 1y ago
https://journal.trialanderror.org/ https://journal.trialanderror.org/ http://arjournals.com/ http://arjournals.com/
jonah-archive 1y ago
https://www.jasnh.com https://www.jasnh.com , introduced in 2002: https://web.archive.org/web/20020601214717/https://www.apa.org/monitor/newjournal.html https://web.archive.org/web/20020601214717/https://www.apa.o...
- Bluestein 1y ago
  This is really really so necessary ... If really pro science, some non-profit should really fund this sort of research.- PS. Heck, if nothing else, it'd give synthetic intellection systems somewhere to not go with their research, and their agency and such ...
  - timr 1y ago
    Before tackling that, a non-profit should fund well-designed randomized controlled trials in areas where none exist. Which is most of them. Commit to funding and publishing the trial, regardless of outcome, once a cross-disciplinary panel of disinterested experts on trial statistics approve the pre-registered methodology. If there are too many qualified studies to fund, choose randomly. This alone would probably kill off a lot of fraudulent science in areas like nutrition and psychology. It's what the government should be doing with NIH and NSF funding, but is not. If you manage to get a good RCT through execution & publication, that should make your career, regardless of outcome.

andix 1y ago

Journals could fix that. They could create a category null results and dedicate it a fixed amount of pages (like 20%). Researchers want to be in journals, if this category doesn’t have a lot of submissions it would be much easier to get published.

Bluestein 1y ago
This is great idea.- Heck, "we tried, and did not get there but ..." should be a category unto itself.-
vouaobrasil 1y ago
Journals are mainly interested in profit, not fixing anything.
- spragl 1y ago
  Surely publishing a result is not in itself costly. But I guess the peer review is. So journals could have a section (the grey pages?) for "unsellable results" that they didnt give a peer review. They would of course need to assess them in some other way, to ensure a reasonable level of quality.
- mattkrause 1y ago
  That should be only sort of true. Some of the very high-profile journals are run by non-profits, including: Science (American Association for the Advancement of Science), PNAS, (National Academy of Sciences), eLife (HHMI/Max Planck/Wellcome Trust). A slew of more specialized journals are run by societies too. In theory, they should be willing to lead the charge. In practice, I think they are largely dependent on income from the journals for a lot of their operations and so are reluctant to rock the boat.

pacbard 1y ago

Not all null results are created equal. There are interesting null results that get published and are well known. For example, Card & Kruger (1994) was a null result paper showing that increasing the minimum wage has a null effect on employment rates. This result went against the common assumption that increasing wages will decrease employment at the time. Other null results are either dirty (e.g., big standard errors) or due to process problems (e.g., experimental failure). These are more difficult to publish because it's difficult to learn anything new from these results. The challenge is that researchers do not know if they are going to get a "good" null or a "bad" one. Most of the time, you have to invest significant effort and time into a project, only to get a null result at the end. These results are difficult to publish in most cases and can lead to the end of careers if someone is pre-tenure or lead to funding problems for anyone.

kazinator 1y ago
What matters for publication is a surprising result, not whether it confirms the main hypothesis or the null one. The "psychological null hypothesis" is that which follows the common assumption, whether that assumption states that there is a relationship between the variables or that there is not.
MITSardine 1y ago
Is that actually a null result though? That sounds like a standard positive result: "We managed to show that minimum wage has no effect on employment rate". A null result would have been: "We tried to apply Famous Theory to showing that minimum wage has no effect on employment rate but we failed because this and that".
- Analemma_ 1y ago
  No, because in theory a minimum wage increase could decrease the unemployment rate. If it does neither, that’s a null result.
  - mitthrowaway2 1y ago
    That's sort of like saying "I measured acceleration due to gravity by dropping a bowling ball off the Tower of Pisa, and got a null result: the ball simply hovered in midair." If the result is very surprising and contradicts established theory, I wouldn't consider it a null result, even if some parameter numerically measures as zero.

simpaticoder 1y ago

You could publish them as a listicle "10 falsehoods organic chemists believe!" Because behind most every null result was an hypothesis that sounded like it was probably true. Most likely, it would sound probably true to most people in the field, so publishing the result is of real value to others. The problem arises that null results are cheap and easy to "find" for things no-one thinks sound plausible, and therefore a trivial way to game the publish or perish system. I suspect that this alone explains the bias against publishing null results.

physarum_salad 1y ago

[dead]

qcnguy 1y ago

They could just publish them on arxiv or their own websites. Publishing is easy. What they mean is, they struggle to get attention to them in a way that boosts their own career by pumping the metrics their administrators care about. Not the same thing at all.

chris_armstrong 1y ago

Onlynulls, the subscription-only service you’d be too ashamed to have your supervisor find you on.

janalsncm 1y ago

Never worked in academia, but in industry null results are really valuable: “we tried that, it didn’t work, don’t waste your time”. Sometimes very intuitive things just don’t work. Sometimes people are less inclined to share those things more publicly. In a perfect world, there’s still a forcing function to get researchers to publish null results. Maybe the head of a department publishes the research they tried but didn’t work out. I wonder how much money has been lost on repeatedly trying the same approaches that don’t work.

sumanthvepa 1y ago

The Michelson-Morely experiment is probably one of the most famous null result in all of science. https://en.wikipedia.org/wiki/Michelson%E2%80%93Morley_experiment https://en.wikipedia.org/wiki/Michelson%E2%80%93Morley_exper...

dr_dshiv 1y ago
It would have been a LOT more famous if it hadn’t been null. But, yes — they provided giant shoulders for Einstein to stand on.

Nevermark 1y ago

We just need a site for posting null result reports. With really good keyword, search functionality. Much less formal requirements than regular papers, make it easy. But with some common sense guidelines. And public upvotes, and commenting, so contributors get some feedback love, and failures can attract potential helpful turnaround ideas. And of course, annual rewards. For humor & not so serious (because, why so serious?) street cred, but with the serous mission of raising consciousness about how negative results are not some narrow bit of information, but that attempts and results, bad or not, are rich sources of new ideas.

alex-moon 1y ago

It's amazing to see this on the front page of HN as it came up in a discussion with my partner early in our relationship. I was saying something about how a lot of people don't understand the robustness of peer review and replication. I was gushing about how it's the most perfect system of knowledge advancement and she replied, "I mean, it's not perfect though," and then said, pretty much verbatim, the title of this article. The problem specifically isn't so much that null results don't get published, it's that they get published as a positive result in something the researchers weren't studying - they have to change their hypothesis retroactively to make it look like a positive result. Worse, this leads to studies that are designed to attempt to study as many things as possible, to hedge their bets. These studies suffer from quality problems because of course you can't really study something deeply if you're controlling a lot of variables.

Bluestein 1y ago
That's crazy, the serendipity.- PS. Good point about the "shotgun" studies.-
SubiculumCode 1y ago
I won't say it doesn't happen. I'll just say that I haven't seen it happen in my science career on any of the projects I have worked on.

bhk 1y ago

Ironic. Isn't Nature the worst offender?

more_corn 1y ago

There really should be a journal dedicated to this.

danjl 1y ago

Why not have a standard section in every successful paper that covers all of the null results that led up to the successful conclusion? There are lots of standard sections. Seems easy to add a requirement for the failures that led to the success. Many papers already include tips, tricks and gotchas, perhaps we just need to formalize this a bit more. Seems much easier than trying to find a way to get people to publish full papers on null results.