4 ms·

The lack of null result publication is a real issue for research and scientific knowledge in general, including for accurate positive results (what if once you

by jraph 1y ago

The lack of null result publication is a real issue for research and scientific knowledge in general, including for accurate positive results (what if once you see a positive result by statistical chance but the experiment was actually conducted before you by 10 other people but that wasn't published because the results were null?) > Papers are written to invite replication They are written to share knowledge, new discoveries. We hope that they are replicated. > It's easy to call it shameful when it isn't you who has to do the work. We are not judging anyone, we are qualifying the situation. And we are speaking of publishing results for experiments that have already conducted, not voluntarily making up null cases and doing the related experiments. It wouldn't make sense and would be harmful. But if you did an experiment that produced a null result, not publishing is a loss of knowledge. Again, we are not judging anybody, it's a failure of the publishing system. But this cannot change if nobody points it out. I'd have troubles understanding a researcher not acknowledging today that the lack of null result publication is an issue. It would show a lack of perspective IMHO. And for someone acknowledging that there's an issue, pushing back is not the right stance.

9rx 1y ago
> They are written to share knowledge, new discoveries. We hope that they are replicated. If they aren't replicated, no new knowledge is gained. Not really. > it's a failure of the publishing system. What failure? Again, the "publishing system" is to submit the "best of the best" research to the world in order to invite replication. While nothing is perfect, we don't want people to have to sift through papers that have effectively no reason to be replicated in a quest to find something that is. That would make things infinitely worse. The internet was created for the minor-leagues. If you are willing to put in the effect to document your "less than the best" research, put it up on a website for all to see already. There is nothing stopping you. But who wants to put in the effort? > We are not judging anyone How do you define "shameful" if it isn't related to judgement of someone?
- jraph 1y ago
  You assume that null results are not worth it and are not the best of science. You seem to be ignoring the issue where researchers collectively waste time redoing "failing" studies again and again because the null results are not published, and other issues not publishing them causes. We fundamentally disagree here and I'm not willing to put in the effort needed to try to convince you otherwise, many other comments are here are better than what I could write. I do hope you take the time to take a step back (and you seem to be in defensive mode, if so, you need to go out of this) and reconsider. > How do you define "shameful" if it isn't related to judgement of someone? Do you know the expression "it's a shame"? > used when you wish a situation was different, and you feel sad or disappointed https://www.ldoceonline.com/dictionary/it-s-a-shame-what-a-shame-etc https://www.ldoceonline.com/dictionary/it-s-a-shame-what-a-s...
  - 9rx 1y ago
    > You assume that null results are not worth it and are not the best of science. No. I make no such assumption. That may often be true, but there is nothing to stop a useful null result from being published. We also get things wrong from time to time. It is very possible that the best baseball player in the world has been overlooked by Major League Baseball. We're pretty good at scoping out the best, but nothing in life is perfect. If the best baseball player in the world ends up in the minor leagues instead, oh well? Does it really matter? You can still watch them there. Same goes for research. If something great doesn't make it into the formal publication system, you can still read it on the studier's website (if they put in the effort to publish it). > You seem to be ignoring the issue where researchers collectively waste time redoing "failing" studies again and again because the null results are not published I may be ignoring it, but if that's the case that's because it is irrelevant. There is no reason to not publish your "failing" studies. That's literally why the public internet was created (the original private internet was created for military, but next in line was university adoption — to be used for exactly that purpose!). > We fundamentally disagree here and I'm not willing to put in the effort needed to try to convince you otherwise Makes sense. There is nothing to convince me of. I was never not convinced. But it remains: Who wants to put in the effort? Unless you are going to start putting guns to people's backs, what are you expecting?

by jraph 1y ago

9rx 1y ago

> They are written to share knowledge, new discoveries. We hope that they are replicated. If they aren't replicated, no new knowledge is gained. Not really. > it's a failure of the publishing system. What failure? Again, the "publishing system" is to submit the "best of the best" research to the world in order to invite replication. While nothing is perfect, we don't want people to have to sift through papers that have effectively no reason to be replicated in a quest to find something that is. That would make things infinitely worse. The internet was created for the minor-leagues. If you are willing to put in the effect to document your "less than the best" research, put it up on a website for all to see already. There is nothing stopping you. But who wants to put in the effort? > We are not judging anyone How do you define "shameful" if it isn't related to judgement of someone?

jraph 1y ago
You assume that null results are not worth it and are not the best of science. You seem to be ignoring the issue where researchers collectively waste time redoing "failing" studies again and again because the null results are not published, and other issues not publishing them causes. We fundamentally disagree here and I'm not willing to put in the effort needed to try to convince you otherwise, many other comments are here are better than what I could write. I do hope you take the time to take a step back (and you seem to be in defensive mode, if so, you need to go out of this) and reconsider. > How do you define "shameful" if it isn't related to judgement of someone? Do you know the expression "it's a shame"? > used when you wish a situation was different, and you feel sad or disappointed https://www.ldoceonline.com/dictionary/it-s-a-shame-what-a-shame-etc https://www.ldoceonline.com/dictionary/it-s-a-shame-what-a-s...
- 9rx 1y ago
  > You assume that null results are not worth it and are not the best of science. No. I make no such assumption. That may often be true, but there is nothing to stop a useful null result from being published. We also get things wrong from time to time. It is very possible that the best baseball player in the world has been overlooked by Major League Baseball. We're pretty good at scoping out the best, but nothing in life is perfect. If the best baseball player in the world ends up in the minor leagues instead, oh well? Does it really matter? You can still watch them there. Same goes for research. If something great doesn't make it into the formal publication system, you can still read it on the studier's website (if they put in the effort to publish it). > You seem to be ignoring the issue where researchers collectively waste time redoing "failing" studies again and again because the null results are not published I may be ignoring it, but if that's the case that's because it is irrelevant. There is no reason to not publish your "failing" studies. That's literally why the public internet was created (the original private internet was created for military, but next in line was university adoption — to be used for exactly that purpose!). > We fundamentally disagree here and I'm not willing to put in the effort needed to try to convince you otherwise Makes sense. There is nothing to convince me of. I was never not convinced. But it remains: Who wants to put in the effort? Unless you are going to start putting guns to people's backs, what are you expecting?