3 ms·

Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimen

Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimental design. Yes, there are bad studies that get through now and then, but by and large you are much more likely to find good methods in a Nature paper compared to a low tier journal. As a graduate student, I rarely trusted results from low tier journals because those are the sloppiest and least likely to be replicated. The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published. The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish. All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error...

kerkeslager 6y ago
> Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimental design. Yes, there are bad studies that get through now and then, but by and large you are much more likely to find good methods in a Nature paper compared to a low tier journal. As a graduate student, I rarely trusted results from low tier journals because those are the sloppiest and least likely to be replicated. The problem we're trying to solve isn't poor experimental design, it's a failure to publish "boring" results--a bias against uninteresting results which should cause us to distrust any interesting result. > The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published. An inconclusive result does provide insight. If you have 1 study showing vaccines cause autism and 37651 studies which are inconclusive, that starts to be pretty conclusive. If the results produced are "utterly garbage", then that really shows us that the "beautiful experimental designs" weren't actually effective. > The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish. You publish the thing you agreed to publish, mention the interesting phenomena in the analysis, and apply to study it. > All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. A journal of Null Results already exists, and doesn't solve the problem because it's just reversing the publication bias. > However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error... I'm pulling this out because it is worth addressing, as it fundamentally misunderstands the problem. An experiment that gets a null result is not a failure. An experiment which fails to observe the variable under test is a failure. If science is working correctly, I would expect the majority of correctly-performed experiments to produce null results, because the obvious phenomena have already been discovered. The idea that "interesting result = success and null result = failure" is fundamentally unscientific and needs to be stricken from the human consciousness. And, the way this bias plays out in practice is that you're sitting here wondering if an experiment produced a null result because of researcher mistake or error, but not applying the same skepticism to "interesting" results. This is the opposite of a rational position: as far as I know, there isn't a replication crisis for null results. It's the interesting results which are having a replication crisis. If you're suspicious that null results are the result of experimental error (as you should be) then you should be far more suspicious that interesting results are the result of experimental error.

kerkeslager 6y ago

> Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimental design. Yes, there are bad studies that get through now and then, but by and large you are much more likely to find good methods in a Nature paper compared to a low tier journal. As a graduate student, I rarely trusted results from low tier journals because those are the sloppiest and least likely to be replicated. The problem we're trying to solve isn't poor experimental design, it's a failure to publish "boring" results--a bias against uninteresting results which should cause us to distrust any interesting result. > The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published. An inconclusive result does provide insight. If you have 1 study showing vaccines cause autism and 37651 studies which are inconclusive, that starts to be pretty conclusive. If the results produced are "utterly garbage", then that really shows us that the "beautiful experimental designs" weren't actually effective. > The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish. You publish the thing you agreed to publish, mention the interesting phenomena in the analysis, and apply to study it. > All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. A journal of Null Results already exists, and doesn't solve the problem because it's just reversing the publication bias. > However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error... I'm pulling this out because it is worth addressing, as it fundamentally misunderstands the problem. An experiment that gets a null result is not a failure. An experiment which fails to observe the variable under test is a failure. If science is working correctly, I would expect the majority of correctly-performed experiments to produce null results, because the obvious phenomena have already been discovered. The idea that "interesting result = success and null result = failure" is fundamentally unscientific and needs to be stricken from the human consciousness. And, the way this bias plays out in practice is that you're sitting here wondering if an experiment produced a null result because of researcher mistake or error, but not applying the same skepticism to "interesting" results. This is the opposite of a rational position: as far as I know, there isn't a replication crisis for null results. It's the interesting results which are having a replication crisis. If you're suspicious that null results are the result of experimental error (as you should be) then you should be far more suspicious that interesting results are the result of experimental error.