6 ms·

One big problem is that journals agree to publish studies after they are completed, which means that publication in prestigious journals is based on the novelty

One big problem is that journals agree to publish studies after they are completed, which means that publication in prestigious journals is based on the novelty of the result rather than on the validity of the study/experiment. It's an absolute fundamental of science that you go into it with an open mind, admitting that you do not know what the result will be. A good study/experiment is not one which produces an interesting result, it's one which is properly designed to answer the question it's asking. Evaluating the quality of studies/experiments based on their results is an anti-scientific practice which should be excised like the cancerous tumor it is. The best way to do that is to accept and commit to publish studies/experiments based solely on their design, before the experiment/study has actually been performed.

ylem 6y ago
This is actually being done in some journals--with registration of studies. I have arguments at times with people about Scientific Reports--I think that as long as the results are technically correct, I think we really need to encourage people to publish "boring" results. I have done this as community service, but it takes a lot of time and effort--even more so if you are correcting a previous boring with result with another boring result.
- kerkeslager 6y ago
  I don't think "encouraging" is enough. It needs to be built into the fabric of how science is performed.
- Seabiscuit 6y ago
  Isn't this the idea behind Registered Reports? (https://www.cos.io/our-services/registered-reports https://www.cos.io/our-services/registered-reports / https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/registered-reports-a-step-change-in-scientific-publishing https://www.elsevier.com/reviewers-update/story/innovation-i...)
renewiltord 6y ago
Oh I like that. With hypothesis pre-registration, journals could commit based on the hypothesis (and perhaps methods) rather than on the result.
- kerkeslager 6y ago
  I think the methods would be the most important element of peer review and commitment to publish. If you have a good method, the hypothesis is almost irrelevant--it's important to understand what variable is under test, but predictions of what result will be found are somewhat arbitrary.
  - Beldin 6y ago
    I would love the idea (for data-driven science) of a peer review before execution of the work. Write up a one- or 2-pager, review methodology and accept/reject based on that. Once the work has been done, there would mostly be editorial comments. Of course, that ignores the analysis section, which is somewhat important. But still, a vast improvement over today's way, where you may end up spending a lot of time only to get a vague reject.
    - chordalkeyboard 6y ago
      This is similar to “triple blind” but the reviewers review methodology before they have access to the results, but after the work has been performed.
  - samatman 6y ago
    It's unclear to me precisely what you mean, and we may be agreeing here. But the 'variable under test' is definitely part of the hypothesis, and a study without a hypothesis is, not useless, but much less likely to produce a useful result. There's a relevant xkcd: https://xkcd.com/882/ https://xkcd.com/882/ If the (pre-registered) hypothesis was that green jelly beans cause acne, then this is at least an interesting result. If you run this experiment with no particular hypothesis, and then decide on that basis that green jelly beans cause acne, this is just a setup for a later failure to replicate. At best. At worst, no one bothers checking your results, the company stops selling green jelly beans due to bad publicity, and people who enjoy the green ones are deprived of them for no good reason.
bonoboTP 6y ago
> Evaluating the quality of studies/experiments That's part of the problem. Publications are seen as achievements. If you got accepted to a prestigious journal or conference, you can list this on your CV as an impressive "award"-like thing. A publication list is not just a list of "Look, this is the kind of stuff I've been working on, have a great read at it", but "Look, my research is so great it got accepted to all these fancy places!". Publications are therefore unfortunately not merely about sharing new info with the research community but an award show. Ideally a publication would be the start of the conversation: "this is what we found, this is the method we propose, what do you think of it, community? Will you pick it up?" The test is then whether the ideas get adopted. But that's harder to measure. Citations try to approximate it, but it's a very crude approximation. A citation, as such, may mean tons of different things: e.g. a) a deep critique (Negative impact) b) being cited as part of a long block of "these other works exist, too" (Low impact) c) another work substantially based on the deep ideas of the original paper (High impact), d) being listed in a table for comparison, ala "we beat this other method", but no other discussion of the original paper (Low impact), etc. If a publication was nothing more than a "hey, look, this is interesting", then I'd say, publishing mostly novel sexy results would be fine! After all, the surprising cases are those that teach us the most. However, as I said earlier, a paper is not only about "hey, this is interesting", but a "hey I want to advance my career", too. And in a twisted way of logic, I can agree that therefore we could put a band-aid over some of the problem by publishing (rewarding) systematic work with negative or boring results. But ultimately, this goes against the original purpose of papers, that is alerting the scientific community to potentially new information that we haven't known about before. ---- Ideally, to assess someone's scientific career, there would be at least one smart, attentive, impartial expert taking their time reading through the publications, taking notes, pondering, digesting it all, consulting other experts etc. However, this is too subjective. Quantitative metrics seem superficially more objective and therefore egalitarian. The original idea is probably that if we just based everything on subjective judgement of scientific importance instead of publication count, there would be even more networking and friendship-based quid-pro-quo back scratching. But everyone is overworked, and those who aren't, want to keep it that way. So nobody wants to put in the effort to actually interact with the deep content of research. It's too complicated and too opaque. ---- The problem is, flashy results are by their nature more attention-grabbing on all levels. It's not just some small perverse incentive. This is how all of us work, this is how history works, how everything works. The winner takes all, the rich get richer etc. We remember the Einsteins of history, those who just worked systematically and didn't find much aren't heroes. And if that's our bar, then people will do everything to look like they clear it. In any system, scientists would have to hype up their impact, it doesn't matter who makes the decisions. Currently, universities want to employ researchers who will make a visible impact. Because that means attracting funding, but also attracting bright people from all around. Career-conscious researchers want to go to universities that help them market themselves well (good PR departments etc). PR is not only for laypeople as the audience, there is such a flood of research nowadays that even the experts of a small niche cannot keep up with everything happening. ---- The root of it is human nature, competition, deception, cliques, hierarchies. But the new about it is the scale of it, and the accompanying mechanization of it all. The idea that you can mass-manufacture innovation. That you can expect thousands upon thousands of researchers to make regular breakthroughs and, to say my field as an example, publish tens of thousands of novel AI-related ideas every year. It's related to credential inflation, and fake signaling: people with good academic track records got the good jobs and the respect, so people try to emulate that. Everyone tries to become the 1%, the rock star. And everyone wants to hire the 1%. So just like an evolutionary pressure, people try to appear like the successful. Soon enough the old signal doesn't work anymore. It used to be a high mark of educational level to have passed high school. Today that's a bare minimum. College used to be a meaningful differentiator. Now more than half of young people are "college educated" in developed countries. The next step is about becoming "researchers". Nowadays, having some publications is not a big differentiator. We see this also in title inflation like monkeying around in Excel is "data science" and "AI". It's not just a monkey's paw. There is no central figure orchestrating it, asking the monkey's paw for more papers. It's a distributed system of agents acting in their self-interest. Nobody wants papers for papers' sake, they want to make defensible, justifiable decisions that will not get them fired and will pass satisfaction up opposite the path where the money is flowing, all the way to the CEOs, politicians and taxpayers.
kashyapc 6y ago
I like what you're saying. Do you (or anyone on this thread) have any comments/thoughts on the following? During a conversation with an academic researcher (non-Computer Science) friend, when I brought up the topic of data sharing, especially in context of the infamous "replication crisis", they have their reasons not sharing. I'm loosely paraphrasing here, while trying hard not to misrepresent/misremember their exact views: "I want to protect my data; I don't have enough time to present my data in a presentable form; and more importantly, they'll just steal my idea and go present it as theirs—and I might lose funding" ... and so on. I can empathize with the academic pressure of "publish or perish". And not least of all, "need some food on the table, and roof over my head". But I still wonder, there must be other effective ways to gently persuade a said researcher (especially in the 'soft sciences'—I'm not using the term derogatorily) on the importance of sharing data that allows reproducibility of a given experiment?
- theptip 6y ago
  One angle that I think is worth exploring is the funding bodies putting requirements in place around publishing data (and the quality of that data) as a condition around funding. If NIST required that you publish the data (say within N years to cover the concern about getting scooped on follow-up papers), and dinged you on future funding applications if you didn't meet their quality/reproducibility metrics, perhaps that would help to align incentives. This is the same sort of idea as requiring research from public funding to be put in open-access journals so that the public can benefit from it.
  - kashyapc 6y ago
    Both your points sound promising to me: requirements by funding bodies and time limit for publishing data. But since I'm an academic layman, I can't punch any valid holes in the idea. :-) Regardless, I hope this idea gets discussed more by the real stake holders. Perhaps you might to do a public write-up of the idea.
- kerkeslager 6y ago
  To be honest, I don't think this is going to be solved from the bottom up. I think a lot of scientists know that they are making compromises between doing science and pursuing their career. But we can't reasonably ask people to do better science when better science means living on an adjunct salary for the rest of their life. The change has to come from publishers. The replication crisis will continue until publishers incentivize replication.
jartelt 6y ago
Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimental design. Yes, there are bad studies that get through now and then, but by and large you are much more likely to find good methods in a Nature paper compared to a low tier journal. As a graduate student, I rarely trusted results from low tier journals because those are the sloppiest and least likely to be replicated. The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published. The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish. All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error...
- kerkeslager 6y ago
  > Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimental design. Yes, there are bad studies that get through now and then, but by and large you are much more likely to find good methods in a Nature paper compared to a low tier journal. As a graduate student, I rarely trusted results from low tier journals because those are the sloppiest and least likely to be replicated. The problem we're trying to solve isn't poor experimental design, it's a failure to publish "boring" results--a bias against uninteresting results which should cause us to distrust any interesting result. > The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published. An inconclusive result does provide insight. If you have 1 study showing vaccines cause autism and 37651 studies which are inconclusive, that starts to be pretty conclusive. If the results produced are "utterly garbage", then that really shows us that the "beautiful experimental designs" weren't actually effective. > The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish. You publish the thing you agreed to publish, mention the interesting phenomena in the analysis, and apply to study it. > All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. A journal of Null Results already exists, and doesn't solve the problem because it's just reversing the publication bias. > However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error... I'm pulling this out because it is worth addressing, as it fundamentally misunderstands the problem. An experiment that gets a null result is not a failure. An experiment which fails to observe the variable under test is a failure. If science is working correctly, I would expect the majority of correctly-performed experiments to produce null results, because the obvious phenomena have already been discovered. The idea that "interesting result = success and null result = failure" is fundamentally unscientific and needs to be stricken from the human consciousness. And, the way this bias plays out in practice is that you're sitting here wondering if an experiment produced a null result because of researcher mistake or error, but not applying the same skepticism to "interesting" results. This is the opposite of a rational position: as far as I know, there isn't a replication crisis for null results. It's the interesting results which are having a replication crisis. If you're suspicious that null results are the result of experimental error (as you should be) then you should be far more suspicious that interesting results are the result of experimental error.
j7ake 6y ago
This pre-registration strategy may work for very applied, translational, or clinical studies but it does not make sense to do this for basic science, where you are constantly refining and redefining the scientific question. The reality is a lot of time spent by scientists is actually defining the problem properly (even after you think you’ve defined the problem properly), and preregistration studies would hinder the flexibility to adapt and change the direction given new understanding.

ylem 6y ago

This is actually being done in some journals--with registration of studies. I have arguments at times with people about Scientific Reports--I think that as long as the results are technically correct, I think we really need to encourage people to publish "boring" results. I have done this as community service, but it takes a lot of time and effort--even more so if you are correcting a previous boring with result with another boring result.

kerkeslager 6y ago
I don't think "encouraging" is enough. It needs to be built into the fabric of how science is performed.
Seabiscuit 6y ago
Isn't this the idea behind Registered Reports? (https://www.cos.io/our-services/registered-reports https://www.cos.io/our-services/registered-reports / https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/registered-reports-a-step-change-in-scientific-publishing https://www.elsevier.com/reviewers-update/story/innovation-i...)

renewiltord 6y ago

Oh I like that. With hypothesis pre-registration, journals could commit based on the hypothesis (and perhaps methods) rather than on the result.

kerkeslager 6y ago
I think the methods would be the most important element of peer review and commitment to publish. If you have a good method, the hypothesis is almost irrelevant--it's important to understand what variable is under test, but predictions of what result will be found are somewhat arbitrary.
- Beldin 6y ago
  I would love the idea (for data-driven science) of a peer review before execution of the work. Write up a one- or 2-pager, review methodology and accept/reject based on that. Once the work has been done, there would mostly be editorial comments. Of course, that ignores the analysis section, which is somewhat important. But still, a vast improvement over today's way, where you may end up spending a lot of time only to get a vague reject.
  - chordalkeyboard 6y ago
    This is similar to “triple blind” but the reviewers review methodology before they have access to the results, but after the work has been performed.
- samatman 6y ago
  It's unclear to me precisely what you mean, and we may be agreeing here. But the 'variable under test' is definitely part of the hypothesis, and a study without a hypothesis is, not useless, but much less likely to produce a useful result. There's a relevant xkcd: https://xkcd.com/882/ https://xkcd.com/882/ If the (pre-registered) hypothesis was that green jelly beans cause acne, then this is at least an interesting result. If you run this experiment with no particular hypothesis, and then decide on that basis that green jelly beans cause acne, this is just a setup for a later failure to replicate. At best. At worst, no one bothers checking your results, the company stops selling green jelly beans due to bad publicity, and people who enjoy the green ones are deprived of them for no good reason.

bonoboTP 6y ago

> Evaluating the quality of studies/experiments That's part of the problem. Publications are seen as achievements. If you got accepted to a prestigious journal or conference, you can list this on your CV as an impressive "award"-like thing. A publication list is not just a list of "Look, this is the kind of stuff I've been working on, have a great read at it", but "Look, my research is so great it got accepted to all these fancy places!". Publications are therefore unfortunately not merely about sharing new info with the research community but an award show. Ideally a publication would be the start of the conversation: "this is what we found, this is the method we propose, what do you think of it, community? Will you pick it up?" The test is then whether the ideas get adopted. But that's harder to measure. Citations try to approximate it, but it's a very crude approximation. A citation, as such, may mean tons of different things: e.g. a) a deep critique (Negative impact) b) being cited as part of a long block of "these other works exist, too" (Low impact) c) another work substantially based on the deep ideas of the original paper (High impact), d) being listed in a table for comparison, ala "we beat this other method", but no other discussion of the original paper (Low impact), etc. If a publication was nothing more than a "hey, look, this is interesting", then I'd say, publishing mostly novel sexy results would be fine! After all, the surprising cases are those that teach us the most. However, as I said earlier, a paper is not only about "hey, this is interesting", but a "hey I want to advance my career", too. And in a twisted way of logic, I can agree that therefore we could put a band-aid over some of the problem by publishing (rewarding) systematic work with negative or boring results. But ultimately, this goes against the original purpose of papers, that is alerting the scientific community to potentially new information that we haven't known about before. ---- Ideally, to assess someone's scientific career, there would be at least one smart, attentive, impartial expert taking their time reading through the publications, taking notes, pondering, digesting it all, consulting other experts etc. However, this is too subjective. Quantitative metrics seem superficially more objective and therefore egalitarian. The original idea is probably that if we just based everything on subjective judgement of scientific importance instead of publication count, there would be even more networking and friendship-based quid-pro-quo back scratching. But everyone is overworked, and those who aren't, want to keep it that way. So nobody wants to put in the effort to actually interact with the deep content of research. It's too complicated and too opaque. ---- The problem is, flashy results are by their nature more attention-grabbing on all levels. It's not just some small perverse incentive. This is how all of us work, this is how history works, how everything works. The winner takes all, the rich get richer etc. We remember the Einsteins of history, those who just worked systematically and didn't find much aren't heroes. And if that's our bar, then people will do everything to look like they clear it. In any system, scientists would have to hype up their impact, it doesn't matter who makes the decisions. Currently, universities want to employ researchers who will make a visible impact. Because that means attracting funding, but also attracting bright people from all around. Career-conscious researchers want to go to universities that help them market themselves well (good PR departments etc). PR is not only for laypeople as the audience, there is such a flood of research nowadays that even the experts of a small niche cannot keep up with everything happening. ---- The root of it is human nature, competition, deception, cliques, hierarchies. But the new about it is the scale of it, and the accompanying mechanization of it all. The idea that you can mass-manufacture innovation. That you can expect thousands upon thousands of researchers to make regular breakthroughs and, to say my field as an example, publish tens of thousands of novel AI-related ideas every year. It's related to credential inflation, and fake signaling: people with good academic track records got the good jobs and the respect, so people try to emulate that. Everyone tries to become the 1%, the rock star. And everyone wants to hire the 1%. So just like an evolutionary pressure, people try to appear like the successful. Soon enough the old signal doesn't work anymore. It used to be a high mark of educational level to have passed high school. Today that's a bare minimum. College used to be a meaningful differentiator. Now more than half of young people are "college educated" in developed countries. The next step is about becoming "researchers". Nowadays, having some publications is not a big differentiator. We see this also in title inflation like monkeying around in Excel is "data science" and "AI". It's not just a monkey's paw. There is no central figure orchestrating it, asking the monkey's paw for more papers. It's a distributed system of agents acting in their self-interest. Nobody wants papers for papers' sake, they want to make defensible, justifiable decisions that will not get them fired and will pass satisfaction up opposite the path where the money is flowing, all the way to the CEOs, politicians and taxpayers.

kashyapc 6y ago

I like what you're saying. Do you (or anyone on this thread) have any comments/thoughts on the following? During a conversation with an academic researcher (non-Computer Science) friend, when I brought up the topic of data sharing, especially in context of the infamous "replication crisis", they have their reasons not sharing. I'm loosely paraphrasing here, while trying hard not to misrepresent/misremember their exact views: "I want to protect my data; I don't have enough time to present my data in a presentable form; and more importantly, they'll just steal my idea and go present it as theirs—and I might lose funding" ... and so on. I can empathize with the academic pressure of "publish or perish". And not least of all, "need some food on the table, and roof over my head". But I still wonder, there must be other effective ways to gently persuade a said researcher (especially in the 'soft sciences'—I'm not using the term derogatorily) on the importance of sharing data that allows reproducibility of a given experiment?

theptip 6y ago
One angle that I think is worth exploring is the funding bodies putting requirements in place around publishing data (and the quality of that data) as a condition around funding. If NIST required that you publish the data (say within N years to cover the concern about getting scooped on follow-up papers), and dinged you on future funding applications if you didn't meet their quality/reproducibility metrics, perhaps that would help to align incentives. This is the same sort of idea as requiring research from public funding to be put in open-access journals so that the public can benefit from it.
- kashyapc 6y ago
  Both your points sound promising to me: requirements by funding bodies and time limit for publishing data. But since I'm an academic layman, I can't punch any valid holes in the idea. :-) Regardless, I hope this idea gets discussed more by the real stake holders. Perhaps you might to do a public write-up of the idea.
kerkeslager 6y ago
To be honest, I don't think this is going to be solved from the bottom up. I think a lot of scientists know that they are making compromises between doing science and pursuing their career. But we can't reasonably ask people to do better science when better science means living on an adjunct salary for the rest of their life. The change has to come from publishers. The replication crisis will continue until publishers incentivize replication.

jartelt 6y ago

Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimental design. Yes, there are bad studies that get through now and then, but by and large you are much more likely to find good methods in a Nature paper compared to a low tier journal. As a graduate student, I rarely trusted results from low tier journals because those are the sloppiest and least likely to be replicated. The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published. The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish. All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error...

kerkeslager 6y ago
> Sorry, but I think you are not on the right track here. Top tier journals typically publish results that are both interesting/novel and that have good experimental design. Yes, there are bad studies that get through now and then, but by and large you are much more likely to find good methods in a Nature paper compared to a low tier journal. As a graduate student, I rarely trusted results from low tier journals because those are the sloppiest and least likely to be replicated. The problem we're trying to solve isn't poor experimental design, it's a failure to publish "boring" results--a bias against uninteresting results which should cause us to distrust any interesting result. > The idea of evaluating experiment design and committing to publish before seeing any data/results also strikes me as deeply flawed. Graduate students can sit around think up many deep questions and draw up tons of beautiful experimental designs. That is the easier part. The harder part is actually running the experiments properly and interpreting the findings correctly. If we are publishing experiment ideas, every graduate student would be submitting 100s of ideas to top tier journals. However, most of those experiments would end up producing inconclusive or utterly garbage results, which would provide little insight if published. An inconclusive result does provide insight. If you have 1 study showing vaccines cause autism and 37651 studies which are inconclusive, that starts to be pretty conclusive. If the results produced are "utterly garbage", then that really shows us that the "beautiful experimental designs" weren't actually effective. > The other issue I see is that many interesting findings come out of serendipitous results that are not related to the initial experiment or hypothesis. You start out trying to answer one question and then stumble upon some else that is very interesting (so you then explore that). If you have already submitted one experimental design and are supposed to publish on that, what do you do if you stumble upon something else that is actually much more interesting? In my experience with research you don't often go directly from one question/hypothesis/method to an answer and then publish. It's much less linear than that with many dead ends along the way. It's just not something you can pre-publish. You publish the thing you agreed to publish, mention the interesting phenomena in the analysis, and apply to study it. > All of that said, one area where I think we would agree is on publishing the results of well performed experiments with null results. I think it would be useful to the community if people published reports saying "we had this interesting question, did a proper experiment, and it turns out the hypothesis was null and there was nothing interesting there." In the current system, that will likely not get published. But, other researchers would benefit from seeing that result because they may have the same hypothesis and may waste time trying to explore the same dead end. A Journal of Null Results could be useful here. A journal of Null Results already exists, and doesn't solve the problem because it's just reversing the publication bias. > However, you do run into the issue of not knowing if an experiment failed because of a researcher mistake or error... I'm pulling this out because it is worth addressing, as it fundamentally misunderstands the problem. An experiment that gets a null result is not a failure. An experiment which fails to observe the variable under test is a failure. If science is working correctly, I would expect the majority of correctly-performed experiments to produce null results, because the obvious phenomena have already been discovered. The idea that "interesting result = success and null result = failure" is fundamentally unscientific and needs to be stricken from the human consciousness. And, the way this bias plays out in practice is that you're sitting here wondering if an experiment produced a null result because of researcher mistake or error, but not applying the same skepticism to "interesting" results. This is the opposite of a rational position: as far as I know, there isn't a replication crisis for null results. It's the interesting results which are having a replication crisis. If you're suspicious that null results are the result of experimental error (as you should be) then you should be far more suspicious that interesting results are the result of experimental error.

j7ake 6y ago

This pre-registration strategy may work for very applied, translational, or clinical studies but it does not make sense to do this for basic science, where you are constantly refining and redefining the scientific question. The reality is a lot of time spent by scientists is actually defining the problem properly (even after you think you’ve defined the problem properly), and preregistration studies would hinder the flexibility to adapt and change the direction given new understanding.