7 ms·

Do you want issues of Nature and cell to be replication studies? As a reader even from within the field, im not interested in browsing through negative studies.

Do you want issues of Nature and cell to be replication studies? As a reader even from within the field, im not interested in browsing through negative studies. It'll be great if I can look them up when needed but im not looking forward to email ToC alerts filled with them. Also who's funding you for replication work? Do you know the pressure you have in tenure track to have a consistent thesis on what you work on? Literally every single know that designs academia is tuned to not incentivize what you complain about. Its not just journals being picky. Also the people committing fraud aren't ones who will say "gosh I will replicate things now!" Replicating work is far more difficult than a lot of original work.

zhdc1 7mo ago
> Do you want issues of Nature and cell to be replication studies? As a reader even from within the field, im not interested in browsing through negative studies. Actually, yes, I do. The marginal cost for publishing a study online at this point is essentially nil.
- recursivecaveat 7mo ago
  I think archives with pretty low standards for notability are a good idea. At some point though you have to pick what actually counts as interesting enough to go in the curated list that is actually suggested reading, where the prestige is attached. If there's no curation by Nature then it falls to bloggers or another journal to sift through the fire-hose and make best-of lists. Most of the value is in the curation, not the publishing. Without exclusivity there's very little signal.
- mmooss 7mo ago
  > The marginal cost for publishing a study online at this point is essentially nil. The marginal cost for doing a study remains the same, which is quite a bit. Society doesn't have unlimited scientific talent or hours. Every year someone spends replicating is a year lost to creating something new and valuable.
Bratmon 7mo ago
> Do you want issues of Nature and cell to be replication studies? More than anything. That might legitimately be enough to save science on its own.
- dekhn 7mo ago
  Maybe nature and cell and a few other journals should be exceptions: they should be the place that the most advanced scientists publish interesting ideas early for the consumption by their competitors. At that level of science, all the competitors can reproduce each other's experiments if necessary; the real value is expanding the knowledge of what seems possible quickly. (I am not seriously proposing this, but it's interesting to think about distinguishing between the very small amount of truly innovative discovery versus the very long tail of more routine methods development and filling out gaps in knowledge)
  - Bratmon 7mo ago
    > that level of science, all the competitors can reproduce each other's experiments if necessary But they don't, and that's the problem!
    - dekhn 7mo ago
      Advanced groups usually replicate their competitor's results in their own hands shortly after publication (or they just trust their competitor's competence). But they don't spend any time publishing it unless they fail to replicate and can explain why they can't replicate. From their perspective, it's a waste of time. I think this has been shown to be a naive approach (given the high rate of image fraud in molecular biology) but people who are in the top of the field have strong incentives to focus on moving the state of the art forward without expending energy on improving the field as a whole.
      - Bratmon 7mo ago
        All that makes it more important for top journals to reward replication, not less!
benterix 7mo ago
> Do you want issues of Nature and cell to be replication studies? Of course I do! Not all of course, and taking (subjectively measured) impact into account. "We tried to replicate the study published in the same journal 3 years ago using a larger sample size and failed to achieve similar results..." OR "after successfully replicating the study we can confirm the therapeutic mechanism proposed by X actually works" - these are extremely important results that are takin into account in meta studies and e.g. form the base of policies worldwide.
- smj-edison 7mo ago
  Honestly even if they didn't publish the whole paper, if there was just a page that was a table of all the replication studies that were done recently, that would be pretty cool.
paganel 7mo ago
>Also who's funding you for replication work? Do you know the pressure you have in tenure track to have a consistent thesis on what you work on? This is partly why much of today's science is bs, pure and simple.
notRobot 7mo ago
"Original research" isn't worth much unless replicated, which is the entire problem being discussed in this thread. Replicating studies are great though because they tell you if the original research actually stands and is valid. > Replicating work is far more difficult than a lot of original work. Only if the original work was BS. And what, just because it's harder, we shouldn't do it?
- ramraj07 7mo ago
  Why blame just the journals when every other system also disintivizes the same.
  - anonymars 7mo ago
    I must be missing something, surely the argument isn't "other systems also disincentivize solving the problem, therefore we shouldn't work to fix this one"
renewiltord 7mo ago
Realistically, everyone will say “yes” to the “do you want” question because if you’re not a reader or a subscriber you benefit from the readers reading replication studies. I believe people will enthusiastically say yes but that they do not routinely read that journal.
- andrewprock 7mo ago
  Suggesting that people would stop reading Nature if they also included replication studies send like an incredible leap.
  - renewiltord 7mo ago
    That is a novel interpretation of my comment certainly.
  - fc417fc802 7mo ago
    It would directly undermine the reason that people read Nature in the first place.
    - MarkusQ 7mo ago
      Not really. "It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so." Knowing that something I thought was true was actually false would have saved me years in several situations.
      - fc417fc802 7mo ago
        I didn't understand us to only be talking about failed replication studies of previous Nature papers which would hopefully be few and far between and thus noteworthy indeed. Rather replication studies in general which on average are arguably less interesting to the reader than even the content of the typical archival journal.
        7mo ago
chocochunks 7mo ago
Even if that negative study could save you one, two, three+ years of work for the same outcome (which you then also can't really do anything with)? Shouldn't there BE funding for replication studies? Shouldn't that count towards tenure? Part of the problem is that publications play such a heavy role in getting tenure in the first place. I'm sure you can more narrowly tune your email alerts FFS.
junofan 7mo ago
> Do you want issues of Nature and cell to be replication studies? Hell yeah. We’re all trying to get that Nature paper. Imagine if you could accomplish that by setting the record straight.
- fc417fc802 7mo ago
  If you're thoroughly debunking a previous Nature paper they just might publish that. But the expectation is that you'll succeed. Publishing that sort of mundane article would reduce the prestige of getting something into the journal. Publishing in a high impact journal is only seen as an achievement in the first place because of what it implies about the content of your paper.
  - sieabahlpark 7mo ago
    [dead]
xandrius 7mo ago
I know you got a ton of responses already but not caring about replicability just invalidates science as a method. If we care only about first to publish we end up in the current situation where we don't even know that we know is actually even remotely correct. All because journals prefer novelty over confirmation. It's like a castle of cards, looks cool but not stable or long-term at all.
lovich 7mo ago
> Replicating work is far more difficult than a lot of original work. I don’t regularly read scientific studies but I’ve read a few of them. How is it possible that a serious study is harder to replicate than it is to do originally. Are papers no longer including their process? Are we at the point where they are just saying “trust me bro” for how they achieved their results? > Do you want issues of Nature and cell to be replication studies? Not issues of Nature but I’ve long thought that universities or the government should fund a department of “I don’t believe you” entirely focused on reproducing scientific results and seeing if they are real
- fc417fc802 7mo ago
  > How is it possible that a serious study is harder to replicate than it is to do originally. They aren't. GP was on point until that last sentence. Just pretend that wasn't there. It's pretty much always much easier to do something when all the key details have been figured out for you in advance. There is some difficulty if something doesn't work to distinguish user error from ambiguity of original publication from outright fraud. That can be daunting. But the vast majority of the time it isn't fraud and simply emailing the original author will get you on track. Most authors are overjoyed to learn about someone using their work. If you want to be cynical about it, how else would you get your citation count up?
carlosjobim 7mo ago
If you're a reader within the field, then you are the one person in the world who should be most interested in negative replication studies.
Qem 7mo ago
> Do you want issues of Nature and cell to be replication studies? There are journals dedicated to replication, like ReScience C[1]. But they are niche. Perhaps we should have more of these. [1] http://rescience.github.io/ http://rescience.github.io/

zhdc1 7mo ago

> Do you want issues of Nature and cell to be replication studies? As a reader even from within the field, im not interested in browsing through negative studies. Actually, yes, I do. The marginal cost for publishing a study online at this point is essentially nil.

recursivecaveat 7mo ago
I think archives with pretty low standards for notability are a good idea. At some point though you have to pick what actually counts as interesting enough to go in the curated list that is actually suggested reading, where the prestige is attached. If there's no curation by Nature then it falls to bloggers or another journal to sift through the fire-hose and make best-of lists. Most of the value is in the curation, not the publishing. Without exclusivity there's very little signal.
mmooss 7mo ago
> The marginal cost for publishing a study online at this point is essentially nil. The marginal cost for doing a study remains the same, which is quite a bit. Society doesn't have unlimited scientific talent or hours. Every year someone spends replicating is a year lost to creating something new and valuable.

Bratmon 7mo ago

> Do you want issues of Nature and cell to be replication studies? More than anything. That might legitimately be enough to save science on its own.

dekhn 7mo ago
Maybe nature and cell and a few other journals should be exceptions: they should be the place that the most advanced scientists publish interesting ideas early for the consumption by their competitors. At that level of science, all the competitors can reproduce each other's experiments if necessary; the real value is expanding the knowledge of what seems possible quickly. (I am not seriously proposing this, but it's interesting to think about distinguishing between the very small amount of truly innovative discovery versus the very long tail of more routine methods development and filling out gaps in knowledge)
- Bratmon 7mo ago
  > that level of science, all the competitors can reproduce each other's experiments if necessary But they don't, and that's the problem!
  - dekhn 7mo ago
    Advanced groups usually replicate their competitor's results in their own hands shortly after publication (or they just trust their competitor's competence). But they don't spend any time publishing it unless they fail to replicate and can explain why they can't replicate. From their perspective, it's a waste of time. I think this has been shown to be a naive approach (given the high rate of image fraud in molecular biology) but people who are in the top of the field have strong incentives to focus on moving the state of the art forward without expending energy on improving the field as a whole.
    - Bratmon 7mo ago
      All that makes it more important for top journals to reward replication, not less!

benterix 7mo ago

> Do you want issues of Nature and cell to be replication studies? Of course I do! Not all of course, and taking (subjectively measured) impact into account. "We tried to replicate the study published in the same journal 3 years ago using a larger sample size and failed to achieve similar results..." OR "after successfully replicating the study we can confirm the therapeutic mechanism proposed by X actually works" - these are extremely important results that are takin into account in meta studies and e.g. form the base of policies worldwide.

smj-edison 7mo ago
Honestly even if they didn't publish the whole paper, if there was just a page that was a table of all the replication studies that were done recently, that would be pretty cool.

paganel 7mo ago

>Also who's funding you for replication work? Do you know the pressure you have in tenure track to have a consistent thesis on what you work on? This is partly why much of today's science is bs, pure and simple.

notRobot 7mo ago

"Original research" isn't worth much unless replicated, which is the entire problem being discussed in this thread. Replicating studies are great though because they tell you if the original research actually stands and is valid. > Replicating work is far more difficult than a lot of original work. Only if the original work was BS. And what, just because it's harder, we shouldn't do it?

ramraj07 7mo ago
Why blame just the journals when every other system also disintivizes the same.
- anonymars 7mo ago
  I must be missing something, surely the argument isn't "other systems also disincentivize solving the problem, therefore we shouldn't work to fix this one"

renewiltord 7mo ago

Realistically, everyone will say “yes” to the “do you want” question because if you’re not a reader or a subscriber you benefit from the readers reading replication studies. I believe people will enthusiastically say yes but that they do not routinely read that journal.

andrewprock 7mo ago
Suggesting that people would stop reading Nature if they also included replication studies send like an incredible leap.
- renewiltord 7mo ago
  That is a novel interpretation of my comment certainly.
- fc417fc802 7mo ago
  It would directly undermine the reason that people read Nature in the first place.
  - MarkusQ 7mo ago
    Not really. "It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so." Knowing that something I thought was true was actually false would have saved me years in several situations.
    - fc417fc802 7mo ago
      I didn't understand us to only be talking about failed replication studies of previous Nature papers which would hopefully be few and far between and thus noteworthy indeed. Rather replication studies in general which on average are arguably less interesting to the reader than even the content of the typical archival journal.
      - 7mo ago

chocochunks 7mo ago

Even if that negative study could save you one, two, three+ years of work for the same outcome (which you then also can't really do anything with)? Shouldn't there BE funding for replication studies? Shouldn't that count towards tenure? Part of the problem is that publications play such a heavy role in getting tenure in the first place. I'm sure you can more narrowly tune your email alerts FFS.

junofan 7mo ago

> Do you want issues of Nature and cell to be replication studies? Hell yeah. We’re all trying to get that Nature paper. Imagine if you could accomplish that by setting the record straight.

fc417fc802 7mo ago
If you're thoroughly debunking a previous Nature paper they just might publish that. But the expectation is that you'll succeed. Publishing that sort of mundane article would reduce the prestige of getting something into the journal. Publishing in a high impact journal is only seen as an achievement in the first place because of what it implies about the content of your paper.
- sieabahlpark 7mo ago
  [dead]

xandrius 7mo ago

I know you got a ton of responses already but not caring about replicability just invalidates science as a method. If we care only about first to publish we end up in the current situation where we don't even know that we know is actually even remotely correct. All because journals prefer novelty over confirmation. It's like a castle of cards, looks cool but not stable or long-term at all.

lovich 7mo ago

> Replicating work is far more difficult than a lot of original work. I don’t regularly read scientific studies but I’ve read a few of them. How is it possible that a serious study is harder to replicate than it is to do originally. Are papers no longer including their process? Are we at the point where they are just saying “trust me bro” for how they achieved their results? > Do you want issues of Nature and cell to be replication studies? Not issues of Nature but I’ve long thought that universities or the government should fund a department of “I don’t believe you” entirely focused on reproducing scientific results and seeing if they are real

fc417fc802 7mo ago
> How is it possible that a serious study is harder to replicate than it is to do originally. They aren't. GP was on point until that last sentence. Just pretend that wasn't there. It's pretty much always much easier to do something when all the key details have been figured out for you in advance. There is some difficulty if something doesn't work to distinguish user error from ambiguity of original publication from outright fraud. That can be daunting. But the vast majority of the time it isn't fraud and simply emailing the original author will get you on track. Most authors are overjoyed to learn about someone using their work. If you want to be cynical about it, how else would you get your citation count up?

carlosjobim 7mo ago

If you're a reader within the field, then you are the one person in the world who should be most interested in negative replication studies.

Qem 7mo ago

> Do you want issues of Nature and cell to be replication studies? There are journals dedicated to replication, like ReScience C[1]. But they are niche. Perhaps we should have more of these. [1] http://rescience.github.io/ http://rescience.github.io/