6 ms·

Advanced groups usually replicate their competitor's results in their own hands shortly after publication (or they just trust their competitor's competence). B

by dekhn 7mo ago

Advanced groups usually replicate their competitor's results in their own hands shortly after publication (or they just trust their competitor's competence). But they don't spend any time publishing it unless they fail to replicate and can explain why they can't replicate. From their perspective, it's a waste of time. I think this has been shown to be a naive approach (given the high rate of image fraud in molecular biology) but people who are in the top of the field have strong incentives to focus on moving the state of the art forward without expending energy on improving the field as a whole.

Bratmon 7mo ago
All that makes it more important for top journals to reward replication, not less!
- jltsiren 7mo ago
  Top journals are not inherently prestigious. They are prestigious because they try to publish only the most interesting and most significant results. If they started publishing successful replication studies, they would lose prestige, and more interesting journals would eventually rise to the top. (Replication studies that fail to replicate a major result in a spectacular way are another matter.)
MarkusQ 7mo ago
"strong incentives to focus on moving the state of the art forward without expending energy on improving the field as a whole" That sort of Orwellian doublethink is exactly the problem. They need to move it forward without improving it, contribute without adding anything, challenge accepted dogma without rocking the boat, and...blech!
- godelski 7mo ago
  > challenge accepted dogma without rocking the boat I think the funniest part is how we have all these heroes of science who faced scrutiny by their peers, but triumphed in the end. They struggled because they challenged the status quo. We celebrate their anti authoritative nature. We congratulate them for their pursuit of truth! And then get mad when it happens. We pretend this is a thing of the past, but it's as common as ever[0,1]. You must create paradigm shifts without challenging the current paradigm! [0] https://www.scientificamerican.com/article/katalin-karikos-nobel-prize-winning-work-on-mrna-was-long-ignored-and-led-to/ https://www.scientificamerican.com/article/katalin-karikos-n... [1] https://www.globalperformanceinsights.com/post/how-a-rejected-paper-won-a-nobel-prize https://www.globalperformanceinsights.com/post/how-a-rejecte...
  - RobotToaster 7mo ago
    "Science is the belief in the ignorance of experts" - Richard Feynman
godelski 7mo ago
Are you explaining this from experience or from speculation? I can tell you that it doesn't match my own experience. I also think it doesn't match your example. Those cases of verified image fraud are typically part of replication efforts. The reason the fraud is able to persist is due to the lack of replication, not the abundance of it.
- dekhn 7mo ago
  Mostly experience (based on being a PhD scientist, a postdoc, a National Lab scientist, and engineer at several bigtech companies), partly speculation (none of the groups/labs I worked in operated at "the highest level", but I worked adjacent to many of those). I'm pretty sure most image fraud went completely unrealized even in the case of replication failure. It looks like (pre AI) it was mostly a few folks who did it as a hobby, unrelated to their regular jobs/replication work.
  - godelski 7mo ago
    In most of the labs I've worked in replication is not a common task[0] > 'm pretty sure most image fraud went completely unrealized even in the case of replication failure Part of my point is that being unable to publish replication efforts means we don't reduce ambiguity in the original experiments. I was taught that I should write a paper well enough that a PhD student (rather than candidate) should be able to reproduce the work. IME replication failures are often explained with "well I must be doing something wrong." A reasonable conclusion, but even if true the conclusion is that the original explanation was insufficiently clear. > It looks like (pre AI) it was mostly a few folks who did it as a hobby I'm sorry, didn't you say >>> Advanced groups usually replicate their competitor's results in their own hands shortly after publication Because your current statement seems to completely contradict your previous one. Or are you suggesting that the groups you didn't work with (and are thus speculating) are the ones who replicate works and the ones you did work with "just trust their competitor's competence")? Because if this is what you're saying then I do not think this "mostly" matches your experience. That your experience more closely matches my own. [0] I should take that back. I started in physics (undergrad) and went to CS for grad. Replication could often be de facto in physics, as it was a necessary step towards progress. You often couldn't improve an idea without understanding/replicating it (both theoretical and experimental). But my experience in CS, including at national labs, was that people didn't even run the code. Even when code was provided as part of reviewing artifacts I found that my fellow reviewers often didn't even look at it, let alone run it... This was common at tier 1 conferences mind you... I only knew one other person that consistently ran code.

by dekhn 7mo ago

Bratmon 7mo ago

All that makes it more important for top journals to reward replication, not less!

jltsiren 7mo ago
Top journals are not inherently prestigious. They are prestigious because they try to publish only the most interesting and most significant results. If they started publishing successful replication studies, they would lose prestige, and more interesting journals would eventually rise to the top. (Replication studies that fail to replicate a major result in a spectacular way are another matter.)

MarkusQ 7mo ago

"strong incentives to focus on moving the state of the art forward without expending energy on improving the field as a whole" That sort of Orwellian doublethink is exactly the problem. They need to move it forward without improving it, contribute without adding anything, challenge accepted dogma without rocking the boat, and...blech!

godelski 7mo ago
> challenge accepted dogma without rocking the boat I think the funniest part is how we have all these heroes of science who faced scrutiny by their peers, but triumphed in the end. They struggled because they challenged the status quo. We celebrate their anti authoritative nature. We congratulate them for their pursuit of truth! And then get mad when it happens. We pretend this is a thing of the past, but it's as common as ever[0,1]. You must create paradigm shifts without challenging the current paradigm! [0] https://www.scientificamerican.com/article/katalin-karikos-nobel-prize-winning-work-on-mrna-was-long-ignored-and-led-to/ https://www.scientificamerican.com/article/katalin-karikos-n... [1] https://www.globalperformanceinsights.com/post/how-a-rejected-paper-won-a-nobel-prize https://www.globalperformanceinsights.com/post/how-a-rejecte...
- RobotToaster 7mo ago
  "Science is the belief in the ignorance of experts" - Richard Feynman

godelski 7mo ago

Are you explaining this from experience or from speculation? I can tell you that it doesn't match my own experience. I also think it doesn't match your example. Those cases of verified image fraud are typically part of replication efforts. The reason the fraud is able to persist is due to the lack of replication, not the abundance of it.

dekhn 7mo ago
Mostly experience (based on being a PhD scientist, a postdoc, a National Lab scientist, and engineer at several bigtech companies), partly speculation (none of the groups/labs I worked in operated at "the highest level", but I worked adjacent to many of those). I'm pretty sure most image fraud went completely unrealized even in the case of replication failure. It looks like (pre AI) it was mostly a few folks who did it as a hobby, unrelated to their regular jobs/replication work.
- godelski 7mo ago
  In most of the labs I've worked in replication is not a common task[0] > 'm pretty sure most image fraud went completely unrealized even in the case of replication failure Part of my point is that being unable to publish replication efforts means we don't reduce ambiguity in the original experiments. I was taught that I should write a paper well enough that a PhD student (rather than candidate) should be able to reproduce the work. IME replication failures are often explained with "well I must be doing something wrong." A reasonable conclusion, but even if true the conclusion is that the original explanation was insufficiently clear. > It looks like (pre AI) it was mostly a few folks who did it as a hobby I'm sorry, didn't you say >>> Advanced groups usually replicate their competitor's results in their own hands shortly after publication Because your current statement seems to completely contradict your previous one. Or are you suggesting that the groups you didn't work with (and are thus speculating) are the ones who replicate works and the ones you did work with "just trust their competitor's competence")? Because if this is what you're saying then I do not think this "mostly" matches your experience. That your experience more closely matches my own. [0] I should take that back. I started in physics (undergrad) and went to CS for grad. Replication could often be de facto in physics, as it was a necessary step towards progress. You often couldn't improve an idea without understanding/replicating it (both theoretical and experimental). But my experience in CS, including at national labs, was that people didn't even run the code. Even when code was provided as part of reviewing artifacts I found that my fellow reviewers often didn't even look at it, let alone run it... This was common at tier 1 conferences mind you... I only knew one other person that consistently ran code.