4 ms·

I think if the data is valid, open access, and reproducible, it shouldn't matter if the paper was written by AI.

tga_d 3y ago
The article says this: > Kahn says that, although there will undoubtedly be positive uses of AI to support researchers writing papers, it will still be necessary to distinguish between legitimate papers written with AI and those that have been completely fabricated.
mike_hearn 3y ago
Unfortunately this assumes that the only thing which can go wrong is lack of reproducibility. Not so. I read a lot of public health papers during COVID and a staggering quantity (IMHO nearly all of them) should not have been published; many of them would have been reproducible despite that. Other things that can and do regularly appear in reproducible, peer reviewed papers: • Nonsensical methodologies • Logical fallacies • Mis-representation of their data • Incorrectly implemented software • Source datasets that are cherry-picked One might think that things like incorrectly implemented software would fall under the umbrella of irreproducible research, but that won't work. Some fields don't really recognize a distinction between model implementation and specification. The model is the implementation and if a description was once published it's quite possibly either too vague to implement, out of date, wrong, or all three. IIRC Prof Neil Ferguson's team actually rejected an attempted replication of one of their epidemic models on the basis that only they were qualified to use it! Pseudo-science like this goes unremarked in universities, only outsiders seem to care. Sometimes you get an honest academic who knows what they're supposed to do and actually does it, but there's no observable benefit to them from doing so because the ones who don't bother don't seem to suffer any consequences. tl;dr The biggest problem with the replication crisis is the framing of it as being about replication. What the world actually faces is a misleading research crisis. You can drive the replicability rate to 100% and you'll still find whole fields consisting of logical fallacies and misinformation.
- RosanaAnaDana 3y ago
  I think you'll be demonstrated to be wrong in short order over a period of the next 1-2 years. My called shot: We'll have fully automated scientific research with useful and valuable conclusions and little to no human intervention. I think it will take some time, but can tell you right now its on its way to happening.

The article says this: > Kahn says that, although there will undoubtedly be positive uses of AI to support researchers writing papers, it will still be necessary to distinguish between legitimate papers written with AI and those that have been completely fabricated.

mike_hearn 3y ago

Unfortunately this assumes that the only thing which can go wrong is lack of reproducibility. Not so. I read a lot of public health papers during COVID and a staggering quantity (IMHO nearly all of them) should not have been published; many of them would have been reproducible despite that. Other things that can and do regularly appear in reproducible, peer reviewed papers: • Nonsensical methodologies • Logical fallacies • Mis-representation of their data • Incorrectly implemented software • Source datasets that are cherry-picked One might think that things like incorrectly implemented software would fall under the umbrella of irreproducible research, but that won't work. Some fields don't really recognize a distinction between model implementation and specification. The model is the implementation and if a description was once published it's quite possibly either too vague to implement, out of date, wrong, or all three. IIRC Prof Neil Ferguson's team actually rejected an attempted replication of one of their epidemic models on the basis that only they were qualified to use it! Pseudo-science like this goes unremarked in universities, only outsiders seem to care. Sometimes you get an honest academic who knows what they're supposed to do and actually does it, but there's no observable benefit to them from doing so because the ones who don't bother don't seem to suffer any consequences. tl;dr The biggest problem with the replication crisis is the framing of it as being about replication. What the world actually faces is a misleading research crisis. You can drive the replicability rate to 100% and you'll still find whole fields consisting of logical fallacies and misinformation.

RosanaAnaDana 3y ago
I think you'll be demonstrated to be wrong in short order over a period of the next 1-2 years. My called shot: We'll have fully automated scientific research with useful and valuable conclusions and little to no human intervention. I think it will take some time, but can tell you right now its on its way to happening.