4 ms·

It speaks to the broken state of measuring scientific and academic output. It's 99% a charade, people churning out machine generated papers nobody reads but get

It speaks to the broken state of measuring scientific and academic output. It's 99% a charade, people churning out machine generated papers nobody reads but get quoted in an attempt to justify the inflated claims of other machine generated garbage papers, in a circular attempt to fudge the stats that decide the funding and promotion. The idea that someone can even read, edit and make co-author level contributions in papers aimed for a top-level journals, all in the span of 37 hours, is utterly ridiculous. It's basically the equivalent of SEO spamming the Page rank algorithm with crummy pages with many back-links and repeated search terms. The search optimized term here is the author's name.

cwmoore 4y ago
Why maximize paperclips when terms like "vegetative electron microscopy" could be trending in the literature.
- DoctorOetker 4y ago
  For those who didn't read the article, or for those who did, but didn't follow its link to the PubPeer thread containing the "vegetative electron microscopy" observation: https://pubpeer.com/publications/7C9F0CCD493B1129135A3A918B0AAB# https://pubpeer.com/publications/7C9F0CCD493B1129135A3A918B0... Search the page for the term and you'll see the relevant comment. The term seems to stem from an OCR cluster-mishap, where a valid scientific document was once OCR'ed incorrectly. The page has 2 columns of text and the left column has a running sentence 'ending' with "vegetative" before continuing on the next line in the same column. But at the same height in the column on the right a sentence continues, visually starting with "electron microscopy". The term is entirely meaningless in the field. This isn't just "using a little ChatGPT to brush up my English" at all. Other comments point out that the contents of abstract and article don't even match, the abstract being about bacteria but the article contents about gene delivery... I don't need convincing that ML will have tremendous positive impact in math and science disciplines, it's obvious, but the way ML was used in this case is not a defensible usage.
  - roundandround 4y ago
    Yeah, from the article I get the sense that he is often agreeing to add his name to papers for network affect, perhaps even as the main author.. So his own criteria for what he writes may be factual, but seems a bit off topic for what a reader should expect on seeing his name.
- catchnear4321 4y ago
  To be free of this pain.
kevviiinn 4y ago
From my experience most researchers don't actually read most papers, we just don't have time. The figures are what tells the real story, not necessarily the interpretation of the authors.
- EpiMath 4y ago
  I don't disagree with you, but this is a horribly sad comment. I worry that the text of papers often gives short shrift to nuance and subtlety that is necessary for reasonable interpretation.
  - peterfirefly 4y ago
    I once read a lot of medical papers -- probably a few hundreds. There was often no real connection between the reported data and the conclusion. Sometimes the conclusion was along the lines of "a weak relation between X and Y was found" but the data showed a strong relation. Sometimes it was the opposite. Sometimes the conclusion was that X was good but the data showed X was bad. Sometimes it was the opposite. It was almost as if many of the authors couldn't do the kind of math expected of high schoolers that apply to university to study science. My takeaway was 1) to deeply distrust doctors as scientists (and as people who could think) and 2) mostly ignore the text surrounding the tables and graphs and just go straight to the data.
    - kevviiinn 4y ago
      Physicians are definitely not scientists, at least the majority of MDs. MD is not a research degree, it's a memorization degree
JTbane 4y ago
TBH I'm terrified of the double-whammy effect of researchers becoming less and less competent as AI takes the reins (a la Wall-E) and good science getting drowned out by crap.
queuebert 4y ago
Modern academia incentivizes quantity over quality, and reviewers for all but the most prestigious journals don't have sufficient time or expertise to properly review submitted papers. So this does not surprise me.
hanselot 4y ago
A couple of years ago I spoke to my near-phd biochem friend and discussed this problem with him at length. I have since come to believe that what science needs is a git of science. Aka, a scientific way to build science over from scratch and explore different approaches and research in a version control kind of way. A property of science is that existing research should be immutable, so you can see the natural progression over time, so for this I thought a blockchain seemed a good fit. Scientists do research and peer review each other in exchange for a token. Other scientists can then easily access and improve/build on existing science, with no journal incentives required to keep academics afloat. And yes I am aware safeguards will need to be put in place to prevent gaming the system, but using AI, it should be relatively easy to actually capture the entire history of science on the chain and verify the knowledge.

cwmoore 4y ago

Why maximize paperclips when terms like "vegetative electron microscopy" could be trending in the literature.

DoctorOetker 4y ago
For those who didn't read the article, or for those who did, but didn't follow its link to the PubPeer thread containing the "vegetative electron microscopy" observation: https://pubpeer.com/publications/7C9F0CCD493B1129135A3A918B0AAB# https://pubpeer.com/publications/7C9F0CCD493B1129135A3A918B0... Search the page for the term and you'll see the relevant comment. The term seems to stem from an OCR cluster-mishap, where a valid scientific document was once OCR'ed incorrectly. The page has 2 columns of text and the left column has a running sentence 'ending' with "vegetative" before continuing on the next line in the same column. But at the same height in the column on the right a sentence continues, visually starting with "electron microscopy". The term is entirely meaningless in the field. This isn't just "using a little ChatGPT to brush up my English" at all. Other comments point out that the contents of abstract and article don't even match, the abstract being about bacteria but the article contents about gene delivery... I don't need convincing that ML will have tremendous positive impact in math and science disciplines, it's obvious, but the way ML was used in this case is not a defensible usage.
- roundandround 4y ago
  Yeah, from the article I get the sense that he is often agreeing to add his name to papers for network affect, perhaps even as the main author.. So his own criteria for what he writes may be factual, but seems a bit off topic for what a reader should expect on seeing his name.
catchnear4321 4y ago
To be free of this pain.

kevviiinn 4y ago

From my experience most researchers don't actually read most papers, we just don't have time. The figures are what tells the real story, not necessarily the interpretation of the authors.

EpiMath 4y ago
I don't disagree with you, but this is a horribly sad comment. I worry that the text of papers often gives short shrift to nuance and subtlety that is necessary for reasonable interpretation.
- peterfirefly 4y ago
  I once read a lot of medical papers -- probably a few hundreds. There was often no real connection between the reported data and the conclusion. Sometimes the conclusion was along the lines of "a weak relation between X and Y was found" but the data showed a strong relation. Sometimes it was the opposite. Sometimes the conclusion was that X was good but the data showed X was bad. Sometimes it was the opposite. It was almost as if many of the authors couldn't do the kind of math expected of high schoolers that apply to university to study science. My takeaway was 1) to deeply distrust doctors as scientists (and as people who could think) and 2) mostly ignore the text surrounding the tables and graphs and just go straight to the data.
  - kevviiinn 4y ago
    Physicians are definitely not scientists, at least the majority of MDs. MD is not a research degree, it's a memorization degree

JTbane 4y ago

TBH I'm terrified of the double-whammy effect of researchers becoming less and less competent as AI takes the reins (a la Wall-E) and good science getting drowned out by crap.

queuebert 4y ago

Modern academia incentivizes quantity over quality, and reviewers for all but the most prestigious journals don't have sufficient time or expertise to properly review submitted papers. So this does not surprise me.

hanselot 4y ago

A couple of years ago I spoke to my near-phd biochem friend and discussed this problem with him at length. I have since come to believe that what science needs is a git of science. Aka, a scientific way to build science over from scratch and explore different approaches and research in a version control kind of way. A property of science is that existing research should be immutable, so you can see the natural progression over time, so for this I thought a blockchain seemed a good fit. Scientists do research and peer review each other in exchange for a token. Other scientists can then easily access and improve/build on existing science, with no journal incentives required to keep academics afloat. And yes I am aware safeguards will need to be put in place to prevent gaming the system, but using AI, it should be relatively easy to actually capture the entire history of science on the chain and verify the knowledge.