5 ms·

Yeah even the entire "Jane Doe / Jame Smith" my first thought is that it could have been a latex default value There was dumb stuff like this before the GPT er

Yeah even the entire "Jane Doe / Jame Smith" my first thought is that it could have been a latex default value There was dumb stuff like this before the GPT era, it's far from convincing

ls612 9mo ago
There are people who just want to punish academics for the sake of punishing academics. Look at all the people downthread salivating over blacklisting or even criminally charging people who make errors like this with felony fraud. Its the perfect brew of anti AI and anti academia sentiment. Also, in my field (economics), by far the biggest source of finding old papers invalid (or less valid, most papers state multiple results) is good old fashioned coding bugs. I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time.
- worik 9mo ago
  > I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time. My hand is up. I do not believe in gaol, but I do agree with the sentiment.
  - ls612 9mo ago
    Let he who is without sin cast the first stone…
    - girvo 9mo ago
      If there were real consequences, we wouldn't be forced to churn out buggy nonsense by our employers. So we'd be able to take the time to do the right thing. Bug free software is possible, the world just says its not worth it today.
      - ls612 9mo ago
        >Bug free software is possible, ... Mr. Turing and his halting problem would like to politely disagree with this assertion.
        ted_dunning 9mo ago
        You misread the comment and DR Turing's paper. Getting all possible software correct is impossible, clearly. Getting all the software you release is more possible because you can choose not to release the software that it is too hard to prove correct. Not that the suggestion is practical or likely, but your assertion that it is impossible is incorrect.
        ls612 9mo ago
        If you want to be pedantic I’m pretty sure every single general purpose OS (and thus also the programs running under it) falls into the category of not provably correct so it’s a distinction without a difference in real life.
- miki123211 9mo ago
  And research codebases (in AI and otherwise) are usually of extremely bad quality. It's usually a bunch of extremely poorly-written scripts, with no indication which order to run them in, how inputs and outputs should flow between them, and which specific files the scripts were run on to calculate the statistics presented in the paper.
  - davidguetta 9mo ago
    Codebase can bé of high quality but still you have no idea how they got the paper result
nativeit 9mo ago
> Between 2020 and 2025, submissions to NeurIPS increased more than 220% from 9,467 to 21,575. In response, organizers have had to recruit ever greater numbers of reviewers, resulting in issues of oversight, expertise alignment, negligence, and even fraud. I don’t think the point being made is “errors didn’t happen pre-GPT”, rather the tasks of detecting errors have become increasingly difficult because of the associated effects of GPT.
- ctoth 9mo ago
  > rather the tasks of detecting errors have become increasingly difficult because of the associated effects of GPT. Did the increase to submissions to NeurIPS from 2020 to 2025 happen because ChatGPT came out in November of 2022? Or was AI getting hotter and hotter during this period, thereby naturally increasing submissions to ... an AI conference?
  - amitav1 9mo ago
    I guess the way one would verify that this is more general trend in academia would be to run this on accepted papers to a non-AI conference?
  - mturmon 9mo ago
    I was an area chair on the NeurIPS program committee in 1997. I just looked and it seems that we had 1280 submissions. At that time, we were ultimately capped by the book size that MIT Press was willing to put out - 150 8-page articles. Back in 1997 we were all pretty sure we were on to something big. I'm sure people made mistakes on their bibliographies at that time as well! And did we all really dig up and read Metropolis, Rosenbluth, Rosenbluth, Teller, and Teller (1953)? Edited to add: Someone made a chart! Here: https://papercopilot.com/statistics/neurips-statistics/ https://papercopilot.com/statistics/neurips-statistics/ You can see the big bump after the book-length restriction was lifted, and the exponential rise starting ~2016.
    - dekhn 9mo ago
      I cited Watson and Crick '53 in my PhD thesis and I did go dig it up and read it. I had to go to the basement of the library, use some sort of weird rotating knob to move a heavy stack of journals over, find some large bound book of the year's journals, and navigate to the paper. When I got the page, it had been cut out by somebody previous and replaced with a photocopied verison. (I also invested a HUGE amount of my time into my bibliography in every paper I've written as first author, curating a database and writing scripts to format in the various journal formats. This involved multiple independent checks from several sources, repeated several times.
bjourne 9mo ago
Still a citation to a work you clearly have not read...

Yeah even the entire "Jane Doe / Jame Smith" my first thought is that it could have been a latex default value There was dumb stuff like this before the GPT era, it's far from convincing

ls612 9mo ago

There are people who just want to punish academics for the sake of punishing academics. Look at all the people downthread salivating over blacklisting or even criminally charging people who make errors like this with felony fraud. Its the perfect brew of anti AI and anti academia sentiment. Also, in my field (economics), by far the biggest source of finding old papers invalid (or less valid, most papers state multiple results) is good old fashioned coding bugs. I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time.

worik 9mo ago
> I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time. My hand is up. I do not believe in gaol, but I do agree with the sentiment.
- ls612 9mo ago
  Let he who is without sin cast the first stone…
  - girvo 9mo ago
    If there were real consequences, we wouldn't be forced to churn out buggy nonsense by our employers. So we'd be able to take the time to do the right thing. Bug free software is possible, the world just says its not worth it today.
    - ls612 9mo ago
      >Bug free software is possible, ... Mr. Turing and his halting problem would like to politely disagree with this assertion.
      - ted_dunning 9mo ago
        You misread the comment and DR Turing's paper. Getting all possible software correct is impossible, clearly. Getting all the software you release is more possible because you can choose not to release the software that it is too hard to prove correct. Not that the suggestion is practical or likely, but your assertion that it is impossible is incorrect.
        ls612 9mo ago
        If you want to be pedantic I’m pretty sure every single general purpose OS (and thus also the programs running under it) falls into the category of not provably correct so it’s a distinction without a difference in real life.
miki123211 9mo ago
And research codebases (in AI and otherwise) are usually of extremely bad quality. It's usually a bunch of extremely poorly-written scripts, with no indication which order to run them in, how inputs and outputs should flow between them, and which specific files the scripts were run on to calculate the statistics presented in the paper.
- davidguetta 9mo ago
  Codebase can bé of high quality but still you have no idea how they got the paper result

nativeit 9mo ago

> Between 2020 and 2025, submissions to NeurIPS increased more than 220% from 9,467 to 21,575. In response, organizers have had to recruit ever greater numbers of reviewers, resulting in issues of oversight, expertise alignment, negligence, and even fraud. I don’t think the point being made is “errors didn’t happen pre-GPT”, rather the tasks of detecting errors have become increasingly difficult because of the associated effects of GPT.

ctoth 9mo ago
> rather the tasks of detecting errors have become increasingly difficult because of the associated effects of GPT. Did the increase to submissions to NeurIPS from 2020 to 2025 happen because ChatGPT came out in November of 2022? Or was AI getting hotter and hotter during this period, thereby naturally increasing submissions to ... an AI conference?
- amitav1 9mo ago
  I guess the way one would verify that this is more general trend in academia would be to run this on accepted papers to a non-AI conference?
- mturmon 9mo ago
  I was an area chair on the NeurIPS program committee in 1997. I just looked and it seems that we had 1280 submissions. At that time, we were ultimately capped by the book size that MIT Press was willing to put out - 150 8-page articles. Back in 1997 we were all pretty sure we were on to something big. I'm sure people made mistakes on their bibliographies at that time as well! And did we all really dig up and read Metropolis, Rosenbluth, Rosenbluth, Teller, and Teller (1953)? Edited to add: Someone made a chart! Here: https://papercopilot.com/statistics/neurips-statistics/ https://papercopilot.com/statistics/neurips-statistics/ You can see the big bump after the book-length restriction was lifted, and the exponential rise starting ~2016.
  - dekhn 9mo ago
    I cited Watson and Crick '53 in my PhD thesis and I did go dig it up and read it. I had to go to the basement of the library, use some sort of weird rotating knob to move a heavy stack of journals over, find some large bound book of the year's journals, and navigate to the paper. When I got the page, it had been cut out by somebody previous and replaced with a photocopied verison. (I also invested a HUGE amount of my time into my bibliography in every paper I've written as first author, curating a database and writing scripts to format in the various journal formats. This involved multiple independent checks from several sources, repeated several times.

bjourne 9mo ago

Still a citation to a work you clearly have not read...