5 ms·

Asking Authors About Their Own Papers

4 points by stefanpie 15d ago

JasonCEC 12d ago
Peer review has its historical issues, but the landscape of science and science-publishing has changed. New problems of authorship and authorial-understanding are now challenged by LLMs writing (at least) good sounding papers - some of which might be of acceptable quality in subject (I am not against AI in the sciences; some of the math work has been great). On the other hand: I am against authors not understanding their own work. High repute journals may need to add "oral exams" to the paper acceptance process...
N_Lens 12d ago
I think authenticity and trust will command a (larger) premium in this new age of slop. The article highlights how only one out of ten paper’s authors were able to answer questions thoroughly and at a high level. This indicates an overwhelming percentage of authors are slopping up their work with AI and submitting it without even reading it.
- SoftTalker 12d ago
  No doubt this is happening, but I wonder how many authors of papers "slated for desk rejection" 10 years ago could answer questions about their papers? We'd need that comparison to understand if this is a new problem or if AI is just a new source of content that the authors of poorly-written papers are using.
  - brianpan 12d ago
    Evidence-based assertions are a good thing, but some things are so obvious that a "comparison" or "research" is not needed. This is already an obvious problem in so many places from high schoolers turning in assignments they don't understand, coders submitting code changes they don't understand, blog posts, and certainly to scientific papers.
    - SoftTalker 12d ago
      Yeah sadly you're probably right.
Calazon 12d ago
I wonder how the ratios would change for papers at different parts of the review process. For what fraction of published papers are the authors unable to answer basic questions about them?
jszymborski 12d ago
FYI in case the author is reading, https://www.cs.cmu.edu/~nihars/preprints/greCAPTCHA.pdf https://www.cs.cmu.edu/~nihars/preprints/greCAPTCHA.pdf is a dead link. EDIT: I found a live link on arxiv https://arxiv.org/html/2609.20481v1 https://arxiv.org/html/2609.20481v1
- encyclopediai 12d ago
  In June 2026 I proposed a CAPTCHA for scientific publications https://chorasimilarity.wordpress.com/2026/06/13/a-captcha-for-scientific-publications/ https://chorasimilarity.wordpress.com/2026/06/13/a-captcha-f... At the moment this was seen as a tongue in cheek proposal.
  - doc_ick 12d ago
    I’d agree it’d be a funny proposal, wouldn’t have worked back then but funny.
    - encyclopediai 12d ago
      Thanks. IMO the most fun is in the CAPTCHA, which turns on its head the Turing test. But it goes even further than their greCAPTCHA and it solves their consumed time problem. Indeed, in their proposal they have a human bottleneck, but in the june 2026 proposal is suggested that one could use an AI to generate the results without the knowledge of the submitted article. If the AI can generate a pretty close result, with the article fed gradually as a prompt, then reject. And even further, that it might be not even a need to publish anymore. Just use the article for training and make a public database with some numbers about the successful researcher, where we see an influence score (how many times an idea from an accepted article are used by other accepted articles), a publication score (how many articles the author had). I wonder if the reality will be more or less surprising, my bet is on "more".
      - totetsu 11d ago
        Sorry to be uncool and try and explain away the joke, but. I'm reading three parts to this.. one is the risk to the job of researches, by the impacts of the systems that they have typically used until now, by being flooded with texts that look like research articles, but are produced by llms.. another is a critique of that system in the first place, in the interaction of both for profit publishing, and research funding based on measuring of publishing in those for-profit journals.. and then there is also a concern about theft of work, and plagiarism, by AI providers, ingesting pre-published work, and training on it, and additionally those providers maybe acting as gate-keepers or .. imposing their own normative values or being opaque and not impartial in how they operate.. Did I parse that correctly?
greenflag 12d ago
One larger problem here is the value of a research paper is rarely the specific knowledge it adds but in the process of researching that adds to the collective knowledge+experience of those involved, especially training graduate students. AI papers shortcut this entirely. Academia has a lot to answer for this too by making papers the currency of success. AI generated papers are almost shortcut learning at a full system level.
- doc_ick 12d ago
  I disagree for academia using papers as an easy medium for verification and providing more knowledge. Llms always short circuit everything, so how would you fix academia?
- PantaloonFlames 12d ago
  Related: Jan 2026 https://www.theatlantic.com/science/2026/01/ai-slop-science-publishing/685704/ https://www.theatlantic.com/science/2026/01/ai-slop-science-... "For more than a century, scientific journals have been the pipes through which knowledge of the natural world flows into our culture. Now they’re being clogged with AI slop." Sept 2026 https://www.theatlantic.com/ideas/2026/09/college-education-future-ai/688655/ https://www.theatlantic.com/ideas/2026/09/college-education-... Academia, particularly the university system, is an untenable collection of interests. The triple stresses of COVID, AI, and funding withdrawal seem to presage what will be a significant disruption.
wackget 12d ago
I'm not familiar with the world of academic publishing, so I want to ask: how is the industry making sure that submissions aren't at least partially AI-generated? Is it standard practice for authors to have to defend their submissions via interview like this? If not, why not? Does the vetting process vary with the quality of the publisher? As an outsider, it's extremely worrying that anyone would even attempt to submit an AI-generated paper for publication in an academic journal. At that level I would have assumed literally everybody should know better than to even try.
- whattheheckheck 12d ago
  The rates of paper publishing show that everyone must be using ai now or the rates wouldn't have gone up
- lelanthran 12d ago
  > Is it standard practice for authors to have to defend their submissions via interview like this? If not, why not? It isn't, but maybe it should be. For post-grad qualifications oral defense is standard, and I didn't mind defending my central thesis then, and won't mind now.
  - doc_ick 12d ago
    Not in a direct interview style, but most us conferences can request additional information or feedback. If they conditionally accept or reject a paper, that conditional relies on feedback from the author(s). Interviews like this are interesting, but in no way can scale to the infinite paper slop conferences are facing.
    - lelanthran 12d ago
      > Not in a direct interview style, but most us conferences can request additional information or feedback. If they conditionally accept or reject a paper, that conditional relies on feedback from the author(s). Requesting feedback is useless, as the article points out - the "authors" could not answer basic questions during the interview, but after the interview were able to send full explanations to the interviewer. If you have indirect feedback ("please answer these questions we have") the "author" will simply feed it into an LLM and send the results back. You need to get the author to do an oral defense to verify that they wrote the paper. This is the main problem with AI generated output, whether it's a research paper, a blog, an email, a comment on a forum, similar: the value in knowing that a human wrote $X sends a signal - that the human understands what it is they wrote, even if they misunderstand the concepts. When you get a message from someone who is a "I only used an LLM to clean it up, the thoughts are all mine"[1] person, you cannot engage with them, because they may not understand the message they transmitted, and so any human engaging with them is only burning their own time for no gain. When you get a message from a real person, you get not only the message, you also get a signal about their understanding. That signal is missing in AI generated messages. ========================== [1] Sure, buddy. We believe you /s.
fn-mote 12d ago
The Medium comments on this post are also on point. Running the same experiment with accepted papers is a good control. Running a similar experiment with reviewers would be interesting, but more obnoxious because they are not being paid. I would keep a private blacklist (shadow ban) the authors who wasted several hours of a reviewer's time to prove they were not legitimate. The existence of such a list would be problematic, though. Could the same system we use here be applied? Accepted authors could "vouch" for "dead" papers in case they were "auto-killed"? This system is broken and providing more evidence that it is broken isn't much of a step towards fixing it.
- doc_ick 12d ago
  That would fail, humans and agents could create new “author” accounts by the swarm or have paid author accounts.
  - malfist 12d ago
    Do you imagine a world where people are willing to go through the legal hassle of changing their name to get past a ban for low effort journal submissions?
    - doc_ick 12d ago
      Legal names aren’t 1:1 with author names. If they were, who’d verify that?
      - otterley 12d ago
        Validating someone’s association with an institution by name seems like a reasonable thing to do. Perhaps it wasn’t done in the past, but times and circumstances have changed. Trust in authorship is lower than ever, and for good reason.
        doc_ick 12d ago
        It likely is, but that’s also on the assumption every author has to be in association with an institution and it’s quick to verify that. What good reason is there for trust in authorship to be low? I would likely agree for if it’s related to llm-slop.
mlmonkey 12d ago
IMHO (not a paper writer, but read a lot during my grad school years), the Genie is out of the bottle. The only way forward, as I see it, is using LLMs for reviews also. Basically, filter all submitted papers with an LLM and ask it to summarize it, find the biggest weaknesses and main strong points, etc. that a human can then use to review the paper. Basically, LLM-as-a-reviewer . Personally, I would love to see a conference where people are explicitly encouraged to use LLMs for doing the work and writing the papers, and LLMs are used to review them too.
- rsfern 12d ago
  I think that’s a lot of risk of anchoring reviewer bias. I’d be more comfortable with a triaged review where the editor’s office uses models to score whether a human editor should evaluate a paper to potentially send out for review, then the editor makes their own assessment, and the reviewers continue to do their job unassisted
- pwinnski 12d ago
  The entire point of an academic paper is to add to the sum of human knowledge. How can an LLM trained on a subset of human knowledge possibly even begin to accurate evaluate such a paper? I trust an LLM to review that the language used in the paper is grammatically correct, but not to evaluate new information for accuracy.
  - MikhailTal 12d ago
    This is very bad logic 1) Humans also are trained on a subset of human knowledge. 2)A lot of papers are just about experimenting something, and then applying simple stats. Eg empirical studies, around 1/3rd of published papers. Like, we tried this drug or did this experiment, from a sample size X here are the results. An expert is needed to maybe comment on the conclusion/hypothesis of the underlying suspected mechanism, but LLMs are still very useful on catching bad statistics or p hacking (so so common)
  - zbyforgotp 12d ago
    Schmidthuber has an answer: https://arxiv.org/abs/0812.4360 https://arxiv.org/abs/0812.4360 For a more practical approach you need to use proxies: https://zby.github.io/commonplace/articles/what-an-automated-reviewer-should-measure/ https://zby.github.io/commonplace/articles/what-an-automated...
softwaredoug 12d ago
Is it really “research” - as in expanding human knowledge - if nobody understands it? The point is deepening human understanding, not producing research papers
sampo 12d ago
> All reviewers, Action Editors and Editors-in-Chief for TMLR are unpaid volunteers. He is not an unpaid volunteer. He's an associate professor at the prestigious Carnegie Mellon University. He is not paid by the journal, but he is paid a salary by the university, and the university expects that a small part of his academic work is to serve as an editor in academic journals.
- azan_ 12d ago
  Elsevier, springer and mdpi have literally billions dollars in profit thanks to this free labor. Why would university pay for performing labor for for-profit companies? The system is broken and we should name things as they are - it’s free labor.
- croissants 12d ago
  There are much easier and more prestigious ways of fulfilling university service requirements than reviewing and editing for TMLR, so I think it is correct in spirit to label it as volunteering.
- otherme123 12d ago
  > and the university expects that a small part of his academic work is to serve as an editor in academic journals. Some do it for the CV, some do it for science, but I don't know any Uni that expects them to be editor more than they expect them to edit the Wikipedia or write popular science. Cool if you do it, but not expected.
WCSTombs 12d ago
This is the journal's policy on LLM use by authors [1]: > LLMs may be used as general-purpose assistive tools. Whichever tools are used, authors are fully responsible for content on which they are listed as (co-) authors. This includes, but is not limited to, content generated by LLMs that could be construed as plagiarism or scientific misconduct (e.g., fabrication of facts). Low-quality contributions (be they submissions or reviews) that appear to be largely LLM-generated will be closely examined for evidence of the issues mentioned previously, such as scientific misconduct. LLMs are not eligible for authorship. We will periodically revise this policy as new information about the use of LLMs in the scientific process becomes available. While it doesn't outright encourage using LLMs, it's right at the door, and IMO a policy this weak is actively contributing to the problem the article's author is complaining about. In my opinion any policy weaker than "using LLMs to generate any part of your submission is not allowed and considered a serious breach of ethics" is insane. People like to say that such policies are unenforceable, but that's really not the point (at first), since there are other things like (somewhat ironically) p-hacking that are pretty hard to detect but still widely recognized as unethical. We haven't exactly solved p-hacking either, but at least most of us can agree that p-hacking should be eliminated. It's hard for me not to read between the lines here. Maybe it's the tinfoil talking, but it being a machine-learning journal, it probably embodies a generally pro-AI philosophy, and thus may not want to discourage too much of it... It may also be worth noting that this journal apparently uses AI itself on the reviewing side [2]. I'm not claiming this is super unethical or anything as long as the main review is human (although I have concerns), it probably should be part of the conversation. [1]: https://jmlr.org/tmlr/editorial-policies.html https://jmlr.org/tmlr/editorial-policies.html [2]: https://medium.com/@TmlrOrg/ai-reviews-at-tmlr-for-assessing-soundness-4405d7813cc8 https://medium.com/@TmlrOrg/ai-reviews-at-tmlr-for-assessing...
- adastra22 12d ago
  So you can’t interact with an LLM, or use any product that has LLM features when performing the research?
  - WCSTombs 12d ago
    That is not what I said at all. For research whose ostensible purpose is to increase human understanding, the absolutely bare minimum we could ask for is for the authors to understand their own work well enough to write their research paper themselves, without having an LLM crank it out. I'm not saying anything about LLMs at other points during the research. Unless I'm missing something, the TMLR policy does not forbid using LLMs to generate entire research papers. It only says the authors are "responsible for the content" and warns against "low-quality contributions."
c7b 12d ago
> Separately, our group has been exploring approaches along these lines to make such evaluations more scalable Actually, that sounds like an interesting idea for peer review in general, to include an interview between referees and authors. If it saves one round of rebuttals/reactions, it needn't even consume a lot more of everyone's time if you're doing those things properly. What it would undermine would be blindness, but something's gotta give, and it was already on its way out.
dovholuknf 12d ago
Sounds like I shouldn't be so hard on AI when it hallucinates things based on this data? :)
logicallee 12d ago
I answered requests to be a peer reviewer. (I'm not sure why I was selected, I don't have many publications or credentials.) I saw a lot of papers with hallucinated references. I also remember one paper that described a methodology that I don't think the authors really performed, I think it was just academic fraud where they pretended to have performed an experiment. At the time that I answered the journal requests, AI could hallucinate fake reports, but agents weren't powerful enough to run the experiments yet. These days agents are able to really perform genuine experiments and write up the results. A prompt like this: "You'll work autonomously end to end to select a research task that meaningfully advances the state of the art in AI, is clearly defined and worth performing, that people would be interested in reading, and that you can perform on this hardware" (insert details) " in a week. Carefully log your steps so that your results can be replicated. Then, do a research review and write your paper about it up with correct, cited references. You must check all of your citations. Look up current lists of "Claudisms", (such as use of the word "genuinely", or "load-bearing"), and remove them from your writeup. After writing your writeup, edit it and pare it down, remove anything unnecessary, keep it fast paced and interesting. Also, try to tell a story, be engaging in your writeup. Don't use violent metaphors, remove references to killing, strangulation, etc. Your writeup should be ready to publish and accurately reflect a real experiment with a meaningful result that advances the state of the art and contributes to understanding. Be concise and focus on why it matters." Okay, so there's the prompt. You can give it to any AI and have a journal-ready publication in a week. I guess you can ask it to add charts and stuff, if you want to be fancy. If I gave my agent the above prompt, would I be one of the authors? Maybe it's fair to say I guided, facilitated, elicited, or advised it. But it's clear that the AI would be the one that is actually selecting and running the experiment and writing up the results. Someone could probably get a publication without even reading the paper they wrote their name on. Their only contribution might be editing their name into the PDF.
- kukkeliskuu 12d ago
  It can be turned around as well, for validating existing research, creating a bot that checks papers against all the well-known logical fallacies, issues with statistical methods, checks the images etc.
muh_gradle 12d ago
completely necessary.
cgio 12d ago
Curation is the new skill. The dismissal of a result on the basis of authorship is one of the reasons blind reviews are there. It sounds like we need more scientists. In al seriousness, this is a skill not just for science. Even at work, the amount of slop is rising exponentially and people are half-treating the symptom with its source, using AI to summarise. We will find our ways eventually.
trombuance 12d ago
[dead]
WD-42 12d ago
This could be renamed "Asking authors about their own pull requests" and the outcome would be exactly the same.
syntamono 12d ago
In the same vein, the Symposium on Theory of Computing (STOC) for 2027 has made some interesting changes to its Call For Papers (https://acm-stoc.org/stoc2027/stoc2027-cfp.html https://acm-stoc.org/stoc2027/stoc2027-cfp.html), notably requiring that papers be submitted beforehand to a preprint repository and that authors submit a 20-30 minutes video presentation explaining their results.
- emil-lp 11d ago
  STOC is the premier outlet of theoretical computer science results, so let's hope others follow: FOCS, SODA,...
- DataDive 11d ago
  What keeps someone from generating this video with AI?
  - Oranguru 11d ago
    AI has not yet reached the point where it can convincingly generate long presentations with the precision required for complex, technical academic topics. It will eventually get there, at which point an additional layer of proof of work will need to be introduced. This is an adversarial race. I think this is an interesting and effective solution, at least for now. Similarly, graduate and master's theses should focus more on the presentation and on eliciting knowledge from the students through critical, thorough questioning than on the tangible outcome of the project, which can easily and bindly be obtained with AI these days.
Havoc 11d ago
LLMs of course, but think the real issue here is incentives. Clearly the participants are incentivized to publish garbage. That's going to crater the signal to noise ratio of papers so best that be fixed asap. How though...
danieltanfh95 11d ago
Academia is in a time for reckoning. Actors who already went through the process (or otherwise) gained sufficient reputation or credentials to self-market their own paper can skip journals entirely. That was what OpenAI did. With a sufficiently powerful AI model and correctional pipelines, generating a paper is trivial given some insight. I think we should be reminded that papers are a channel to distribute papers. Editors are unpaid now, but the economics of a journals are such that the editors are incentivized to curate or distribute papers to schools that pay for the paper. Some perceive quality as a core metric for this. However, in my experience of dealing with computational biology, a paper in so and so journal hardly means a stamp of quality as compared to a paper in some github repo with code to reproduce the paper. This, simply, is broken, because journals and peer reviewers cannot guarantee that data and results in the paper is correct (assuming that it is not maths or theoretical) without reproducing the results in the paper. None of these are helpful towards students who are already struggling to keep up with the cadence of producing papers.
ChrisMarshallNY 11d ago
That’s a great “story from the trenches.” I’m wondering if anyone just flat-out admits they used LLMs in their work. I have no problem, doing that, myself, but I also have the luxury of not having my livelihood on the line, and not being overly-concerned about what people think of me. Eventually, I assume that AI will affect every aspect of the industry, and it will actually be a signal of effectiveness, to claim its use. I could see a day, when claims of not using AI would be like artisans, declaring their work to be “genuine hand-crafted,” and relegated to fairly small, specialized corners of the industry.
- Tanjreeve 11d ago
  If we can't reproduce the work or it doesn't work or it's not actually doing anything of value it's less like "handmade furniture Vs Ikea" and more like "handmade furniture Vs a bundle of sticks". I don't know what it is but something is clearly wrong that people are upset about the people involved not really understanding the things they're building when they have fulfilled "the brief" using LLMs. Sometimes it feels silly like going through the motions of typing code is somehow helpful. But I think there is some line where people are justified in being concerned when a business or a paper doesn't actually understand what they're ostensibly "owning" or improving as a product/paper.
  - ChrisMarshallNY 11d ago
    True, but we don't expect programmers to be able to be given a printout of machine language instructions from their shipped binary, and explain it. At one time, in my career, that was actually expected. Times, they are a changin'...
    - Tanjreeve 11d ago
      When we moved to higher level languages we still had the expectation that software would work and/or be able to be improved. If it actively doesn't work or noone can use research because the originator can't explain it then we're going backwards and wasting money/time. "Times they are a changing" just seems like more of the premature backslapping over a tool that allows us to rig up software to go in a direction quicker. At the end of the pipe someone has to pay for it or use the research and they don't care if it's an LLM or not same as they don't care if it was written in an IDE or not. And that works both ways.
figassis 11d ago
> When authors could not answer questions about the technical parts, and sometimes even basic questions about the paper, it is difficult to see how they could have verified the paper's contents If you cannot answer, you did not author the paper, meaning you are misrepresenting your contribution, and there is already a process for this. And this is actually a really good test for any field. Use AI as much as you want, but you need to be able to explain your work. Applies to SWE as well, you need to understadn what you built, at the code level and system level.
lh712 11d ago
An observation from within the system: A year or so ago I completed a PhD in theoretical high energy physics. I did that after working for about 5 years outside of academia [the reasons for that were financial issues in my family]. Because of this hiatus (I suppose) and the fact that I had troubles getting reference letters (my master thesis advisor passed away at quite a young age right at the time when I tried to start applying for positions; we actually agreed to meet in person to discuss next steps but the meeting never happened) I had issues getting accepted into PhD programs and I ended up with an offer from a rather weak institution in my home country (which is in EU). I had no other options to choose from (and I could not wait another year) so I accepted. However, I also looked at the publishing record of the team that I was about to join. Their works did not seem stellar (and I did not expect that) but they seemed to publish regularly, on several topics, and with several collaborating institutions. When I started I quickly realized that the knowledge/expertise level was much lower than I expected (and I did not expect too much). The group consisted of the group leader and three senior researchers, and I am pretty sure that my knowledge of QFT when I *started* working with them was the best in the group. (Now, I had studied in my own time during some part of the years I was outside of academia, and I think my knowledge at that time exceeded that of an average 1st year hep-theory PhD student in Europe, but it was also nothing spectacular; definitely not what I would call "competent"; which is admittedly a high bar in quantum field theory / hep theory, but something I would expect senior researchers to more or less satisfy.) I realized that the one topic on which the group was publishing on their own was just a continual rehash of the same thing (applied to various different problems, so perhaps not completely useless) and the other topics, those which seemed more advanced when I originally looked at the publication record, were all done essentially outside the group by other people. The group members contributed with some non-essential help (often resembling a work done by a student, such as finalizing a manuscript or double checking calculations) or with nothing; and were added as coauthors due to some other reasons (I suppose: past loyalties, friend groups, advantage of having foreign or external institution co-authors). [Note: I was not included into any of those collaborations so I am not speaking with a full knowledge of the inner workings there.] During this time I saw that people being added as coauthors had a very noisy relation to what they did or didn't do on any given paper. I was added as a co-author on a paper that I made essentially no work on [not fair], I was added as a co-author on papers which drew upon some of my earlier results [fair, but I did not subscribe to the overall spirit of the paper or even to the paper being published at all], I was a co-author of papers where number of authors and my positions in the list roughly corresponded to my contribution [as it should be], I was a co-author of papers that were done basically by myself but there were several other co-authors included who made a little to no contribution [and the ordering of the author list was always alphabetical not reflecting the relative contribution]. There is also another aspect of this: If you are an early career researcher (PhD student, PostDoc, and even non-tenured professor) you might have a very limited control over who gets included on the publications you work on, and on what publications your name appears. In my case, I am a very disagreeable person when I think things are being done wrongly and yet I have not managed to refuse from being included on papers I did not want to be a co-author of, or to prevent people who contributed nothing to be included on my papers. [I mean the only way to achieve that meant escalating conflict into levels which (a) would likely disable any further cooperation with the group, and (b) might be even morally questionable concerning the level of distress it would make to other people who just seemed to be fine about "the normal way" things were being done.] In any case, while there are still some really good people in academia (actually the best people I met in my life were nearly all in academia), in overall (by no means fully indicated in the above paragraphs) I feel very bitter about it, and I think it is so dysfunctional and morally corrupted, that it is nearly inevitable that it collapses in future. The current and future shockwaves of the change in the public sentiment and funding, the evolution of demographics, and the consequences of AI, just hasten the process that would have probably unfolded, sooner or later, anyway.
pc86 11d ago
This triggers a much bigger question for me - why does nothing happen to these authors based on this feedback? I don't mean "oh you missed a meeting or withdrew your paper so you get fired" by any means, but why isn't there a way for academic committees or employers to look up PhDs and see "oh interesting this person has a history of submitting solo-authored papers then being completely unable to answer all but the most high-level questions about it." Short of just the general "vibes" type of reputation that follows someone around this type of behavior seems pretty low risk, which is a large part of why people engage in it. Perhaps the risk should increase a couple orders of magnitude to stop it from happening.
- Dashadower 11d ago
  There needs to be repercussions. At the moment you can try sending slop papers to each journal and if just one sticks it makes it more than worth.

4 points by stefanpie 15d ago

Peer review has its historical issues, but the landscape of science and science-publishing has changed. New problems of authorship and authorial-understanding are now challenged by LLMs writing (at least) good sounding papers - some of which might be of acceptable quality in subject (I am not against AI in the sciences; some of the math work has been great). On the other hand: I am against authors not understanding their own work. High repute journals may need to add "oral exams" to the paper acceptance process...

N_Lens 12d ago

I think authenticity and trust will command a (larger) premium in this new age of slop. The article highlights how only one out of ten paper’s authors were able to answer questions thoroughly and at a high level. This indicates an overwhelming percentage of authors are slopping up their work with AI and submitting it without even reading it.

SoftTalker 12d ago
No doubt this is happening, but I wonder how many authors of papers "slated for desk rejection" 10 years ago could answer questions about their papers? We'd need that comparison to understand if this is a new problem or if AI is just a new source of content that the authors of poorly-written papers are using.
- brianpan 12d ago
  Evidence-based assertions are a good thing, but some things are so obvious that a "comparison" or "research" is not needed. This is already an obvious problem in so many places from high schoolers turning in assignments they don't understand, coders submitting code changes they don't understand, blog posts, and certainly to scientific papers.
  - SoftTalker 12d ago
    Yeah sadly you're probably right.

Calazon 12d ago

I wonder how the ratios would change for papers at different parts of the review process. For what fraction of published papers are the authors unable to answer basic questions about them?

jszymborski 12d ago

FYI in case the author is reading, https://www.cs.cmu.edu/~nihars/preprints/greCAPTCHA.pdf https://www.cs.cmu.edu/~nihars/preprints/greCAPTCHA.pdf is a dead link. EDIT: I found a live link on arxiv https://arxiv.org/html/2609.20481v1 https://arxiv.org/html/2609.20481v1

encyclopediai 12d ago
In June 2026 I proposed a CAPTCHA for scientific publications https://chorasimilarity.wordpress.com/2026/06/13/a-captcha-for-scientific-publications/ https://chorasimilarity.wordpress.com/2026/06/13/a-captcha-f... At the moment this was seen as a tongue in cheek proposal.
- doc_ick 12d ago
  I’d agree it’d be a funny proposal, wouldn’t have worked back then but funny.
  - encyclopediai 12d ago
    Thanks. IMO the most fun is in the CAPTCHA, which turns on its head the Turing test. But it goes even further than their greCAPTCHA and it solves their consumed time problem. Indeed, in their proposal they have a human bottleneck, but in the june 2026 proposal is suggested that one could use an AI to generate the results without the knowledge of the submitted article. If the AI can generate a pretty close result, with the article fed gradually as a prompt, then reject. And even further, that it might be not even a need to publish anymore. Just use the article for training and make a public database with some numbers about the successful researcher, where we see an influence score (how many times an idea from an accepted article are used by other accepted articles), a publication score (how many articles the author had). I wonder if the reality will be more or less surprising, my bet is on "more".
    - totetsu 11d ago
      Sorry to be uncool and try and explain away the joke, but. I'm reading three parts to this.. one is the risk to the job of researches, by the impacts of the systems that they have typically used until now, by being flooded with texts that look like research articles, but are produced by llms.. another is a critique of that system in the first place, in the interaction of both for profit publishing, and research funding based on measuring of publishing in those for-profit journals.. and then there is also a concern about theft of work, and plagiarism, by AI providers, ingesting pre-published work, and training on it, and additionally those providers maybe acting as gate-keepers or .. imposing their own normative values or being opaque and not impartial in how they operate.. Did I parse that correctly?

greenflag 12d ago

One larger problem here is the value of a research paper is rarely the specific knowledge it adds but in the process of researching that adds to the collective knowledge+experience of those involved, especially training graduate students. AI papers shortcut this entirely. Academia has a lot to answer for this too by making papers the currency of success. AI generated papers are almost shortcut learning at a full system level.

doc_ick 12d ago
I disagree for academia using papers as an easy medium for verification and providing more knowledge. Llms always short circuit everything, so how would you fix academia?
PantaloonFlames 12d ago
Related: Jan 2026 https://www.theatlantic.com/science/2026/01/ai-slop-science-publishing/685704/ https://www.theatlantic.com/science/2026/01/ai-slop-science-... "For more than a century, scientific journals have been the pipes through which knowledge of the natural world flows into our culture. Now they’re being clogged with AI slop." Sept 2026 https://www.theatlantic.com/ideas/2026/09/college-education-future-ai/688655/ https://www.theatlantic.com/ideas/2026/09/college-education-... Academia, particularly the university system, is an untenable collection of interests. The triple stresses of COVID, AI, and funding withdrawal seem to presage what will be a significant disruption.

wackget 12d ago

I'm not familiar with the world of academic publishing, so I want to ask: how is the industry making sure that submissions aren't at least partially AI-generated? Is it standard practice for authors to have to defend their submissions via interview like this? If not, why not? Does the vetting process vary with the quality of the publisher? As an outsider, it's extremely worrying that anyone would even attempt to submit an AI-generated paper for publication in an academic journal. At that level I would have assumed literally everybody should know better than to even try.

whattheheckheck 12d ago
The rates of paper publishing show that everyone must be using ai now or the rates wouldn't have gone up
lelanthran 12d ago
> Is it standard practice for authors to have to defend their submissions via interview like this? If not, why not? It isn't, but maybe it should be. For post-grad qualifications oral defense is standard, and I didn't mind defending my central thesis then, and won't mind now.
- doc_ick 12d ago
  Not in a direct interview style, but most us conferences can request additional information or feedback. If they conditionally accept or reject a paper, that conditional relies on feedback from the author(s). Interviews like this are interesting, but in no way can scale to the infinite paper slop conferences are facing.
  - lelanthran 12d ago
    > Not in a direct interview style, but most us conferences can request additional information or feedback. If they conditionally accept or reject a paper, that conditional relies on feedback from the author(s). Requesting feedback is useless, as the article points out - the "authors" could not answer basic questions during the interview, but after the interview were able to send full explanations to the interviewer. If you have indirect feedback ("please answer these questions we have") the "author" will simply feed it into an LLM and send the results back. You need to get the author to do an oral defense to verify that they wrote the paper. This is the main problem with AI generated output, whether it's a research paper, a blog, an email, a comment on a forum, similar: the value in knowing that a human wrote $X sends a signal - that the human understands what it is they wrote, even if they misunderstand the concepts. When you get a message from someone who is a "I only used an LLM to clean it up, the thoughts are all mine"[1] person, you cannot engage with them, because they may not understand the message they transmitted, and so any human engaging with them is only burning their own time for no gain. When you get a message from a real person, you get not only the message, you also get a signal about their understanding. That signal is missing in AI generated messages. ========================== [1] Sure, buddy. We believe you /s.

fn-mote 12d ago

The Medium comments on this post are also on point. Running the same experiment with accepted papers is a good control. Running a similar experiment with reviewers would be interesting, but more obnoxious because they are not being paid. I would keep a private blacklist (shadow ban) the authors who wasted several hours of a reviewer's time to prove they were not legitimate. The existence of such a list would be problematic, though. Could the same system we use here be applied? Accepted authors could "vouch" for "dead" papers in case they were "auto-killed"? This system is broken and providing more evidence that it is broken isn't much of a step towards fixing it.

doc_ick 12d ago
That would fail, humans and agents could create new “author” accounts by the swarm or have paid author accounts.
- malfist 12d ago
  Do you imagine a world where people are willing to go through the legal hassle of changing their name to get past a ban for low effort journal submissions?
  - doc_ick 12d ago
    Legal names aren’t 1:1 with author names. If they were, who’d verify that?
    - otterley 12d ago
      Validating someone’s association with an institution by name seems like a reasonable thing to do. Perhaps it wasn’t done in the past, but times and circumstances have changed. Trust in authorship is lower than ever, and for good reason.
      - doc_ick 12d ago
        It likely is, but that’s also on the assumption every author has to be in association with an institution and it’s quick to verify that. What good reason is there for trust in authorship to be low? I would likely agree for if it’s related to llm-slop.

mlmonkey 12d ago

IMHO (not a paper writer, but read a lot during my grad school years), the Genie is out of the bottle. The only way forward, as I see it, is using LLMs for reviews also. Basically, filter all submitted papers with an LLM and ask it to summarize it, find the biggest weaknesses and main strong points, etc. that a human can then use to review the paper. Basically, LLM-as-a-reviewer . Personally, I would love to see a conference where people are explicitly encouraged to use LLMs for doing the work and writing the papers, and LLMs are used to review them too.

rsfern 12d ago
I think that’s a lot of risk of anchoring reviewer bias. I’d be more comfortable with a triaged review where the editor’s office uses models to score whether a human editor should evaluate a paper to potentially send out for review, then the editor makes their own assessment, and the reviewers continue to do their job unassisted
pwinnski 12d ago
The entire point of an academic paper is to add to the sum of human knowledge. How can an LLM trained on a subset of human knowledge possibly even begin to accurate evaluate such a paper? I trust an LLM to review that the language used in the paper is grammatically correct, but not to evaluate new information for accuracy.
- MikhailTal 12d ago
  This is very bad logic 1) Humans also are trained on a subset of human knowledge. 2)A lot of papers are just about experimenting something, and then applying simple stats. Eg empirical studies, around 1/3rd of published papers. Like, we tried this drug or did this experiment, from a sample size X here are the results. An expert is needed to maybe comment on the conclusion/hypothesis of the underlying suspected mechanism, but LLMs are still very useful on catching bad statistics or p hacking (so so common)
- zbyforgotp 12d ago
  Schmidthuber has an answer: https://arxiv.org/abs/0812.4360 https://arxiv.org/abs/0812.4360 For a more practical approach you need to use proxies: https://zby.github.io/commonplace/articles/what-an-automated-reviewer-should-measure/ https://zby.github.io/commonplace/articles/what-an-automated...

softwaredoug 12d ago

Is it really “research” - as in expanding human knowledge - if nobody understands it? The point is deepening human understanding, not producing research papers

sampo 12d ago

> All reviewers, Action Editors and Editors-in-Chief for TMLR are unpaid volunteers. He is not an unpaid volunteer. He's an associate professor at the prestigious Carnegie Mellon University. He is not paid by the journal, but he is paid a salary by the university, and the university expects that a small part of his academic work is to serve as an editor in academic journals.

azan_ 12d ago
Elsevier, springer and mdpi have literally billions dollars in profit thanks to this free labor. Why would university pay for performing labor for for-profit companies? The system is broken and we should name things as they are - it’s free labor.
croissants 12d ago
There are much easier and more prestigious ways of fulfilling university service requirements than reviewing and editing for TMLR, so I think it is correct in spirit to label it as volunteering.
otherme123 12d ago
> and the university expects that a small part of his academic work is to serve as an editor in academic journals. Some do it for the CV, some do it for science, but I don't know any Uni that expects them to be editor more than they expect them to edit the Wikipedia or write popular science. Cool if you do it, but not expected.

WCSTombs 12d ago

This is the journal's policy on LLM use by authors [1]: > LLMs may be used as general-purpose assistive tools. Whichever tools are used, authors are fully responsible for content on which they are listed as (co-) authors. This includes, but is not limited to, content generated by LLMs that could be construed as plagiarism or scientific misconduct (e.g., fabrication of facts). Low-quality contributions (be they submissions or reviews) that appear to be largely LLM-generated will be closely examined for evidence of the issues mentioned previously, such as scientific misconduct. LLMs are not eligible for authorship. We will periodically revise this policy as new information about the use of LLMs in the scientific process becomes available. While it doesn't outright encourage using LLMs, it's right at the door, and IMO a policy this weak is actively contributing to the problem the article's author is complaining about. In my opinion any policy weaker than "using LLMs to generate any part of your submission is not allowed and considered a serious breach of ethics" is insane. People like to say that such policies are unenforceable, but that's really not the point (at first), since there are other things like (somewhat ironically) p-hacking that are pretty hard to detect but still widely recognized as unethical. We haven't exactly solved p-hacking either, but at least most of us can agree that p-hacking should be eliminated. It's hard for me not to read between the lines here. Maybe it's the tinfoil talking, but it being a machine-learning journal, it probably embodies a generally pro-AI philosophy, and thus may not want to discourage too much of it... It may also be worth noting that this journal apparently uses AI itself on the reviewing side [2]. I'm not claiming this is super unethical or anything as long as the main review is human (although I have concerns), it probably should be part of the conversation. [1]: https://jmlr.org/tmlr/editorial-policies.html https://jmlr.org/tmlr/editorial-policies.html [2]: https://medium.com/@TmlrOrg/ai-reviews-at-tmlr-for-assessing-soundness-4405d7813cc8 https://medium.com/@TmlrOrg/ai-reviews-at-tmlr-for-assessing...

adastra22 12d ago
So you can’t interact with an LLM, or use any product that has LLM features when performing the research?
- WCSTombs 12d ago
  That is not what I said at all. For research whose ostensible purpose is to increase human understanding, the absolutely bare minimum we could ask for is for the authors to understand their own work well enough to write their research paper themselves, without having an LLM crank it out. I'm not saying anything about LLMs at other points during the research. Unless I'm missing something, the TMLR policy does not forbid using LLMs to generate entire research papers. It only says the authors are "responsible for the content" and warns against "low-quality contributions."

c7b 12d ago

> Separately, our group has been exploring approaches along these lines to make such evaluations more scalable Actually, that sounds like an interesting idea for peer review in general, to include an interview between referees and authors. If it saves one round of rebuttals/reactions, it needn't even consume a lot more of everyone's time if you're doing those things properly. What it would undermine would be blindness, but something's gotta give, and it was already on its way out.

dovholuknf 12d ago

Sounds like I shouldn't be so hard on AI when it hallucinates things based on this data? :)

logicallee 12d ago

I answered requests to be a peer reviewer. (I'm not sure why I was selected, I don't have many publications or credentials.) I saw a lot of papers with hallucinated references. I also remember one paper that described a methodology that I don't think the authors really performed, I think it was just academic fraud where they pretended to have performed an experiment. At the time that I answered the journal requests, AI could hallucinate fake reports, but agents weren't powerful enough to run the experiments yet. These days agents are able to really perform genuine experiments and write up the results. A prompt like this: "You'll work autonomously end to end to select a research task that meaningfully advances the state of the art in AI, is clearly defined and worth performing, that people would be interested in reading, and that you can perform on this hardware" (insert details) " in a week. Carefully log your steps so that your results can be replicated. Then, do a research review and write your paper about it up with correct, cited references. You must check all of your citations. Look up current lists of "Claudisms", (such as use of the word "genuinely", or "load-bearing"), and remove them from your writeup. After writing your writeup, edit it and pare it down, remove anything unnecessary, keep it fast paced and interesting. Also, try to tell a story, be engaging in your writeup. Don't use violent metaphors, remove references to killing, strangulation, etc. Your writeup should be ready to publish and accurately reflect a real experiment with a meaningful result that advances the state of the art and contributes to understanding. Be concise and focus on why it matters." Okay, so there's the prompt. You can give it to any AI and have a journal-ready publication in a week. I guess you can ask it to add charts and stuff, if you want to be fancy. If I gave my agent the above prompt, would I be one of the authors? Maybe it's fair to say I guided, facilitated, elicited, or advised it. But it's clear that the AI would be the one that is actually selecting and running the experiment and writing up the results. Someone could probably get a publication without even reading the paper they wrote their name on. Their only contribution might be editing their name into the PDF.

kukkeliskuu 12d ago
It can be turned around as well, for validating existing research, creating a bot that checks papers against all the well-known logical fallacies, issues with statistical methods, checks the images etc.

muh_gradle 12d ago

completely necessary.

cgio 12d ago

Curation is the new skill. The dismissal of a result on the basis of authorship is one of the reasons blind reviews are there. It sounds like we need more scientists. In al seriousness, this is a skill not just for science. Even at work, the amount of slop is rising exponentially and people are half-treating the symptom with its source, using AI to summarise. We will find our ways eventually.

trombuance 12d ago

[dead]

WD-42 12d ago

This could be renamed "Asking authors about their own pull requests" and the outcome would be exactly the same.

syntamono 12d ago

In the same vein, the Symposium on Theory of Computing (STOC) for 2027 has made some interesting changes to its Call For Papers (https://acm-stoc.org/stoc2027/stoc2027-cfp.html https://acm-stoc.org/stoc2027/stoc2027-cfp.html), notably requiring that papers be submitted beforehand to a preprint repository and that authors submit a 20-30 minutes video presentation explaining their results.

emil-lp 11d ago
STOC is the premier outlet of theoretical computer science results, so let's hope others follow: FOCS, SODA,...
DataDive 11d ago
What keeps someone from generating this video with AI?
- Oranguru 11d ago
  AI has not yet reached the point where it can convincingly generate long presentations with the precision required for complex, technical academic topics. It will eventually get there, at which point an additional layer of proof of work will need to be introduced. This is an adversarial race. I think this is an interesting and effective solution, at least for now. Similarly, graduate and master's theses should focus more on the presentation and on eliciting knowledge from the students through critical, thorough questioning than on the tangible outcome of the project, which can easily and bindly be obtained with AI these days.

Havoc 11d ago

LLMs of course, but think the real issue here is incentives. Clearly the participants are incentivized to publish garbage. That's going to crater the signal to noise ratio of papers so best that be fixed asap. How though...

danieltanfh95 11d ago

Academia is in a time for reckoning. Actors who already went through the process (or otherwise) gained sufficient reputation or credentials to self-market their own paper can skip journals entirely. That was what OpenAI did. With a sufficiently powerful AI model and correctional pipelines, generating a paper is trivial given some insight. I think we should be reminded that papers are a channel to distribute papers. Editors are unpaid now, but the economics of a journals are such that the editors are incentivized to curate or distribute papers to schools that pay for the paper. Some perceive quality as a core metric for this. However, in my experience of dealing with computational biology, a paper in so and so journal hardly means a stamp of quality as compared to a paper in some github repo with code to reproduce the paper. This, simply, is broken, because journals and peer reviewers cannot guarantee that data and results in the paper is correct (assuming that it is not maths or theoretical) without reproducing the results in the paper. None of these are helpful towards students who are already struggling to keep up with the cadence of producing papers.

ChrisMarshallNY 11d ago

That’s a great “story from the trenches.” I’m wondering if anyone just flat-out admits they used LLMs in their work. I have no problem, doing that, myself, but I also have the luxury of not having my livelihood on the line, and not being overly-concerned about what people think of me. Eventually, I assume that AI will affect every aspect of the industry, and it will actually be a signal of effectiveness, to claim its use. I could see a day, when claims of not using AI would be like artisans, declaring their work to be “genuine hand-crafted,” and relegated to fairly small, specialized corners of the industry.

Tanjreeve 11d ago
If we can't reproduce the work or it doesn't work or it's not actually doing anything of value it's less like "handmade furniture Vs Ikea" and more like "handmade furniture Vs a bundle of sticks". I don't know what it is but something is clearly wrong that people are upset about the people involved not really understanding the things they're building when they have fulfilled "the brief" using LLMs. Sometimes it feels silly like going through the motions of typing code is somehow helpful. But I think there is some line where people are justified in being concerned when a business or a paper doesn't actually understand what they're ostensibly "owning" or improving as a product/paper.
- ChrisMarshallNY 11d ago
  True, but we don't expect programmers to be able to be given a printout of machine language instructions from their shipped binary, and explain it. At one time, in my career, that was actually expected. Times, they are a changin'...
  - Tanjreeve 11d ago
    When we moved to higher level languages we still had the expectation that software would work and/or be able to be improved. If it actively doesn't work or noone can use research because the originator can't explain it then we're going backwards and wasting money/time. "Times they are a changing" just seems like more of the premature backslapping over a tool that allows us to rig up software to go in a direction quicker. At the end of the pipe someone has to pay for it or use the research and they don't care if it's an LLM or not same as they don't care if it was written in an IDE or not. And that works both ways.

figassis 11d ago

> When authors could not answer questions about the technical parts, and sometimes even basic questions about the paper, it is difficult to see how they could have verified the paper's contents If you cannot answer, you did not author the paper, meaning you are misrepresenting your contribution, and there is already a process for this. And this is actually a really good test for any field. Use AI as much as you want, but you need to be able to explain your work. Applies to SWE as well, you need to understadn what you built, at the code level and system level.

lh712 11d ago

An observation from within the system: A year or so ago I completed a PhD in theoretical high energy physics. I did that after working for about 5 years outside of academia [the reasons for that were financial issues in my family]. Because of this hiatus (I suppose) and the fact that I had troubles getting reference letters (my master thesis advisor passed away at quite a young age right at the time when I tried to start applying for positions; we actually agreed to meet in person to discuss next steps but the meeting never happened) I had issues getting accepted into PhD programs and I ended up with an offer from a rather weak institution in my home country (which is in EU). I had no other options to choose from (and I could not wait another year) so I accepted. However, I also looked at the publishing record of the team that I was about to join. Their works did not seem stellar (and I did not expect that) but they seemed to publish regularly, on several topics, and with several collaborating institutions. When I started I quickly realized that the knowledge/expertise level was much lower than I expected (and I did not expect too much). The group consisted of the group leader and three senior researchers, and I am pretty sure that my knowledge of QFT when I *started* working with them was the best in the group. (Now, I had studied in my own time during some part of the years I was outside of academia, and I think my knowledge at that time exceeded that of an average 1st year hep-theory PhD student in Europe, but it was also nothing spectacular; definitely not what I would call "competent"; which is admittedly a high bar in quantum field theory / hep theory, but something I would expect senior researchers to more or less satisfy.) I realized that the one topic on which the group was publishing on their own was just a continual rehash of the same thing (applied to various different problems, so perhaps not completely useless) and the other topics, those which seemed more advanced when I originally looked at the publication record, were all done essentially outside the group by other people. The group members contributed with some non-essential help (often resembling a work done by a student, such as finalizing a manuscript or double checking calculations) or with nothing; and were added as coauthors due to some other reasons (I suppose: past loyalties, friend groups, advantage of having foreign or external institution co-authors). [Note: I was not included into any of those collaborations so I am not speaking with a full knowledge of the inner workings there.] During this time I saw that people being added as coauthors had a very noisy relation to what they did or didn't do on any given paper. I was added as a co-author on a paper that I made essentially no work on [not fair], I was added as a co-author on papers which drew upon some of my earlier results [fair, but I did not subscribe to the overall spirit of the paper or even to the paper being published at all], I was a co-author of papers where number of authors and my positions in the list roughly corresponded to my contribution [as it should be], I was a co-author of papers that were done basically by myself but there were several other co-authors included who made a little to no contribution [and the ordering of the author list was always alphabetical not reflecting the relative contribution]. There is also another aspect of this: If you are an early career researcher (PhD student, PostDoc, and even non-tenured professor) you might have a very limited control over who gets included on the publications you work on, and on what publications your name appears. In my case, I am a very disagreeable person when I think things are being done wrongly and yet I have not managed to refuse from being included on papers I did not want to be a co-author of, or to prevent people who contributed nothing to be included on my papers. [I mean the only way to achieve that meant escalating conflict into levels which (a) would likely disable any further cooperation with the group, and (b) might be even morally questionable concerning the level of distress it would make to other people who just seemed to be fine about "the normal way" things were being done.] In any case, while there are still some really good people in academia (actually the best people I met in my life were nearly all in academia), in overall (by no means fully indicated in the above paragraphs) I feel very bitter about it, and I think it is so dysfunctional and morally corrupted, that it is nearly inevitable that it collapses in future. The current and future shockwaves of the change in the public sentiment and funding, the evolution of demographics, and the consequences of AI, just hasten the process that would have probably unfolded, sooner or later, anyway.

pc86 11d ago

This triggers a much bigger question for me - why does nothing happen to these authors based on this feedback? I don't mean "oh you missed a meeting or withdrew your paper so you get fired" by any means, but why isn't there a way for academic committees or employers to look up PhDs and see "oh interesting this person has a history of submitting solo-authored papers then being completely unable to answer all but the most high-level questions about it." Short of just the general "vibes" type of reputation that follows someone around this type of behavior seems pretty low risk, which is a large part of why people engage in it. Perhaps the risk should increase a couple orders of magnitude to stop it from happening.

Dashadower 11d ago
There needs to be repercussions. At the moment you can try sending slop papers to each journal and if just one sticks it makes it more than worth.