22 ms·

GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

• fulafel 9mo ago
  Is there a comparison to rate of reference errors in other forums?
• cogman10 9mo ago
  Yuck, this is going to really harm scientific research. There is already a problem with papers falsifying data/samples/etc, LLMs being able to put out plausible papers is just going to make it worse. On the bright side, maybe this will get the scientific community and science journalists to finally take reproducibility more seriously. I'd love to see future reporting that instead of saying "Research finds amazing chemical x which does y" you see "Researcher reproduces amazing results for chemical x which does y. First discovered by z".
  • godzillabrennus 9mo ago
    Have they solved the issue where papers that cite research already invalidated are still being cited?
    • cogman10 9mo ago
      AFAIK, no, but I could see there being cause to push citations to also cite the validations. It'd be good if standard practice turned into something like Paper A, by bob, bill, brad. Validated by Paper B by carol, clare, charlotte. or Paper A, by bob, bill, brad. Unvalidated.
      • gcr 9mo ago
        Academics typically use citation count and popularity as a rough proxy for validation. It's certainly not perfect, but it is something that people think about. Semantic Scholar in particular is doing great work in this area, making it easy to see who cites who: https://www.semanticscholar.org/ https://www.semanticscholar.org/ Google Scholar's PDF reader extension turns every hyperlinked citation into a popout card that shows citation counts inline in the PDF: https://chromewebstore.google.com/detail/google-scholar-pdf-reader/dahenjhkoodjbpjheillcadbppiidmhp?hl=en https://chromewebstore.google.com/detail/google-scholar-pdf-...
        • rtkwe 9mo ago
          That is a factor most people miss when thinking about the replication crisis. For the harder physical sciences a wrong paper will fairly quickly be found because as people go to expand on the ideas/use that data and get results that don't match the model informed by paper X they're going to eventually figure out that X is wrong. There might be issues with getting incentives to write and publish that negative result but each paper where the results of a previous paper are actually used in the new paper is a form of replication.
    • reliabilityguy 9mo ago
      Nope. I am still reviewing papers that propose solutions based on a technique X, conveniently ignoring research from two years ago that shows that X cannot be used on its own. Both the paper I reviewed and the research showing X cannot be used are in the same venue!
      • b00ty4breakfast 9mo ago
        does it seem to be legitimate ignorance or maybe folks pushing ahead regardless of x being disproved?
        • freedomben 9mo ago
          IMHO, It's mostly ignorance coming a push/drive to "publish or perish." When the stakes are so high and output is so valued, and when reproducability isn't required, it disincentivizes thorough work. The system is set up in a way that is making it fail. There is also the reality that "one paper" or "one study" can be found contradicted almost anything, so if you just went with "some other paper/study debunks my premise" then you'd end up producing nothing. Plus many inside know that there's a lot of slop out there that gets published, so they can (sometimes reasonably IMHO) dismiss that "one paper" even when they do know about it. It's (mostly) not fraud or malicious intent or ignorance, it's (mostly) humans existing in the system in which they must live.
        • reliabilityguy 9mo ago
          Poor scholarship. However, given the feedback by other reviewers, I was the only one who knew that X doesn’t work. I am not sure how these people mark themselves as “experts” in the field if they are not following the literature themselves.
  • f311a 9mo ago
    For ML/AI/Comp sci articles, providing reproducible code is a great option. Basically, PoC or GTFO.
    • deleted 9mo ago
      [deleted]
    • StableAlkyne 9mo ago
      The most annoying ones are those which discuss loosely the methodology but then fail to publish the weights or any real algorithms. It's like buying a piece of furniture from IKEA, except you just get an Allen key, a hint at what parts to buy, and blurry instructions.
      • alansaber 8mo ago
        This is so egregious. The value of such papers is basically nothing but they're extremely common.
  • j45 9mo ago
    It will better expose the behaviour of false scientists.
  • StableAlkyne 9mo ago
    > I'd love to see future reporting that instead of saying "Research finds amazing chemical x which does y" you see "Researcher reproduces amazing results for chemical x which does y. First discovered by z". Most people (that I talk to, at least) in science agree that there's a reproducibility crisis. The challenge is there really isn't a good way to incentivize that work. Fundamentally (unless you're independent wealthy and funding your own work), you have to measure productivity somehow, whether you're at a university, government lab, or the private sector. That turns out to be very hard to do. If you measure raw number of papers (more common in developing countries and low-tier universities), you incentivize a flood of junk. Some of it is good, but there is such a tidal wave of shit that most people write off your work as a heuristic based on the other people in your cohort. So, instead it's more common to try to incorporate how "good" a paper is, to reward people with a high quantity of "good" papers. That's quantifying something subjective though, so you might try to use something like citation count as a proxy: if a work is impactful, usually it gets cited a lot. Eventually you may arrive at something like the H-index, which is defined as "The highest number H you can pick, where H is the number of papers you have written with H citations." Now, the trouble with this method is people won't want to "waste" their time on incremental work. And that's the struggle here; even if we funded and rewarded people for reproducing results, they will always be bumping up the citation count of the original discoverer. But it's worse than that, because literally nobody is going to cite your work. In 10 years, they just see the original paper, a few citing works reproducing it, and to save time they'll just cite the original paper only. There's clearly a problem with how we incentivize scientific work. And clearly we want to be in a world where people test reproducibility. However, it's very very hard to get there when one's prestige and livelihood is directly tied to discovery rather than reproducibility.
    • warkdarrior 9mo ago
      > If you measure raw number of papers (more common in developing countries and low-tier universities), you incentivize a flood of junk. This is exactly what rewarding replication papers (that reproduce and confirm an existing paper) will lead to.
      • pixl97 9mo ago
  • agumonkey 9mo ago
    I think, at least I hope, that a part of the LLM value will be to create their retirement for specific needs. Instead of asking it to solve any problem, restrict the space to a tool that can help you then reach your goal faster without the statistical nature of LLMs.
  • mike_hearn 9mo ago
    Reproducibility is overrated and if you could wave a wand to make all papers reproducible tomorrow, it wouldn't fix the problem. It might even make it worse. https://blog.plan99.net/replication-studies-cant-fix-science-0e195234a280 https://blog.plan99.net/replication-studies-cant-fix-science...
    • biophysboy 9mo ago
      ? More samples reduces the variance of a statistic. Obviously it cannot identify systematic bias in a model, or establish causality, or make a "bad" question "good". Its not overrated though -- it would strengthen or weaken the case for many papers.
      • mike_hearn 9mo ago
        If you have a strong grip on exactly what it means, sure, but look at any HN thread on the topic of fraud in science. People think replication = validity because it's been described as the replication crisis for the last 15 years. And that's the best case! Funding replication studies in the current environment would just lead to lots of invalid papers being promoted as "fully replicated" and people would be fooled even harder than they already are. There's got to be a fix for the underlying quality issues before replication becomes the next best thing to do.
        • doctorpangloss 9mo ago
          while i agree that "reproducibility is overrated", i went ahead and read your medium post. my feedback to you is, my summary of that writing: "mike_hearn's take on policy-adjacent writing conducted by public health officials and published in journals that interacted with mike_hearn's valid and common but nonetheless subjective political dispute about COVID-19." i don't know how any of that writing generalizes to other parts of academic research. i mean, i know that you say it does, but i don't think it does. what exactly do you think most academic research institutions and the federal government spend money on? for example, wet lab research. you don't know anything about wet lab research. i think if you took a look at a typical e.g. basic science in immunology paper, built on top of mouse models, you would literally lose track of any of its meaning after the first paragraph, you would feed it into chatgpt, and you would struggle to understand the topic well enough to read another immunology paper, you would have an immense challenge talking about it with a researcher in the field. it would take weeks of reading. you have no medicine background, so you wouldn't understand the long horizon context of any of it. you wouldn't be able to "chatbot" your way into it, it would be a real education. so after all of that, would you still be able to write the conclusion you wrote in the medium post? i don't think so, because you would see that by many measures, you cannot generalize a froo-froo policy between "subjective political dispute about COVID-19" writing and wet lab research. you'd gain the wisdom to see that they're different things, and you lack the background, and you'd be much more narrow in what you'd say. it doesn't even have to be in the particulars, it's just about wisdom. that is my feedback. you are at once saying that there is greater wisdom to be had in the organization and conduct of research, and then, you go and make the highly low wisdom move to generalize about all academic research. which you are obviously doing not because it makes sense to, you're a smart guy. but because you have some unknown beef with "academics" that stems from anger about valid, common but nonetheless subjective political disputes about COVID-19.
  • vld_chk 9mo ago
    In my mental model, the fundamental problem of reproducibility is that scientists have very hard time to find a penny to fund such research. No one wants to grant “hey I need $1m and 2 years to validate the paper from last year which looks suspicious”. Until we can change how we fund science on the fundamental level; how we assign grants — it will be indeed very hard problem to deal with.
    • parpfish 9mo ago
      In theory, asking grad students and early career folks to run replications would be a great training tool. But the problem isn’t just funding, it’s time. Successfully running a replication doesn’t get you a publication to help your career.
      • iugtmkbdfil834 9mo ago
        Yeah, but doesn't publishing an easily falsifiable paper end one?
        • wizzwizz4 9mo ago
          Not in most fields, unless misconduct is evident. (And what constitutes "misconduct" is cultural: if you have enough influence in a community, you can exert that influence on exactly where that definitional border lies.) Being wrong is not, and should not be, a career-ending move.
          • iugtmkbdfil834 9mo ago
            If we are aiming for quality, then being wrong absolutely should be. I would argue that is how it works in real life anyway. What we quibble over is what is the appropriate cutoff.
            • rtkwe 9mo ago
              There's a big gulf between being wrong because you or a collaborator missed an uncontrolled confounding factor and falsifying or altering results. Science accepts that people sometimes make mistakes in their work because a) they can also be expected to miss something eventually and b) a lot of work is done by people in training in labs you're not directly in control of (collaborators). They already aim for quality and if you're consistently shown to be sloppy or incorrect when people try to use your work in their own. The final bit is a thing I think most people miss when they think about replication. A lot of papers don't get replicated directly but their measurements do when other researchers try to use that data to perform their own experiments, at least in the more physical sciences this gets tougher the more human centric the research is. You can't fake or be wrong for long when you're writing papers about the properties of compounds and molecules. Someone is going to come try to base some new idea off your data and find out you're wrong when their experiment doesn't work. (or spend months trying to figure out what's wrong and finally double check the original data).
  • benob 9mo ago
    Maybe it will also change the whole publication as evaluation of science.
  • lxgr 9mo ago
    > LLMs being able to put out plausible papers is just going to make it worse If correct form (LaTeX two-column formatting, quoting the right papers and authors of the year etc.) has been allowing otherwise reject-worthy papers to slip through peer review, academia arguably has bigger problems than LLMs.
    • LPisGood 9mo ago
      Correct form and relevant citations have been, for generations up to a couple of years ago, mighty strong signals that a work is good and done by a serious and reliable author. This is no longer the case and we are worse off for it.
  • CamperBob2 9mo ago
    I'd need to see the same scrutiny applied to pre-AI papers. If a field has a poor replication rate, meaning there's a good chance that a given published paper is just so much junk science, is that better or worse than letting AI hallucinate the data in the first place?
  • Sparkyte 9mo ago
    If there is one thing which scientific reports must require is not using AI to produce the documentation. They can be of the data but not of the source or anything else. AI is a tool, not a replacement for actual work.
  • lallysingh 9mo ago
    On the bright side, an LLM can really help set up a reproduction environment. Perhaps repro should become the basis of peer review?
    • mort96 9mo ago
      No, it can't. No LLM can purchase the equipment and chemicals and machinery you need to reproduce experiments, nor should you want it.
      • lallysingh 8mo ago
        I was still thinking of CS :/
  • colechristensen 9mo ago
    Reading the article, this is about CITATIONS which are trivially verifiable. This is just article publishers not doing the most basic verification failing to notice that the citations in the article don't exist. What this should trigger is a black mark for all of the authors and their institutions, both of which should receive significant reputational repercussions for publishing fake information. If they fake the easiest to verify information (does the cited work exist) what else are they faking?
  • godelski 9mo ago
    > to finally take reproducibility more seriously I've long argued for this, as reproduction is the cornerstone of science. There's a lot of potential ways to do this but one that I like is linking to the original work. Suppose you're looking at the OpenReview page and they have a link for "reproduction efforts" and with at minimum an annotation for confirmation or failure. This is incredibly helpful to the community as a whole. Reproduction failures can be incredibly helpful even when the original work has no fraud. In those cases a reprising failure reveals important information about the necessary conditions that the original work relies on. But honestly, we'll never get this until we drop the entire notion of "novel" or "impact" and "publish or perish". Novel is in the eye of the reviewer and the lower the reviewer's expertise the less novel a work seems (nothing is novel as a high enough level). Impact can almost never be determined a priori, and when it can you already have people chasing those directions because why the fuck would they not? But publish or perish is the biggest sin. It's one of those ideas that looks nice on paper, like you are meaningfully determining who is working hard and who is hardly working. But the truth is that you can't tell without being in the weeds. The real result is that this stifles creativity, novelty, and impact as it forces researchers to chase lower hanging fruit. Things you're certain will work and can get published. It creates a negative feedback loop as we compete: "X publishes 5 papers a year, why can't you?" I've heard these words even when X has far fewer citations (each of my work had "more impact"). Frankly, I believe fraud would dramatically reduce were researchers not risking job security. The fraud is incentivized by the cutthroat system where you're constantly trying to defend your job, your work, and your grants. They'll always be some fraud but (with a few exceptions) researchers aren't rockstar millionaires. It takes a lot of work to get to point where fraud even works, so there's a natural filter. I have the same advice as Mervin Kelly, former director of Bell Labs: How do you manage genius? You don't
  • andai 9mo ago
    I heard that most papers in a given field are already not adding any value. (Maybe it depends on the field though.) There seems to be a rule in every field that "99% of everything is crap." I guess AI adds a few more nines to the end of that. The gems are lost in a sea of slop. So I see useless output (e.g. crap on the app store) as having negative value, because it takes up time and space and energy that could have been spent on something good. My point with all this is that it's not a new problem. It's always been about curation. But curation doesn't scale. It already didn't. I don't know what the answer to that looks like.
  • anishrverma 9mo ago
    Yeah, spot on. If all we do is add more plausible sounding text on top of already fragile review and incentive structures, that really could make things worse rather than better Your second point is the important one. AI may be the thing that finally forces the community to take reproducibility, attribution, and verification seriously. That’s very much the motivation behind projects like Liberata, which try to shift publishing away from novelty first narratives and toward explicit credit for replication, verification, and followthrough. If that cultural shift happens, this moment might end up being a painful but necessary correction.
• qwertox 9mo ago
  It would be great if those scientists who use AI without disclosing it get fucked for life.
  • direwolf20 9mo ago
    "scientists" FYI. Making shit up isn't science.
  • yesitcan 9mo ago
    One fuck seems appropriate.
  • oofbey 9mo ago
    Harsh sentiment. Pretty soon every knowledge worker will use AI every day. Should people disclose spellcheckers powered by AI? Disclosing is not useful. Being careful in how you use it and checking work is what matters.
    • ambicapter 9mo ago
      > Should people disclose spellcheckers powered by AI? Thank you for that perfect example of a strawman argument! No, spellcheckers that use AI is not the main concern behind disclosing the use of AI in generating scientific papers, government reports, or any large block of nonfiction text that you paid for that is supposed to make to sense.
    • fisf 9mo ago
      People are accountable for the results they produce using AI. So a scientist is responsible for made up sources in their paper, which is plain fraud.
      • oofbey 9mo ago
        I completely agree. But “disclosing the use of AI” doesn’t solve that one bit.
        • 9mo ago
• jordanpg 9mo ago
  If these are so easy to identify, why not just incorporate some kind of screening into the early stages of peer review?
  • DetectDefect 9mo ago
    Because real work takes time and effort, and there is no real incentive for it here.
  • tossandthrow 9mo ago
    What makes you believe that are easy to identify?
    • emil-lp 9mo ago
      One could require DOIs for each reference. That's both realistic to achieve and easy to verify. Although then why not just cite existing papers for bogus reasons?
    • jordanpg 9mo ago
      Isn't that what GPTZero does?
• Tom1380 9mo ago
  No ETH Zurich, let's go
• direwolf20 9mo ago
  Wow! They're literally submitting references to papers by Firstname Lastname, John Doe and Jane Smith and nobody is noticing or punishing them.
  • emil-lp 9mo ago
    They might (I hope) still be punished after discovery.
  • an0malous 9mo ago
    It’s the way of the future
  • heliumtera 9mo ago
    [flagged]
    • azan_ 9mo ago
      Yes, it only led to all advancements in the history of humanity, what a joke!
      • heliumtera 9mo ago
        I am sure all advancements in the history of humanity was properly peer reviewed! Including coca cola and Linux!
        • azan_ 9mo ago
          If you wanted to attack peer review you should've attacked peer review, not entire science. And if "muh science" was some kind of code for peer review then it's not my fault that you are awful at articulating your point. It's still not clear what the hell do you mean.
          • heliumtera 9mo ago
            Science funding is strictly tied to publication on "respected journals" which is strictly tied to this gatekeeping horrendous process. I won't deny I am terrible at articulating my point, but I will maintain it. We can undeniably say that science, scientific institutions, scientific periodic journals, funding and any other financial instrument constructed to promote scientific advancements is rotten by design and should be abandoned immediately. This joke serves no good. "But what about muh scientific method?" Yeah yeah yeah, whoever thinks modern science honors logic and reason is part of the problem and has being played, and forever will be
• CGMthrowaway 9mo ago
  Which is worse: a) p-hacking and suppressing null results b) hallucinations c) falsifying data Would be cool to see an analysis of this
  • Proziam 9mo ago
    All 3 of these should be categorized as fraud, and punished criminally.
    • internetter 9mo ago
      criminally feels excessive?
      • Proziam 9mo ago
        If I steal hundreds of thousands of dollars (salary, plus research grants and other funds) and produce fake output, what do you think is appropriate? To me, it's no different than stealing a car or tricking an old lady into handing over her fidelity account. You are stealing, and society says stealing is a criminal act.
        • WarmWash 9mo ago
          We have a civil court system to handle stuff like this already.
          • Proziam 9mo ago
            Stealing more than a few thousand dollars is a felony, and felonies are handled in criminal court, not civil. EDIT - The threshold amount varies. Sometimes it's as low as a few hundred dollars. However, the point stands on its own, because there's no universe where the sum in question is in misdemeanor territory.
            • WarmWash 9mo ago
• dtartarotti 9mo ago
  It is very concerning that these hallucinations passed through peer review. It's not like peer review is a fool-proof method or anything, but the fact that reviewers did not check all references and noticed clearly bogus ones is alarming and could be a sign that the article authors weren't the only ones using LLMs in the process...
  • amanaplanacanal 9mo ago
    Is it common for peer reviewers to check references? Somehow I thought they mostly focused on whether the experiment looked reasonable and the conclusions followed.
    • emil-lp 9mo ago
      In journal publications it is, but without DOIs it's difficult. In conference publications, it's less common. Conference publications (like NEURips) is treated as announcement of results, not verified.
      • matusp 9mo ago
        Nobody in ML or AI is verifying all your references. Reviewers will point out if you miss a super related work, but that's it. This is especially true with the recent (last two decades?) inflation in citation counts. You regularly have papers with 50+ references for all kinds of claims and random semirelated work. The citation culture is really uninspiring.
• smallpipe 9mo ago
  Could you run a similar analysis for pre-2020 papers? It'd be interesting to know how prevalent making up sources was before LLMs.
  • tasuki 9mo ago
    Also, it'd be interesting how many pre-2020 papers their "AI detector" marks as AI-generated. I distrust LLMs somewhat, but I distrust AI detectors even more.
  • theptip 9mo ago
    Yeah, it’s kind of meaningless to attribute this to AI without measuring the base rate. It’s for sure plausible that it’s increasing, but I’m certain this kind of thing happened with humans too.
  • jcmp 8mo ago
    at the end of the article they made a clear distinction between flawed and hallucinated cititations. I feels its hard to argue that through a mistake a hallucinated citation emerge: > Real Citation Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521:436-444, 2015. Flawed Citation Y. LeCun, Y. Bengio, and Geoff Hinton. Deep leaning. nature, 521(7553):436-444, 2015. Hallucinated Citation Samuel LeCun Jackson. Deep learning. Science & Nature: 23-45, 2021.
• bonsai_spool 9mo ago
  This suggests that nobody was screening this papers in the first place—so is it actually significant that people are using LLMs in a setting without meaningful oversight? These clearly aren't being peer-reviewed, so there's no natural check on LLM usage (which is different than what we see in work published in journals).
  • emil-lp 9mo ago
    As one who reviews 20+ papers per year, we don't have time to verify each reference. We verify: is the stuff correct, and is it worthy of publication (in the given venue) given that it is correct. There is still some trust in the authors to not submit made-up-stuff, albeit it is diminishing.
    • paulmist 9mo ago
      I'm surprised the conference doesn't provide tooling to validate all references automatically.
      • Sharlin 9mo ago
        How would you do that? Even in cases where there's a standard format, a DOI on every reference, and some giant online library of publication metadata, including everything that only exists in dead tree format, that just lets you check whether the cited work exists, not whether it's actually a relevant thing to cite in the context.
    • its_ethan 9mo ago
      Sorry, but if someone makes a claim and cites a reference, how do you verify "is the stuff correct" without checking that reference?
      • emil-lp 9mo ago
        Those are typically things you are familiar with or can easily check. Fake references are more common in the introduction where you list relevant material to strengthen your results. They often don't change the validity of the claim, but the potential impact or value.
• deleted 9mo ago
  [deleted]
• geremiiah 9mo ago
  A lot of research in AI/ML seems to me to be "fake it and never make it". Literally it's all about optics, posturing, connections, publicity. Lots of bullshit and little substance. This was true before AI slop, too. But the fact that AI slop can make it pass the review really showcases how much a paper's acceptance hinges on things, other than the substance and results of the paper. I even know PIs who got fame and funding based on some research direction that supposedly is going to be revolutionary. Except all they had were preliminary results that from one angle, if you squint, you can envision some good result. But then the result never comes. That's why I say, "fake it, and never make it".
• gcr 9mo ago
  I was getting completely AI-generated reviews for a WACV publication back in 2024. The area chairs are so overworked that authors don't have much recourse, which sucks but is also really hard to handle unless more volunteers step up to the bat to help organize the conference. (If you're qualified to review papers, please email the program chair of your favorite conference and let them know -- they really need the help!) As for my review, the review form has a textbox for a summary, a textbox for strengths, a textbox for weaknesses, and a textbox for overall thoughts. The review I received included one complete set of summary/strengths/weaknesses/closing thoughts in the summary text box, another distinct set of summary/strengths/weaknesses/closing thoughts in the strengths, another complete and distinct review in the weaknesses, and a fourth complete review in the closing thoughts. Each of these four reviews were slightly different and contradicted each other. The reviewer put my paper down as a weak reject, but also said "the pros greatly outweigh the cons." They listed "innovative use of synthetic data" as a strength, and "reliance on synthetic data" as a weakness.
  • cubefox 9mo ago
    Wow...
  • alansaber 8mo ago
    This is 100% negligence.
• gcr 9mo ago
  NeurIPS leadership doesn’t think hallucinated references are necessarily disqualifying; see the full article from Fortune for a statement from them: https://archive.ph/yizHN https://archive.ph/yizHN > When reached for comment, the NeurIPS board shared the following statement: “The usage of LLMs in papers at AI conferences is rapidly evolving, and NeurIPS is actively monitoring developments. In previous years, we piloted policies regarding the use of LLMs, and in 2025, reviewers were instructed to flag hallucinations. Regarding the findings of this specific work, we emphasize that significantly more effort is required to determine the implications. Even if 1.1% of the papers have one or more incorrect references due to the use of LLMs, the content of the papers themselves are not necessarily invalidated. For example, authors may have given an LLM a partial description of a citation and asked the LLM to produce bibtex (a formatted reference). As always, NeurIPS is committed to evolving the review and authorship process to best ensure scientific rigor and to identify ways that LLMs can be used to enhance author and reviewer capabilities.”
  • Analemma_ 9mo ago
    Kinda gives the whole game away, doesn’t it? “It doesn’t actually matter if the citations are hallucinated.” In fairness, NeurIPS is just saying out loud what everyone already knows. Most citations in published science are useless junk: it’s either mutual back-scratching to juice h-index, or it’s the embedded and pointless practice of overcitation, like “Human beings need clean water to survive (Franz, 2002)”. Really, hallucinated citations are just forcing a reckoning which has been overdue for a while now.
    • jacquesm 9mo ago
      There should be a way to drop any kind of circular citation ring from the indexes.
      • gcr 9mo ago
        It's tough because some great citations are hard to find/procure still. I sometimes refer to papers that aren't on the Internet (eg. old wonderful books / journals).
• Molitor5901 9mo ago
  AI might just extinguish the entire paradigm of publish or perish. The sheer volume of papers makes it nearly impossible to properly decide which papers have merit, which are non-replicate and suspect, and which are just a desperate rush to publish. The entire practice needs to end.
  • shermantanktop 9mo ago
    But how could we possibly evaluate faculty and researcher quality without counting widgets on an assembly line? /s It’s a problem. The previous regime prior to publishing-mania was essentially a clubby game of reputation amongst peers based on cocktail party socialization. The publication metrics came out of the harder sciences, I believe, and then spread to the softest of humanities. It was always easy to game a bit if you wanted to try, but now it’s trivial to defeat.
  • SJC_Hacker 9mo ago
    Its not publish or perish so much as get grant money or perish. Publishing is just the way to get grants. A PI explained it to me once, something like this Idea(s) -> Grant -> Experiments -> Data -> Paper(s) -> Publication(s) -> Idea(s) -> Grant(s) Thats the current cycle ... remove any step and its a dead end
• deleted 9mo ago
  [deleted]
• jsksdkldld 9mo ago
  [dead]
• TAULIC15 9mo ago
  [flagged]
• armcat 9mo ago
  This is awful but hardly surprising. Someone mentioned reproducible code with the papers - but there is a high likelihood of the code being partially or fully AI generated as well. I.e. AI generated hypothesis -> AI produces code to implement and execute the hypothesis -> AI generates paper based on the hypothesis and the code. Also: there were 15 000 submissions that were rejected at NeurIPS; it would be very interesting to see what % of those rejected were partially or fully AI generated/hallucinated. Are the ratios comperable?
  • blackbear_ 9mo ago
    Whether the code is AI generated or not is not important, what matters is that it really works. Sharing code enables others to validate the method on a different dataset. Even before LLMs came around there were lots of methods that looked good on paper but turned out not to work outside of accepted benchmarks
• depressionalt 9mo ago
  This is nice and all, but what repercussion does GPTZero get when their bullshit AI detection hallucinates a student using AI? And when that student receives academic discipline because of it? Many such cases of this. More than 100! They claim to have custom detection for GPT-5, Gemini, and Claude. They're making that up!
  • freedomben 9mo ago
    Indeed. My son has been accused by bullshit AI detection as having used AI, and it has devastated his work quality. After being "disciplined" for using AI (when he didn't), he now intentionally tries to "dumb down" his writing so that it doesn't sound so much like AI. The result is he writes much worse. What a shitty, shitty outcome. I've even found myself leaving typos and things in (even on sites like HN) because if you write too well, inevitably some comment replier will call you out as being an LLM even when you aren't. I'm as annoyed by the LLM posts as everybody else, but the answer surely is not to dumb us down into Idiocracy.
    • Sharlin 9mo ago
      It's almost as if this whole LLM stuff wasn't a net benefit to the society after all.
    • Der_Einzige 9mo ago
      Stop using em dashes, the fancy quotes that can’t be easily typed. Stop using overused words like certainly and delve. Stop using LLM template slop like “it’s not X, it’s Y”. Stop always doing lists of 3s. We know you didn’t use to use so many emojis or bolded text. Also, AI really fking hates the exclamation mark so that’s a great proof of humanity! Most people getting flagged are getting flagged because they actually used AI and couldn’t even be bothered to manually deslop it. People who are too lazy to put even a tiny bit of human intentionality into their work deserve it.
      • freedomben 8mo ago
        That's all good advice, but it's not enough. He never uses em dashes or emojis in papers, and in the past when using exclamation marks he had teachers say, "don't use these in academic papers, they're not appropriate." Also mac OS loves to use the fancy quotes by default so when he's writing on a Mac, it's a pain in the ass to use regular quotes. It seems absurd to me that you'd have to jump through that hoop anyway just so it doesn't look like AI. We've had teachers show us the screenshot output from their AI tool and it flags on things like "vocabulary word unusual for grade level." In my early 20s when I was dating my now-wife, she had a great vocabulary and I admired her for it, so I spent a lot of effort improving my vocabulary (well worth it by the way). When my son was born I intentionally used "big words" all the time with him (and explained what they meant when he didn't know) in the hopes that he would have a naturally large vocabulary when he got older. It worked very well. He routinely uses words even in conversation that even his teachers don't know. He writes even better than he speaks. But now being a statistical outlier is punishing him. It flags plenty of other things like direct quotes (which he puts in quotation marks as he should) and includes it in the "score", so a quote heavy paper will sometimes show something like "65% produced by AI". He uses Google Docs so we can literally go through the whole history and see him writing the paper through time. > Most people getting flagged are getting flagged because they actually used AI and couldn’t even be bothered to manually deslop it. I'm sure that's true, but it doesn't excuse people using an automated tool that they don't understand and messing with other people's lives because of it. Just like when some cloud provider decides that your workload looks too much like crypto mining or something so AI auto-bans your account and shuts off your stuff.
• GrowingSideways 9mo ago
  [dead]
• theptip 9mo ago
  This is mostly an ad for their product. But I bet you can get pretty good results with a Claude Code agent using a couple simple skills. Should be extremely easy for AI to successfully detect hallucinated references as they are semi-structured data with an easily verifiable ground truth.
• leggerss 9mo ago
  I don't understand: why aren't there automated tools to verify citations' existence? The data for a citation has a structured styling (APA, MLA, Chicago) and paper metadata is available via e.g. a web search, even if the paper contents are not I guess GPTZero has such a tool. I'm confused why it isn't used more widely by paper authors and reviewers
  • gh02t 9mo ago
    Citations are too open ended and prone to variation, and legitimate minor mistskes that wouldn't bother a human verifier but would break automated tools to easily verify in their current form. DOI was supposed to solve some of the literal mechanical variation of the existence of a source, but journal paywalls and limited adoption mean that is not a universal solution. Plus DOI still doesn't easily verify the factual accuracy of a citation, like "does the source say what the citation says it does," which is the most important part. In my experience you will see considerable variation in citation formats, even in journals that strictly define it and require using BibTex. And lots of journals leave their citation format rules very vague. Its a problem that runs deep.
    • leggerss 9mo ago
      Thanks for the thoughtful reply!
  • eichin 9mo ago
    Looks like GPTZero Source Finder was only released a year ago - if anything, I'm surprised slop-writers aren't using it preemptively, since they're "ahead of the curve" relative to reviewers on this sort of thing...
• yepyeaisntityea 9mo ago
  No surprises. Machine learning has, at least since 2012, been the go-to field for scammers and grifters. Machine learning, and technology in general, is basically a few real ideas, a small number of honest hard workers, and then millions of fad chasers and scammers.
• mt_ 9mo ago
  It would be ironic if the very detection of hallucinations contained hallucinations of its own.
• deleted 9mo ago
  [deleted]
• pandemic_region 9mo ago
  What if they would only accept handwritten papers? Basically the current system is beyond repair, so may as well go back to receiving 20 decent papers instead of 20k hallucinated ones.
• deleted 9mo ago
  [deleted]
• doug_durham 9mo ago
  Getting papers published is now more about embellishing your CV versus a sincere desire to present new research. I see this everywhere at every level. Getting a paper published anywhere is a checkbox in completing your resume. As an industry we need to stop taking this into consideration when reviewing candidates or deciding pay. In some sense it has become an anti-signal.
  • londons_explore 9mo ago
    I'd like to see a financial approach to deciding pay by giving researchers a small and perhaps nonlinear or time bounded share of any profits that arise from their research. Then peoples CV's could say "My inventions have led to $1M in licensing revenue" rather than "I presented a useless idea at a decent conference because I managed to make it sound exciting enough to get accepted".
    • direwolf20 9mo ago
      That's what patents do.
    • autoexec 9mo ago
      A lot of good research isn't ever going to make anyone a single dime, but that doesn't mean it doesn't matter.
  • autoexec 9mo ago
    It'd be nice if there were a dedicated journal for papers published just because you have to publish for your CV or to get your degree. That way people can keep publishing for the sake of publishing, but you could see at a glance what the deal was.
  • biophysboy 9mo ago
    I think its fairer to say that perverse incentives have added more noise to the publishing signal. Publishing 0 times is not better than 100 times, even if 90% of those are Nth author formality/politeness citations.
• nerdjon 9mo ago
  The downstream effects of this are extremely concerning. We have already seen the damage caused by human written research that was later retracted like the “research” on vaccines causing autism. As we get more and more papers that may be citing information that was originally hallucinated in the first place we have a major reliability issue here. What is worse is people that did not use AI in the first place will be caught in the crosshairs since they will be referencing incorrect information. There needs to be a serious amount of education done on what these tools can and cannot do and importantly where they fail. Too many people see these tools as magic since that is what the big companies are pushing them as. Other than that we need to put in actual repercussions for publishing work created by an LLM without validating it (or just say you can’t in the first place but I guess that ship has sailed) or it will just keep happening. We can’t just ignore it and hope it won’t be a problem. And yes, humans can make mistakes too. The difference is accountability and the ability to actually be unsure about something so you question yourself to validate.
• MORPHOICES 9mo ago
  [dead]
• ctoth 9mo ago
  How you know it's really real is that they clearly tell the FPR, and compare against a pre-llm baseline. But I saw it in Apple News, so MISSION ACCOMPLISHED!
• yobbo 9mo ago
  As long as these sorts of papers serve more important purposes for the careers of the authors than anything related to science or discovery of knowledge, then of course this happens and continues. The best possible outcome is that these two purposes are disconflated, with follow-on consequences for the conferences and journals.
• poulpy123 9mo ago
  All papers proved to have used a LLM beyond writing improvement should be automatically retracted
• brador 9mo ago
  The problem isn’t scale. The problem is consequences (lack of). Doing this should get you barred from research. It won’t.
  • uhfraid 9mo ago
    Scale IS a problem, just not the only one. Consequences are the inevitable solution. Accountability starting with authors, followed by organizations/institutions. Warning for first offense, ban after
• CrzyLngPwd 9mo ago
  This is not the AI future we dreamed of, or feared.
• nospice 9mo ago
  We've been talking about a "crisis of reproducibility" for years and the incentive to crank out high volumes of low-quality research. We now have a tool that brings down the cost of producing plausibly-looking research down to zero. So of course we're going to see that tool abused on a galactic scale. But here's the thing: let's say you're an university or a research institution that wants to curtail it. You catch someone producing LLM slop, and you confirm it by analyzing their work and conducting internal interviews. You fire them. The fired researcher goes public saying that they were doing nothing of the sort and that this is a witch hunt. Their blog post makes it to the front page of HN, garnering tons of sympathy and prompting many angry calls to their ex-employer. It gets picked up by some mainstream outlets, too. It happened a bunch of times. In contrast, there are basically no consequences to institutions that let it slide. No one is angrily calling the employers of the authors of these 100 NeurIPS papers, right? If anything, there's the plausible deniability of "oh, I only asked ChatGPT to reformat the citations, the rest of the paper is 100% legit, my bad".
• meindnoch 9mo ago
  Jamie, bring up their nationalities.
• neom 9mo ago
  I wrote before about my embarrassing time with ChatGPT during a period (https://news.ycombinator.com/item?id=44767601 https://news.ycombinator.com/item?id=44767601) - I decided to go back through those old 4o chats with 5.2 pro extended thinking, the reply was pretty funny because it first slightly ridiculed me, heh - but what it showed was: basically I would say "what 5 research papers from any area of science talk to these ideas" and it would find 1 and invent 4 if it didn't know 4 others, and not tell me, and then I'd keep working with it and it would invent what it thought might be in the papers long the way, making up new papers in it's own work to cite to make it's own work valid, lol. Anyway, I'm a moron, sure, and no real harm came of it for me, just still slightly shook I let that happen to me.
  • Shocka1 9mo ago
    Just to clarify, you didn't actually look up the publications it was citing? For example, you just stayed in ChatGPT web and used the resources it provided there? Not ridiculing you of course, but am just curious. The last paper I wrote a couple months back I had GPT search out the publications for me, but I would always open a new tab and retrieve the actual publication.
    • neom 9mo ago
      I didn't because I wasn't really doing anything serious to my mind, I think? basically felt like watching an episode of pbs spacetime, I think the difference is it's more like playing a video game while thinking you're watching an episode of spacetime, if that makes sense? I don't use chatgpt for me real work that much, and I'm not a scientist, so it was for me just mucking around, it pushed me slightly over a line into "I was just playing but now this seems real", it didn't occur to me to go back through and check all the papers, I guess because quite a lot of chatting had happened since then and, I dunno, I just didn't think to? Not sure that makes much sense. This was also over a year ago, during the time they had the gpt4o sycophancy mode that made the news, and it wasn't backed by webserch, so I took for granted what was in it's training data. No good excuse I'm afraid. tldr: poor critical thinking skills on my part there! :)
• londons_explore 9mo ago
  And this is the tip of the iceberg, because these are the easy to check/validate things. I'm sure plenty of more nuanced facts are also entirely without basis.
• techIA 9mo ago
  They will turn it into a party drug.
• trash_cat 9mo ago
  Clearly there is some demand for those papers, and research, to exist. Good opportunity to fill the gaps.
• captainbland 9mo ago
  What's wild is so many of these are from prestigious universities. MIT, Princeton, Oxford and Cambridge are all on there. It must be a terrible time to be an academic who's getting outcompeted by this slop because somebody from an institution with a better name submitted it.
  • cflewis 9mo ago
    I'm going to be charitable and say that the papers from prestigious universities were honest mistakes rather than paper mill university fabrications. One thing that has bothered me for a very long time is that computer science (and I assume other scientific fields) has long since decided that English is the lingua franca, and if you don't speak it you can't be part of it. Can you imagine if being told that you could only do your research if you were able to write technical papers in a language you didn't speak, maybe even using glyphs you didn't know? It's crazy when you think about it even a little bit, but we ask it of so many. Let's not include the fact that 90% of the English-speaking population couldn't crank out a paper to the required vocabulary level anyway. A very legitimate, not trying to cheat, use for LLMs is translation. While it would be an extremely broad and dangerous brush to paint with, I wonder if there is a correlation between English-as-a-Second (or even third)-Language authors and the hallucinations. That would indicate that they were trying to use LLMs to help craft the paper to the expected writing level. The only problem being that it sometimes mangles citations, and if you've done good work and got 25+ citations, it's easy for those errors to slip through.
    • zipy124 8mo ago
      I can't speak for the American universities, but remember there is no entrance exam for UK PhDs, you just require a 2:1 or 1st class bachelor's degree/masters (going straight without a masters is becoming more common) usually, which is trivial to obtain. The hard part is usually getting funding, but if you provide your own funding you can go to any university you want. They are only really hard universities to get into for a bachelors, not for masters or PhD where you are more of a money/labour source than anything else.
      • 8mo ago
• teekert 9mo ago
  We have the h score and such, can we have something similar that goes down when you pull stunts like these? Preferably link it to people’s orcid ids.
• dev_l1x_be 9mo ago
  I am wondering if we are going to reach hallucination collapse sooner than we reach AGI.
• Nevermark 9mo ago
  With regard to confabulating (hallucinating) sources, or anything else, it is worth noting this is a first class training requirement imposed on models. Not models simply picking up the habit from humans. When training a student, normally we expect a lack of knowledge early, and reward self-awareness, self-evaluation and self-disclosure of that. But the very first epoch of a model training run, when the model has all the ignorance of a dropped plate of spaghetti, we optimize the network to respond to information, as anything from a typical human to an expert, without any base of understanding. So the training practice for models is inherently extreme enforced “fake it until you make it”, to a degree far beyond any human context or culture. (Regardless, humans need to verify, not to mention read, the sources they site. But it will be nice when models can be trusted to accurately access what they know/don’t-know too.)
• rabbitlord 9mo ago
  You will find out that Top CS conference is never scientific, if you really go to their GitHub and run their code.
• ctoth 9mo ago
  The innumeracy is load-bearing for the entire media ecosystem. If readers could do basic proportional reasoning, half of health journalism and most tech panic coverage would collapse overnight. GPTZero of course knows this. "100 hallucinations across 53 papers at prestigious conference" hits different than "0.07% of citations had issues, compared to unknown baseline, in papers whose actual findings remain valid."
  • MeetingsBrowser 9mo ago
    I’m not sure that’s fair in this context. In the past, a single paper with questionable or falsified results at a top tier conference was big news. Something that casts doubt on the validity of 53 papers at a top AI conference is at least notable. > whose actual findings remain valid Remain valid according to who? The same group that missed hundreds of hallucinated citations?
    • ctoth 9mo ago
      Which of these papers had falsified results and not bad citations? What is the base rate of bad citations pre-AI? And finally yes. Peer review does not mean clicking every link in the footnotes to make sure the original paper didn't mislink, though I'm sure after this bruhaha this too will be automated.
      • MeetingsBrowser 9mo ago
        > Peer review does not mean clicking every link in the footnotes It wasn't just broken links, but citing authors like "lastname, firstname" and made up titles. I have done peer reviews for a (non-AI) CS conference and did at least skim the citations. For papers related to my domain, I was familiar with most of the citations already, and looked into any that looked odd. Being familiar with the state of the art is, in theory, what qualifies you to do peer reviews.
  • nonethewiser 9mo ago
    > "0.07% of citations had issues Nope, you are getting this part wrong. On purpose or by accident? Because it's pretty clear if you read the article they are not counting all citations that simply had issues. See "Defining Hallucinated Citations".
• pacbard 9mo ago
  The ironic part about these hallucinations is that a research paper includes a literature review because the goal of the research is to be in dialogue with prior work, to show a gap in the existing literature, and to further the knowledge that this prior work has built. By using an LLM to fabricate citations, authors are moving away from this noble pursuit of knowledge built on the "shoulders of giants" and show that behind the curtain output volume is what really matters in modern US research communities.
  • andy_xor_andrew 9mo ago
    I guess that makes this "standing on the shoulders of fabrications"
    • stogot 9mo ago
      Fabrication should be immediate academic ban for life
      • physPop 9mo ago
        Paper mills existed long before LLMs
    • einpoklum 9mo ago
      Standing on the shoulders of imaginary giants then :-\
  • AlienRobot 9mo ago
    That's going to be the philosophical question of our times: do LLMs generate slop out of nowhere or does it simply amplify the slop machinery that was already there?
• rfrey 9mo ago
  There's a lot of good arguments in this thread about incentives: extremely convincing about why current incentives lead to exactly this behaviour, and also why creating better incentives is a very hard problem. If we grant that good carrots are hard to grow, what's the argument against leaning into the stick? Change university policies and processes so that getting caught fabricating data or submitting a paper with LLM hallucinations is a career ending event. Tip the expected value of unethical behaviours in favour of avoiding them. Maybe we can't change the odds of getting caught but we certainly can change the impact. This would not be easy, but maybe it's more tractable than changing positive incentives.
  • currymj 9mo ago
    the harsher the punishment, the more due process required. i don't think there are any AI detection tools that are sufficiently reliable that I would feel comfortable expelling a student or ending someone's career based on their output. for example, we can all see what's going on with these papers (and it appears to be even worse among ICLR submissions). but it is possible to make an honest mistake with your BibTeX. Or to use AI for grammar editing, which is widely accepted, and have it accidentally modify a data point or citation. There are many innocent mistakes which also count as plausible excuses. in some cases further investigation maybe can reveal a smoking gun like fabricated data, which is academic misconduct whether done by hand or because an AI generated the LaTeX tables. punishments should be harsher for this than they are.
    • rfrey 9mo ago
      Fabricated citations seem to be a popular and non ambiguous way for AI to sabotage science.
• godelski 9mo ago
  Given that many of these detections are being made from references, I don't understand why we're not using automatic citation checkers. Just ask authors to submit their bib file so we don't need to do OCR on the PDF. Flag the unknown citations and ask reviewers to verify their existence. Then contact authors and ban if they can't produce the cited work. This is low hanging fruit here! Detecting slop where the authors vet citations is much harder. The big problem with all the review rules is they have no teeth. If it were up to me we'd review in the open, or at least like ICLR. Publish the list of known bad actors and let is look at the network. The current system is too protective of egregious errors like plagiarism. Authors can get detected in one conference, pull, and submit to another, rolling the dice. We can't allow that to happen and we should discourage people from associating with these conartists. AI is certainly a problem in the world of science review, but it's far from the only one and I'm not even convinced it's the biggest. The biggest is just that reviewers are lazy and/or not qualified to review the works they're assigned. It takes at least an hour to properly review a paper in your niche, much more when it's outside. We're over worked as is, with 5+ works to review, not to mention all the time we got to spend reworking our own works that were rejected due to the slot machine. We could do much better if we dropped this notion of conference/journal prestige and focused on the quality of the works and reviews. Addressing those issues also addresses the AI issues because, frankly, *it doesn't matter if the whole work was done by AI, what matters is if the work is real.*
• abktowa 9mo ago
  Implicitly this makes sense but the amount cited in this article is still hard for me to grasp. Wow.
• gtirloni 9mo ago
  Why focus on hallucinations/LLMs and not on the authors? There are rules for submitting papers. If I drop a loaded gun and it fires, killing someone, we don't go after the gun's manufacturer in most cases.
  • phyzome 9mo ago
    This isn't directly to your point, but: A civil suit for such an incident would generally name both the weapon owner (for negligence, etc.) and the manufacturer (for dangerous design).
  • Der_Einzige 9mo ago
    Actually, if you’re the US navy, you DO go after the manufacturer! Go look up the P320 pistol and the tons of accidental discharges that’s it’s caused. https://stateline.org/2025/03/10/more-law-enforcement-agencies-reconsider-use-of-popular-sig-sauer-p320-handgun/ https://stateline.org/2025/03/10/more-law-enforcement-agenci...
    • gtirloni 8mo ago
      Thanks. But's not actually the point I'm trying to make. What I'm saying is that the authors have a responsibility, whether they wrote the papers themselves, asked an AI to write and didn't read it thoroughly, or asked their grandparents while on LSD to write it... it all comes back to whoever put their names on the paper and submitted it. I think AI is a red herring here.
• not2b 9mo ago
  This is going to be a huge problem for conferences. While journals have a longer time to get things right, as a conference reviewer (for IEEE conferences) I was often asked to review 20+ papers in a short time to determine who gets a full paper, who gets to present just a poster, etc. There was normally a second round, but often these would just look at submissions near the cutoff margin in the rankings. Obvious slop can be quickly rejected, but it will be easier to sneak things in.
  • cyber_kinetist 9mo ago
    AI conferences are already fucked. Students who are doing their Master's degrees are reviewing those top-tier papers, since there are just too many submissions for existing reviewers.
• currymj 9mo ago
  Especially for your first NeurIPS paper as a PhD student, getting one published is extremely lucrative. Most big tech PhD intern job postings have NeurIPS/ICML/ICLR/etc. first author paper as a de facto requirement to be considered. It's like getting your SAG card. If you get one of these internships, it effectively doubles or triples your salary that year right away. You will make more in that summer than your PhD stipend. Plus you can now apply in future summers and the jobs will be easier to get. And it sets your career on a good path. A conservative estimate of the discounted cash value of a student's first NeurIPS paper would certainly be five figures. It's potentially much higher depending on how you think about it, considering potential path dependent impacts on future career opportunities. We should not be surprised to see cheating. Nonetheless, it's really bad for science that these attempts get through. I also expect some people did make legitimate mistakes letting AI touch their .bib.
  • Der_Einzige 9mo ago
    This is 100% true, if anything you’re massively undercounting the value of publications. Most industry AI jobs that aren’t research based know that NeurIPS publications are a huge deal. Many of the managers don’t even know what a workshop is (so you can pass off NeurIPS workshop work as just “NeurIPS”) A single first author main conference work effectively allows a non Ph.D holder to be treated like they have a Ph.d (be qualified for professional researcher jobs). This means that a decent engineer with 1 NeurIPS publication is easily worth 300K+ YOY assuming US citizen. Even if all they have is a BS ;) And if you are lucky to get a spotlight or an oral, that’s probably worth closer to 7 figures…
• alcasa 9mo ago
  Didn't know the L in Samuel L Jackson was for LeCun.
• j2kun 9mo ago
  I spot-checked one of the flagged papers (from Google, co-authored by a colleague of mine) The paper was https://openreview.net/forum?id=0ZnXGzLcOg https://openreview.net/forum?id=0ZnXGzLcOg and the problem flagged was "Two authors are omitted and one (Kyle Richardson) is added. This paper was published at ICLR 2024." I.e., for one cited paper, the author list was off and the venue was wrong. And this citation was mentioned in the background section of the paper, and not fundamental to the validity of the paper. So the citation was not fabricated, but it was incorrectly attributed (perhaps via use of an AI autocomplete). I think there are some egregious papers in their dataset, and this error does make me pause to wonder how much of the rest of the paper used AI assistance. That said, the "single error" papers in the dataset seem similar to the one I checked: relatively harmless and minor errors (which would be immediately caught by a DOI checker), and so I have to assume some of these were included in the dataset mainly to amplify the author's product pitch. It succeeded.
  • davidguetta 9mo ago
    Yeah even the entire "Jane Doe / Jame Smith" my first thought is that it could have been a latex default value There was dumb stuff like this before the GPT era, it's far from convincing
    • ls612 9mo ago
      There are people who just want to punish academics for the sake of punishing academics. Look at all the people downthread salivating over blacklisting or even criminally charging people who make errors like this with felony fraud. Its the perfect brew of anti AI and anti academia sentiment. Also, in my field (economics), by far the biggest source of finding old papers invalid (or less valid, most papers state multiple results) is good old fashioned coding bugs. I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time.
      • worik 9mo ago
        > I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time. My hand is up. I do not believe in gaol, but I do agree with the sentiment.
• cyber_kinetist 9mo ago
  It has been several years since the reviewing process for top AI conferences have been broken as hell, due to having too many submissions and only a few reviewers (up to the point that Masters students are reviewing the papers). It was only a matter of time before these conferences will be filled with AI-written papers.
• mat_b 9mo ago
  > we discovered 100s of hallucinated citations missed by the 3+ reviewers who evaluated each paper. This says just as much about the humans involved.
  • mkehrt 9mo ago
    Well for one, it's definitely not the responsibility of the reviewers to check that all the citations exist. That would be insane.
• Prof_Sigmund 9mo ago
  The authors talk about "a model's ability to align with human decisions" as a matter of the past. The omission in the paper is RLHF (Reinforcement Learning from Human Feedback). All these companies are "teaching machines to predict the preferences of people who click 'Accept All Cookies' without reading," by using low-paid human evaluators — “AI teachers.” If we go back to Google, before its transformation into an AI powerhouse — as it gutted its own SERPs, shoving traditional blue links below AI-generated overlords that synthesize answers from the web’s underbelly, often leaving publishers starving for clicks in a zero-click apocalypse — what was happening? The same kind of human “evaluators” were ranking pages. Pushing garbage forward. The same thing is happening with AI. As much as the human "evaluators" trained search engines to elevate clickbait, the very same humans now train large language models to mimic the judgment of those very same evaluators. A feedback loop of mediocrity — supervised by the... well, not the best among us. The machines still, as Stephen Wolfram wrote, for any given sequence, use the same probability method (e.g., “The cat sat on the...”), in which the model doesn’t just pick one word. It calculates a probability score for every single word in its vast vocabulary (e.g., “mat” = 40% chance, “floor” = 15%, “car” = 0.01%), and voilà! — you have a “creative” text: one of a gazillion mindlessly produced, soulless, garbage “vile bile” sludge emissions that pollute our collective brains and render us a bunch of idiots, ready to swallow any corporate poison sent our way. In my opinion, even worse: the corporates are pushing toward “safety” (likely from lawsuits), and the AI systems are trained to sell, soothe, and please — not to think, or enhance our collective experience.
• Lerc 9mo ago
  So the headline says >GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers And I'm left wondering if they mean 100 papers or 100 hallucinations The subheading says >GPTZero's analysis 4841 papers accepted by NeurIPS 2025 show there are at least 100 with confirmed hallucinations Which accidentally a word, but seems to clarify that they do legitimately mean 100 papers. A later heading says >Table of 100 Hallucinated Citations in Published Across 53 NeurIPS Papers Which suggests either the opposite, or that they chose a subset of their findings to point out a coincidentally similar number of incidents. How many papers did they find hallucinations in? I'm still not certain. Is it 100, 53 or some other number altogether? Does their quality of scrutiny match the quality of their communication. If they did in-fact find 100 Hallucinations in 53 papers, would the inconsistency against their claim of "papers accepted by NeurIPS 2025 show there are at least 100 with confirmed hallucinations" meet their own bar for a hallucination?
  • j2kun 9mo ago
    They counted multiple hallucinations in a single paper toward the 100, and explicitly call out one paper with 13 incorrect citations that are claimed (reasonably, IMO) to be hallucinated.
    • Lerc 8mo ago
      So you are saying their claim of >GPTZero's analysis 4841 papers accepted by NeurIPS 2025 show there are at least 100 with confirmed hallucinations Is not true. [Edit - that sounds a bit harsh making it seem like you are accusing them, it's more that this is a logical conclusion of your(imo reasonable) interpretation.
      • j2kun 8mo ago
        I think it is true and intentionally vague for marketing purposes. And FWIW, I support the effort writ large
• OptionX 9mo ago
  The old create the problem and sell the solution shtick.
  • gold23 9mo ago
    *expose the problem
• gowld 9mo ago
  Why does "Robust Label Proportions Learning" have a "Scan" link, while all the others have a "Sources" link? Was this web page generated by AI?
• gowld 9mo ago
  "100 Hallucinated Citations in Published Across 53 NeurIPS Papers" No one cares about citations. They are hallucinated because they are required to be present for political reasons, even though they have no relevance.
• gowld 9mo ago
  I searched Google for one of the hallucinations: [N. Flammarion. Chen "sam generalizes"] AI Overview: Based on the research, [Chen and N. Flammarion (2022)](https://gptzero.me/news/neurips/ https://gptzero.me/news/neurips/) investigate why Sharpness-Aware Minimization (SAM) generalizes better than SGD, focusing on optimization perspectives The link is a link to the OP web page calling the "research" a hallucination.
• djoldman 9mo ago
  I would love to see this analysis run on pre-GPT era papers.
• waldarbeiter 9mo ago
  My website of choice whenever I have to deal with references is dblp [1]. In my opinion more reliable than Google scholar in creating correct BibTeX. Also when searching for a paper you clearly see where it has been published or if it is only on arxiv. [1] https://dblp.org/ https://dblp.org/
• sdellis 9mo ago
  This is an advertisement disguised as a "report".
  • gold23 9mo ago
    Nonetheless this investigation is important to anyone with a stronger desire to preserve intellectual honesty than disdain for a company trying to expand their offering.
  • snihalani 9mo ago
    it's painful that HN has become that
  • scoper31134 8mo ago
    a good advertisement
• SaaSasaurus 9mo ago
  I'm surprised it's only 100, honestly. Also feels a little sensationalized... Before AI I wonder how many "hallucinations" were in human-written papers. Is there any data on this?
  • rovr138 9mo ago
    These are 100, already reviewed papers and accepted papers, by researchers in their areas of expertise. Usually PhD's and Professors.... They judge. These are not all the submissions that they received. The review process can be... brutal for some people (depending on the quality of their submission)
• abalone 9mo ago
  At least in one case the authors claimed to use ChatGPT to "generate the citations after giving it author-year in-text citations, titles, or their paraphrases." They pasted the hallucinations in without checking. They've since responded with corrections to real papers that in most cases are very similar to the hallucination, lending credibility to their claim.[1] Not great, but to be clear this is different from fabricating the whole paper or the authors inventing the citations. (In this case at least.) [1] https://openreview.net/forum?id=IiEtQPGVyV https://openreview.net/forum?id=IiEtQPGVyV
  • bjourne 8mo ago
    I counted 15 hallucinated citations. The authors explanation is plausible, but it is still 15 citations to works they clearly have not read. Any university teaches you that citing sources you personally have not verified supports you claim(s) is fraudulent. Apologizing is not enough, they should retract the article.
    • abalone 8mo ago
      What makes you say they "clearly have not read" their citations? Are you assuming that because they used ChatGPT to generate the citation section based on their description of the papers that they haven't read the papers? Are you suggesting that their clarifications of which real papers the ChatGPT citations were meant to map to are fake, and if so which ones?
• lifetimerubyist 9mo ago
  Surely this will help with the trust in our institutions that has been completely eroded over the last 5 years.
• thestructuralme 9mo ago
  The most striking part of the report isn't just the 100 hallucinations—it’s the "submission tsunami" (220% increase since 2020) that made this possible. We’re seeing a literal manifestation of a system being exhausted by simulation. When a reviewer is outgunned by the volume of generative slop, the structure of peer review collapses because it was designed for human-to-human accountability, not for verifying high-speed statistical mimicry. In these papers, the hallucinations are a dead giveaway of a total decoupling of intelligence from any underlying "self" or presence. The machine calculates a plausible-looking citation, and an exhausted reviewer fails to notice the "Soul" of the research is missing. It feels like we’re entering a loop where the simulation is validated by the system, which then becomes the training data for the next generation of simulation. At that point, the human element of research isn't just obscured—it's rendered computationally irrelevant.
• deepsun 9mo ago
  Can we just hallucinate the whole conference by now? Like "Hey AI, generate me the whole conference agenda, schedule, papers, tracks, workshops, and keynote" and not pay the $1k?
• pama 9mo ago
  This feels like a big nothingburger to me. Try an analysis on conference submissions (perhaps even published papers) from 1995 for comparison, and one from 2005, one from 2015. I recall the typos/errors/ommissions because I reviewed for them and I used them. Even then: so what? If I could find the reference relatively easily and with enough confidence I was fine. Rarely I couldnt find it and contacted the author. The job of the reviewer (or even author) isnt to be a nitpicky editor—that’s the editor’s job. Editing does not happen until the final printed publication is near, and only for accepted papers, nowadays sometimes it never happens. Now that is a problem perhaps, but it has nothing to do with the authors’ use of LLMs.
• anishrverma 9mo ago
  The prevalence of hallucinations in the system is another signs for change in the system. The citations should be treated less like narrative context and more like verifiable objects Better detectors, like the article implies, won’t solve the problem, since AI will likely keep improving It’s about the fact that our publishing workflows implicitly assume good faith manual verification, even as submission volume and AI assisted writing explode. That assumption just doesn’t hold anymore A student initiative at Duke University has been working on what it might look like to address this at the publishing layer itself, by making references, review labor, and accountability explicit rather than implicit There’s a short explainer video for their system: https://liberata.info/ https://liberata.info/ It’s hard to argue that the current status quo will scale, so we need novel solutions like this.
• einpoklum 9mo ago
  I don't know about you, but where I'm from, we call citations from sources which don't exist "fabrications" or "fraud" - not "hallucination", which sounds like some medical condition which evokes pity.
• BitsAndObjects 9mo ago
  Th incorrect citations problem will disappear when AI web search and fetch becomes 100x cheaper than it is today. Right now, the APIs are too expensive to do proper multihundred results of papers (the search space for any paper is much larger than the final list of citations). However, we’ll be left with AI written papers and no real way to determine if they’re based on reality or just a “stochastic mirror” (an approximate reflection of reality).
• olivia-banks 9mo ago
  I'm an author on a paper on breast cancer, and one of our co-authors generated the majority of their work with AI. It just makes me angry.
• NightBlossom 9mo ago
  This feels less like scientific integrity and more like predatory marketing. I find this public "shame list" approach by GPTZero deeply unethical and technically suspect for several reasons: 1. Doxxing disguised as specific criticism: Publishing the names of authors and papers without prior private notification or independent verification is not how academic corrections work. It looks like a marketing stunt to generate buzz at the expense of researchers' reputations. 2. False Positives & Methodology: How does their tool distinguish between an actual AI "hallucination" and a simple human error (e.g., a typo in a year, a broken link, or a messy BibTeX entry)? Labeling human carelessness as "AI fabrication" is libelous. 3. The "Protection Racket" Vibe: The underlying message seems to be: "Buy our tool, or next time you might be on this list." It’s creating a problem (fear of public shaming) to sell the solution. We should be extremely skeptical of a vendor using a prestigious conference as a billboard for their product by essentially publicly shaming participants without due process.
  • nickpsecurity 9mo ago
    Yeah, my first question was whether or not the hallucination checker can hallucinate.
  • nonethewiser 9mo ago
    I think its great. They explicitly distinguish between a "flawed citation" (missing author, typo in title) and a hallucination (completely fabricated journal, fake DOI, nonexistent authors). You can literally click through and verify each one yourself. If you think they're wrong about a specific example, point it out. It doesn't matter if these are honest mistakes or not - they should be highlighted and you should be happy to have a tool that can find them before you publish. It's ridiculous to call it doxxing. The papers are already published at NeurIPS with author names attached. GPTZero isn't revealing anything that wasn't already public. They are pointing out what they think are hallucinations which everyone can judge for themselves. It might even be terrible at detecting things. Which actually, I do not think is the case after reading the article. But even so, if they are unreliable I think the problem takes care of itself.
  • RestartKernel 9mo ago
• scoper31134 9mo ago
  using AI in scientific research paper? its pretty pathetic
• baden-1927 9mo ago
  [dead]
• scoper31134 9mo ago
  what? why are you guys defending using AI at research paper? this world has gone insane
• lighthouse1212 9mo ago
  [dead]
• shawn10067 8mo ago
  The takeaway for me isn't that LLMs produce bad references—humans do that too—but that cutting corners shows in the final product. If your background section contains made‑up citations, it makes readers wonder how careful you were with the parts they can't check as easily. If you're going to use AI tools for draft writing, you still need to vet every fact.
• ensocode 8mo ago
  Gave GPTZero a random ChatGPT text about finances. It was 84% confident, that it was entirely human writing
• rurban 8mo ago
  I'd really like to have studied in these times, where it's so much easier with all the new tools. I could have been a triple doctor. At work I've automated tools to write automated technical certificates for wind parks. I've wrote code automatically to solve problems I couldn't solve by my own. Complicated Linear Algebra stuff, which was always too hard. I should have written papers automatically, at least my wife writes her reports with ChatGPT already. Others are writing film scripts by tools. Good times.
• maxeeezy 8mo ago
  [dead]
• neves 8mo ago
  I'm surprised by these results. I would have expected non-Anglo-American universities to rank at the top of the list. One of the most valuable features of LLMs from the beginning has been their ability to improve written language. This is particularly beneficial for non-English-speaking researchers in preventing language-related biases. However, the list shows that LLM usage is more intensive in the English-speaking world. Why?
• naasking 8mo ago
  AIs are much better and hallucinate much less when they are given focused tasks, eg. instead of asking AI to write a background on complete with citations, ask the AI specifically to generate a list of citations relevant to X, programmatically check the references are correct using a true index like doi, then ask AI to use the reference list to write a background section on X. This would be a valuable research tool that uses AI without the hallucinations.
• floriferous 8mo ago
  While this is really concerning, it feels like a small new category of errors to check for. The article mentions an increase of 220% of the amount of submissions. That's incredible news for science, probably lots of honest scientists able to produce more work and eventually lead to more science being done.
• Wokemon 8mo ago
  [dead]
• TheRealPomax 8mo ago
  They're not halluciations. Don't anthropomorphise this nonsene, call it what it is because this is not a new problem: this is garbage data, and that garbage data should have been caught. Having a submission pipeline that verifies sources even exist (not that they're citing the right thing) is one the bare minimum responsibilities of a paid journal. This has almost nothing to do with AI, and everything to do with a journal not putting in the trivial effort (given how much it costs to get published by them) required to ensure subject integrity. Yeah AI is the new garbage generator, but this problem isn't new, citation verification's been part of review ever since citations became a thing.
• AntonioEritas 8mo ago
  "I got my €220 back (ouch that's a lot of money for this kind of service, thanks capitalism)" 220 is actually quite the deal. In fact, heavy usage means Anthropic loses money on you. Do you have any idea how much compute cost to offer these kind of services?