7 ms·
> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue
by btown 5mo ago
> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue.
This is incredibly good for science. arXiv is free, but it's a privilege not a right!
I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply enough?
As a certain doctor once said: the whole point of the doomsday machine is lost if you keep it a secret!
- dataflow 5mo ago> This is incredibly good for science. I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress their life or contribute to humanity just because an AI hallucinated a reference one time in their life. That's punitive instead of rehabilitative.
- ajkjk 5mo agoIt's not the kind of mistake that is possible unless you're engaging in fraud anyway.
- dataflow 5mo ago> It's not the kind of mistake that is possible unless you're engaging in fraud anyway. Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists, and the AI inserting or modifying a citation incorrectly without them realizing? If you find evidence of fraud by all means lay down the hammer. Using a single hallucinated citation like it's some kind of ironclad proxy just because you think they must be committing fraud is insane.
- mixtureoftakes 5mo agoif you're not checking citations in the paper youre publishing AND trusting a non SOTA, hallucination prone ai model to come up with sources for it, its probably for the best of everyone that this paper isn't published. yes there will be rare exceptions but in general i feel like this is a really good addition.
- flexagoon 5mo ago> non SOTA, hallucination prone ai model What SOTA models are not hallucination prone?
- asdff 5mo agoWhy would you ask the ai to find a citation you know exists? Just reach for that citation.
- andrepd 5mo agoIf you are citing a work you paste a citation to that work. If you are bullshitting you ask an AI to come up with a citation. Jesus, there is zero reason to ever "generate a citation" if you are not, in fact, commiting fraud.
- galaxyLogic 5mo agoI much agree. But I wonder shouldn't the citations all be hyperlinks and thus easy to verify?
- no-name-here 5mo agoHow specific are the citations? If it's “Sentence 4 on page 97 supports” or “Paper says ‘___’” then I imagine it would be fairly easy. If it's “(__ page long) paper supports x”, then very difficult?
- fc417fc802 5mo agoThat's like saying that there's zero reason to ever ask an LLM to do basic math for you. Sure you probably shouldn't do that but sometimes it's convenient and so people will inevitably do exactly that regardless of the somewhat frequent wrong answers that are guaranteed to ensue.
- wrs 5mo agoA "mistake" would be a typo in a real citation. A hallucinated citation is evidence of just plain laziness and negligence, which taints the entire submission.
- dataflow 5mo agoNo it is not. Seriously. All you need for this to happen is for your lab partner to ask AI to add a missing citation that they are already familiar with at the last minute before a midnight submission deadline, and for the AI to hallucinate something else, and for them to honestly miss this. It does not even imply any involvement on your part, let alone that either of you were lazy or negligent on the actual research or substance of the paper. The lack of any sympathy or imagination here is astounding.
- asdff 5mo agoThere are no deadlines for journal submissions. Even if you felt you were running close to your revisions being due, an email to an editor will probably fix this for you. And what you described is still negligent, not verifying the garbage output bot did not in fact output garbage.
- AnimalMuppet 5mo agoEven more, there are no deadlines for arXiv submissions.
- cozzyd 5mo agoKinda. There is a deadline to appear in the next days posting and this can be important if e.g. you want to get something on the arxiv before a talk or a proposal deadline.
- bigstrat2003 5mo agoThe lack of understanding that you are responsible for the content you create, no matter what tools you use, is what's astounding.
- Loughla 5mo agoDon't use AI? Problem solved?
- patcon 5mo agoA citation is where you derived knowledge... If you haven't checked it and you are submitting something that should represent a ton of labour (and which will consume labour to review), you don't understand what you're doing. It is not just crossing T's and dotting I'd. Your being set behind is less important than the fact that your publishing is setting everyone else behind. Such a banned person is being helped to "step out of the way", and someone more competent will assuredly step forward to consume the limited maintenance labour more thoughtfully
- dataflow 5mo ago> Your being set behind is less important than the fact that your publishing is setting everyone else behind One hallucinated citation does not in any way imply anyone is being left behind. All it means nobody is checked that particular line of the manuscript after it was written. The rest of the paper could still be solid and treated accordingly. If you find evidence of the contrary, of course treat it accordingly, but this is so obviously not that.
- some_furry 5mo agoNo. It's fraud.
- andrepd 5mo agoYou clearly misunderstand. You cite a work in your paper because you have read that work, and build upon it or want to refer to it to back up a specific claim. Generating references is fraud period, because you are implying that you have read a work when in fact you just asked an AI "please insert some reference-shaped text here" to make it look like a proper paper. It is sadly not a necessary, but certainly a VERY sufficient, reason to conclude a paper is fraudulent.
- dumpsterdiver 5mo ago> One hallucinated citation does not in any way imply anyone is being left behind. The parent said “setting” others behind, which refers to lost time. Being “left” behind implies a degraded trajectory, which is defined not by time lost, but by the final destination. Different but related things (e.g. lost time can indeed affect your final destination, for instance, after growing old correcting a scourge of hallucinated citations - which should have been table stakes all along).
- conartist6 5mo agoYes, it is fraud
- toast0 5mo ago> It's just one darn hallucinated citation for heaven's sake, not fraud or something. It is fraud. > It doesn't account for the substance or quality of their work at all. References are part of the work. If you're making up the references, what else are you making up? > People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress their life or contribute to humanity just because an AI hallucinated a reference one time in their life. A one year ban is not permanent. Having a negative consequence for making poor decisions seems like an inducement to learn from the mistake? In an ideal world, one would be keeping notes on references used while doing the research that lead to writing the paper. Choosing not to do that is one poor decision. Having a positive outlook, if asking an AI to provide references that may have been missed, one should at least verify the references exist and are relevant. Choosing not to do that is also a poor decision, even if one did take notes on references used while researching.
- blazespin 5mo agoit's very silly, but not a big deal. Arxiv is becoming irrelevant these days anyways. In fact would be better if they just banned AI, so we could just get off the luddite platforms. Automated research is the future, end of story. And really it couldn't have come out at a better time, given the increasingly diminishing returns on human powered research.
- andrepd 5mo agoPoe's law striking hard.
- AnimalMuppet 5mo agoIf automated research is the future, it has to be research, not making stuff up. Which of those two does "hallucinated references" fit into?
- dataflow 5mo ago> It is fraud. No, it is emphatically not. Fraud requires intent to deceive. > A one year ban is not permanent. ...what text are you reading? Nobody was calling the one-year ban permanent, or even against it. I was literally in favor of it in my comment. I explicitly said it is already plenty sufficient. What I said is there's no need to go beyond that. My entire gripe was that they very much are going beyond that with a permanent penalty. Did you completely miss where they said "...followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue"?
- mianos 5mo agoYou are being ironic right?
- goolz 5mo agoIf you cannot be bothered to check your references when writing academic quality papers then you have no place writing them in the first place. The punishment is not chopping off a finger, it is a polite reminder to do the bare minimum.
- slashdave 5mo agoWell, in the good old days, when we have refereed journals, it would be part of the publishing process.
- themafia 5mo ago> There's no need to permanently cripple someone's ability to progress their life or contribute to humanity I don't think you need to publish on arXive to contribute meaningfully to humanity. > That's punitive instead of rehabilitative. Unfortunately science is competitive. Yours is a race to the bottom where the people who can afford the most expensive models and who are least concerned with the truth can publish the most papers and benefit financially and professionally by doing so. This is not a zero sum arena, grant money and opportunities will possibly be rewarded to them, and not to another team who is producing more careful and genuine output.
- vhantz 5mo agoWhat's the difference between a "hallucinated" citation and consciously inserting reference to a non-existent paper and hopping it goes unnoticed? How do we determine which one was done consciously and which was "a minor first time mistake"? Your standards are lower than what they would accept at my high-school. Seriously. And generally, if you are generating papers with LLMs, let other LLMs read them. Why would we waste human hours considering something that was generated? At this point publish your prompt because that's the actual work you're doing.
- simianparrot 5mo agoIn science, one hallucinated reference can corrupt the entire rest of the work. So you're completely wrong.
- gizajob 5mo agoAnd every piece of work in future which cites the paper with the hallucinated reference.
- nkrisc 5mo agoIt’s easy to avoid this whole issue: write the paper yourself.
- redsocksfan45 5mo ago[dead]
- avanwyk 5mo agoWe got pretty far without a single hallucinated citation (because they simply didn't exist). It's really simple: check your references, or don't bother of publishing. It's not a high hurdle.
- paulpauper 5mo agoMy take: this seems excessive. ArXiv doesn't even check the submission closely, so how can they know? They say "errors, mistakes" They use an automated system to check if the basic requirements were met, and sometimes papers are flagged for further superficial human review, but there is no way they can possibly do this at scale or check every reference. This would be like trying to do peer review, but for a preprint archive that gets easily 100x more volume than any journal. Second, there is such a huuuuge gap between publishing on arvix and peer review. I can attest personally that it's not even close. I've gotten probably dozen rejections from peer review and no problems publishing in arxiv math. This is because peer review checks not just for if something is new or correct, but also if it's of "interest to math community," which is inherently subjective, but also makes peer review many magnitudes harder than publishing on arxiv. Even when a well-known professor in number theory praised the paper when I got an endorsement and a second emailed me and and encouraged me to publish it, it still got rejected 3 times and still waiting. Being required to publish in a peer reviewed journal will close off arxiv for many researchers for good. It also defeats the point of it being a pre-print.
- helterskelter 5mo agoYou could at least filter out hallucinated references which simply don't exist pretty trivially, I'd imagine.
- paulpauper 5mo agoIt's more than that. if there are mistakes, then you can also be flagged. read the whole tweet: If generative AI tools generate inappropriate language, plagiarized content, biased content, errors, mistakes, incorrect references, or misleading content, and that output is included in scientific works, it is the responsibility of the author(s).
- nutjob2 5mo agoIf you'd read the whole series of tweets it's obvious that is not their intention and there needs to be "incontrovertible evidence that the authors did not check the results of LLM generation" for the penalty to apply. It's not hard to divine their intentions: you are entirely responsible for what you summit and if it's clearly slop(py) you get a ban. In a reply they state that they are seeking to apply this rule fairly and accurately and are mindful of unintended effects.
- nomel 5mo agoI bet, since this has been posted, someone here has already vibe coded a reference checker that they plan to put behind a subscription. This is good for reference checking, but I doubt this will do much for the most likely shoddy science that accompanies hallucinated references.
- nradov 5mo agoThe frontier LLMs are getting pretty good at checking this sort of thing. You could prompt them to not only verify the references are real but that they actually state what the article claims. Some human review will still be needed but I'll bet this approach could find a lot of academic fraud.
- nomel 5mo ago> The frontier LLMs are getting pretty good at checking this sort of thing. No, this is career ending high stakes. it requires old school "actually check a record of reality" type methods, like a database query or http get to one of the many services that hold this info.
- small_scombrus 5mo agoI think they're saying that frontier LLMs may be usable to spot citations that are correct by shape (a real citation) but incorrect by usage (unrelated to the text) I kind of hate the idea, but you probably could do a lazy LLM check of every paper and every citation and have it flag possible wrong (second sense) citations for human review But you'd need a LOT of tokens and a LOT of human-hours
- mpalmer 5mo ago> have it flag possible wrong (second sense) citations for human review And then what, we're done? How have we avoided the need for the same exhaustive human review? It only saves human review time if you trust the LLM not to miss things.
- ____mr____ 5mo ago> subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. Isn't the whole point of arxiv to be somewhere you post your low-reviewed/unpublished/prepublished papers? doesnt this somewhat defeat the point and basically means a lifetime ban?
- purerandomness 5mo agoIf it isn't, it absolutely should be. The penalty cannot be high enough.
- jkman 5mo agoEh guess so, I think it's appropriate
- kordlessagain 5mo agoIt's a gatekeepers dream, for sure.
- dan-g 5mo agoSorry maybe I’m missing something, but how is this gatekeeping?
- chadgpt3 5mo agoarXiv is gatekeeping science by only accepting fact-based papers instead of fairytales.
- bigfishrunning 5mo agoI feel like requiring that your cited references are real is a pretty low bar
- ls612 5mo agoIt sounds like even copy/paste errors from Google Scholar would qualify for the lifetime ban. Let he who is without sin…
- bigfishrunning 5mo agoLet's be honest though, most of the not-real cited papers aren't copy/paste errors. If a few legitimate people fall through the cracks, it's a shame, but the wave of slop has to be dealt with somehow
- T-A 5mo ago>> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. Catch-22 [1]: You will need to provide an arXiv article id. This is necessary for your paper to be processed correctly, the submitted version must be the same as the arXiv version. [1] https://jcap.sissa.it/jcap/help/helpLoader.jsp?pgType=author https://jcap.sissa.it/jcap/help/helpLoader.jsp?pgType=author