8 ms·

If a carpenter builds a crappy shelf “because” his power tools are not calibrated correctly - that’s a crappy carpenter, not a crappy tool. If a scientist uses

If a carpenter builds a crappy shelf “because” his power tools are not calibrated correctly - that’s a crappy carpenter, not a crappy tool. If a scientist uses an LLM to write a paper with fabricated citations - that’s a crappy scientist. AI is not the problem, laziness and negligence is. There needs to be serious social consequences to this kind of thing, otherwise we are tacitly endorsing it.

RossBencina 10mo ago
No qualified carpenter expects to use a hammer to drill a hole.
gdulli 10mo ago
That's like saying guns aren't the problem, the desire to shoot is the problem. Okay, sure, but wanting something like a metal detector requires us to focus on the more tangible aspect that is the gun.
- baxtr 10mo ago
  If I gave you a gun would you start shooting people just because you had one?
  - agentultra 10mo ago
    If I gave you a gun without a safety could you be the one to blame when it goes off because you weren’t careful enough? The problem with this analogy is that it makes no sense. LLMs aren’t guns. The problem with using them is that humans have to review the content for accuracy. And that gets tiresome because the whole point is that the LLM saves you time and effort doing it yourself. So naturally people will tend to stop checking and assume the output is correct, “because the LLM is so good.” Then you get false citations and bogus claims everywhere.
    - sigbottle 10mo ago
      Sorry, I'm not following the gun analogies at all But regardless, I thought the point was that... > The problem with using them is that humans have to review the content for accuracy. There are (at least) two humans in this equation. The publisher, and the reader. The publisher at least should do their due diligence, regardless of how "hard" it is (in this case, we literally just ask that you review your OWN CITATIONS that you insert into your paper). This is why we have accountability as a concept.
    - oceansweep 10mo ago
      Yes. That is absolutely the case. One of the Most popular handguns does not have a safety switch that must be toggled before firing. (Glock series handguns) If someone performs a negligent discharge, they are responsible, not Glock. It does have other safety mechanisms to prevent accidental fires not resulting from a trigger pull.
      - agentultra 10mo ago
        You seem to be getting hung up on the details of guns and missing the point that it’s a bad analogy. Another way LLMs are not guns: you don’t need a giant data centre owned by a mega corp to use your gun. Can’t do science because GlockGPT is down? Too bad I guess. Let’s go watch the paint dry. The reason I made it is because this is inherently how we designed LLMs. They will make bad citations and people need to be careful.
    - zdragnar 10mo ago
      > If I gave you a gun without a safety could you be the one to blame when it goes off because you weren’t careful enough? Absolutely. Many guns don't have safties. You don't load a round in the chamber unless you intend on using it. A gun going off when you don't intend is a negligent discharge. No ifs, ands or buts. The person in possession of the gun is always responsible for it.
      - bluGill 10mo ago
        > A gun going off when you don't intend is a negligent discharg false. A gun goes off when not intended too often to claim that. It has happned to me - I then took the gun to a qualified gunsmith for repairs. A gun they fires and hits anything you didn't intend to is negligent discharge even if you intended to shoot. Gun saftey is about assuming a gun that could possible fire will and ensuring nothing bad can happen. When looking at gun in a store (that you might want to buy) you aim it at an upper corner where even if it fires the odds of something bad resulting is the least lively to happen (it should be unloaded - and you may have checked, but you still aim there!) same with cat toy lazers - they should be safe to shine in an eye - but you still point in a safe direction.
  - komali2 10mo ago
    Ok sure I'm down for this hypothetical. I will bring 50 random people in front of you, and you will hand all 50 of them loaded guns. Still feeling it?
    - bandofthehawk 10mo ago
      Ever been to a shooting range? It's basically a bunch of random people with loaded guns.
      - komali2 10mo ago
        That's not as random as letting me choose them! They had to be allowed onto the range, show ID, afford the gun, probably do a background check to get the gun unless they used a loophole (which usually requires some social capital). I'm proposing the true proposal of many guns rights advocates: anyone might have a gun. So let me choose the 50 and you give them guns! Why not?
  - deleted 10mo ago
    [deleted]
  - hipshaker 10mo ago
    If you look at gun violence in the U.S that is , speaking as a European, kind of what I see happening.
  - gdulli 10mo ago
    That doesn't address my point at all but no, I'm not a violent or murderous person. And most people aren't. Many more people do, however, want to take shortcuts to get their work done with the least amount of effort possible.
    - SauntSolaire 10mo ago
      > Many more people do, however, want to take shortcuts to get their work done with the least amount of effort possible. Yes, and they are the ones responsible for the poor quality of work that results from that.
  - raincole 10mo ago
    If the society rewarded me money and fame when I kill someone then I would. Why wouldn't I? Like it or not, in our society scientists' job is to churn out papers. Of course they'll use the most efficient way to churn out papers.
  - intended 10mo ago
    The issue with this argument, for anyone who comes after, is not when you give a gun to a SINGLE person, and then ask them "would you do a bad thing". The issue is when you give EVERYONE guns, and then are surprised when enough people do bad things with them, to create externalities for everyone else. There is some sort of trip up when personal responsibility, and society wide behaviors, intersect. Sure most people will be reasonable, but the issue is often the cost of the number of irresponsible or outright bad actors.
  - rcpt 10mo ago
    Probably not but, empirically, there are a lot of short tempered people who would.
deleted 10mo ago
[deleted]
TomatoCo 10mo ago
To continue the carpenter analogy, the issue with LLMs is that the shelf looks great but is structurally unsound. That it looks good on surface inspection makes it harder to tell that the person making it had no idea what they're doing.
- embedding-shape 10mo ago
  Regardless, if a carpenter is not validating their work before selling it, it's the same as if a researcher doesn't validate their citations before publishing. Neither of them have any excuses, and one isn't harder to detect than the other. It's just straight up laziness regardless.
  - judofyr 10mo ago
    I think this is a bit unfair. The carpenters are (1) living in world where there’s an extreme focus on delivering as quicklyas possible, (2) being presented with a tool which is promised by prominent figures to be amazing, and (3) the tool is given at a low cost due to being subsidized. And yet, we’re not supposed to criticize the tool or its makers? Clearly there’s more problems in this world than «lazy carpenters»?
    - embedding-shape 10mo ago
      > And yet, we’re not supposed to criticize the tool or its makers? Exactly, they're not forcing anyone to use these things, but sometimes others (their managers/bosses) forced them to. Yet it's their responsibility for choosing the right tool for the right problem, like any other professional. If a carpenter shows up to put a roof yet their hammer or nail-gun can't actually put in nails, who'd you blame; the tool, the toolmaker or the carpenter?
      - judofyr 10mo ago
        > If a carpenter shows up to put a roof yet their hammer or nail-gun can't actually put in nails, who'd you blame; the tool, the toolmaker or the carpenter? I would be unhappy with the carpenter, yes. But if the toolmaker was constantly over-promising (lying?), lobbying with governments, pushing their tools into the hands of carpenters, never taking responsibility, then I would also criticize the toolmaker. It’s also a toolmaker’s responsibility to be honest about what the tool should be used for. I think it’s a bit too simplistic to say «AI is not the problem» with the current state of the industry.
CapitalistCartr 10mo ago
I'm an industrial electrician. A lot of poor electrical work is visible only to a fellow electrician, and sometimes only another industrial electrician. Bad technical work requires technical inspectors to criticize. Sometimes highly skilled ones.
- andy99 10mo ago
  I’ve reviewed a lot of papers, I don’t consider it the reviewers responsibility to manually verify all citations are real. If there was an unusual citation that was relied on heavily for the basis of the work, one would expect it to be checked. Things like broad prior work, you’d just assume it’s part of background. The reviewer is not a proofreader, they are checking the rigour and relevance of the work, which does not rest heavily on all of the references in a document. They are also assuming good faith.
  - zdragnar 10mo ago
    This is half the basis for the replication crisis, no? Shady papers come out and people cite them endlessly with no critical thought or verification. After all, their grant covers their thesis, not their thesis plus all of the theses they cite.
  - Aurornis 10mo ago
    > I don’t consider it the reviewers responsibility to manually verify all citations are real I guess this explains all those times over the years where I follow a citation from a paper and discover it doesn’t support what the first paper claimed.
  - auggierose 10mo ago
    In short, a review has no objective value, it is just an obstacle to be gamed.
    - amanaplanacanal 10mo ago
      In theory, the review tries to determine if the conclusion reached actually follows from whatever data is provided. It assumes that everything is honest, it's just looking to see if there were mistakes made.
left-struck 10mo ago
It’s like the problem was there all along, all LLMs did was expose it more
- criley2 10mo ago
  https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Modern science is designed from the top to the bottom to produce bad results. The incentives are all mucked up. It's absolutely not surprising that AI is quickly becoming yet-another factor lowering quality.
- theoldgreybeard 10mo ago
  Yes, LLMs didnt create the problem they just accelerated it to a speed that beggars belief.
thaumasiotes 10mo ago
> If a scientist uses an LLM to write a paper with fabricated citations - that’s a crappy scientist. Really? Regardless of whether it's a good paper?
- zwnow 10mo ago
  How is it a good paper if the info in it cant be trusted lmao
  - thaumasiotes 10mo ago
    Whether the information in the paper can be trusted is an entirely separate concern. Old Chinese mathematics texts are difficult to date because they often purport to be older than they are. But the contents are unaffected by this. There is a history-of-math problem, but there's no math problem.
    - zwnow 10mo ago
      Not really true nowadays. Stuff in whitepapers needs to be verifiable which is kinda difficult with hallucinations. Whether the students directly used LLMs or just read content online that was produced with them and cited after just shows how difficult these things made gathering information that's verifiable.
      - thaumasiotes 10mo ago
        > Stuff in whitepapers needs to be verifiable which is kinda difficult with hallucinations. That's... gibberish. Anything you can do to verify a paper, you can do to verify the same paper with all citations scrubbed. Whether the citations support the paper, or whether they exist at all, just doesn't have anything to do with what the paper says.
        zwnow 10mo ago
        I dont think you know how whitepapers work then
hansmayer 10mo ago
Scientists who use LLMs to write a paper are crappy scientists indeed. They need to be held accountable, even ostracised by the scientific community. But something is missing from the picture. Why is it that they came up with this idea in the first place? Who could have been peddling the impression (not an outright lie - they are very careful) about LLMs being these almost sentient systems with emergent intelligence, alleviating all of your problems, blah blah blah. Where is the god damn cure for cancer the LLMs were supposed to invent? Who else is it that we need to keep accountable, scrutinised and ostracised for the ever-increasing mountains of AI-crap that is flooding not just the Internet content but now also penetrating into science, every day work, daily lives, conversations, etc. If someone released a tool that enabled and encouraged people to commit suicide in multiple instances that we know of by now, and we know since the infamous "plandemic" facebook trend that the tech bros are more than happy to tolerate worsening societal conditions in the name of their platform growth, who else do we need to keep accountable, scrutinise and ostracise as a society, I wonder?
- the8472 10mo ago
  > Where is the god damn cure for cancer the LLMs were supposed to invent? Assuming that cure is meant as hyperbole, how about https://www.biorxiv.org/content/10.1101/2025.04.14.648850v3 https://www.biorxiv.org/content/10.1101/2025.04.14.648850v3 ? AI models being used for bad purposes doesn't preclude them being used for good purposes.
  - hansmayer 10mo ago
    ...No, it was not meant as a hyperbole, as we were literally being told that these models will be able to do all of our work. I won't settle for the bullshit incremental wins here and there we see occassionally - I attribute those essentially to the old 'infinite number of monkeys typing on the infinite number of typewriters producing "Crime and Peace". No. that's not it - we were promised a god damn revolution, no less. Again, where is the cure for cancer and post-scarcity society ? Where is the AGI we were promised for the 2025? Let's hold the ghouls promising all that accountable for a change.
Forgeties79 10mo ago
If my calculator gives me the wrong number 20% of the time yeah I should’ve identified the problem, but ideally, that wouldn’t have been sold to me as a functioning calculator in the first place.
- imiric 10mo ago
  Indeed. The narrative that this type of issue is entirely the responsibility of the user to fix is insulting, and blame deflection 101. It's not like these are new issues. They're the same ones we've experienced since the introduction of these tools. And yet the focus has always been to throw more data and compute at the problem, and optimize for fancy benchmarks, instead of addressing these fundamental problems. Worse still, whenever they're brought up users are blamed for "holding it wrong", or for misunderstanding how the tools work. I don't care. An "artificial intelligence" shouldn't be plagued by these issues.
  - SauntSolaire 10mo ago
    > It's not like these are new issues. Exactly, that's why not verifying the output is even less defensible now than it ever has been - especially for professional scientists who are responsible for the quality of their own work.
    - Forgeties79 10mo ago
      If I have to constantly assess every single line done by an LLM then we are fast approaching a point where it’s no longer being helpful and I’m just grading homework for a C student. I’m not saying that isn’t what has to be done, but it kind of clashes with the whole “this will make you more productive” argument if you ask me
  - Forgeties79 10mo ago
    > Worse still, whenever they're brought up users are blamed for "holding it wrong", or for misunderstanding how the tools work. I don't care. An "artificial intelligence" shouldn't be plagued by these issues. My feelings exactly, but you’re articulating it better than I typically do ha
belter 10mo ago
"...each of which were missed by 3-5 peer reviewers..." Its sloppy work all the way down...
- cindyllm 10mo ago
  [dead]
deleted 10mo ago
[deleted]
rectang 10mo ago
“X isn’t the problem, people are the problem.” — the age-old cry of industry resisting regulation.
- codywashere 10mo ago
  what regulation are you advocating for here?
  - kibwen 10mo ago
    At the very least, authors who have been caught publishing proven fabrications should be barred by those journals from ever publishing in them again. Mind you, this is regardless of whether or not an LLM was involved.
    - JumpCrisscross 10mo ago
      > authors who have been caught publishing proven fabrications should be barred by those journals from ever publishing in them again This is too harsh. Instead, their papers should be required to disclose the transgression for a period of time, and their institution should have to disclose it publicly as well as to the government, students and donors whenever they ask them for money.
  - rectang 10mo ago
    I’m not advocating, I’m making a high-level observation: Industry forever pushes for nil regulation and blames bad actors for damaging use. But we always have some regulation in the end. Even if certain firearms are legal to own, howitzers are not — although it still takes a “bad actor” to rain down death on City Hall. The same dynamic is at play with LLMs: “Don’t regulate us, punish bad actors! If you still have a problem, punish them harder!” Well yes, we will punish bad actors, but we will also go through a negotiation of how heavily to constrain the use of your technology.
    - codywashere 10mo ago
      so, what regulation do we need on LLMs? the person you originally responded to isn’t against regulation per their comment. I’m not against regulation. what’s the pitch for regulation of LLMs?
only-one1701 10mo ago
Absolutely brutal case of engineering brain here. Real "guns don't kill people, people kill people" stuff.
- theoldgreybeard 10mo ago
  If you were to wager a guess, what do you think my views on gun rights are?
  - only-one1701 10mo ago
    Probably something equally as nuanced and correct as the statement I replied to!
    - theoldgreybeard 10mo ago
      You're projecting.
- somehnguy 10mo ago
  Your second statement is correct. What about it makes it “engineering brain”?
  - rcpt 10mo ago
    If the blame were solely on the user then we'd see similar rates of deaths from gun violence in the US vs. other countries. But we don't, because users are influenced by the UX
  - venturecruelty 10mo ago
    Somehow people don't kill people nearly as easily, or with as high of a frequency or social support, in places that don't have guns that are more accessible than healthcare. So weird.
raincole 10mo ago
Given we tacitly accepted replication crisis we'll definitely tacitly accept this.
jodleif 10mo ago
I find this to be a bit “easy”. There is such a thing as bad tools. If it is difficult to determine if the tool is good or bad i’d say some of the blame has to be put on the tool.
photochemsyn 10mo ago
Yeah, I can't imagine not being familiar with every single reference in the bibliography of a technical publication with one's name on it. It's almost as bad as those PIs who rely on lab techs and postdocs to generate research data using equipment that they don't understand the workings of - but then, I've seen that kind of thing repeatedly in research academia, along with actual fabrication of data in the name of getting another paper out the door, another PhD granted, etc. Unfortunately, a large fraction of academic fraud has historically been detected by sloppy data duplication, and with LLMs and similar image generation tools, data fabrication has never been easier to do or harder to detect.
nialv7 10mo ago
Ah, the "guns don't kill people, people kill people" argument. I mean sure, but having a tool that made fabrication so much easier has made the problem a lot worse, don't you think?
- theoldgreybeard 10mo ago
  Yes I do agree with you that having a tool that gives rocket fuel to a fraud engine should probably be regulated in some fashion. Tiered licensing, mandatory safety training, and weapon classification by law enforcement works really well for Canada’s gun regime, for example.
bigstrat2003 10mo ago
> If a carpenter builds a crappy shelf “because” his power tools are not calibrated correctly - that’s a crappy carpenter, not a crappy tool. It's both. The tool is crappy, and the carpenter is crappy for blindly trusting it. > AI is not the problem, laziness and negligence is. Similarly, both are a problem here. LLMs are a bad tool, and we should hold people responsible when they blindly trust this bad tool and get bad results.
Hammershaft 10mo ago
AI dramatically changes the perceived cost/benefit of laziness and negligence, which is leading to much more of it.
kklisura 10mo ago
> AI is not the problem, laziness and negligence is This reminds me about discourse about a gun problem in US, "guns don't kill people, people kill people", etc - it is a discourse used solely for the purpose of not doing anything and not addressing anything about the underlying problem. So no, you're wrong - AI IS THE PROBLEM.
- Yoofie 10mo ago
  No, the OP is right in this case. Did you read TFA? It was "peer reviewed". > Worryingly, each of these submissions has already been reviewed by 3-5 peer experts, most of whom missed the fake citation(s). This failure suggests that some of these papers might have been accepted by ICLR without any intervention. Some had average ratings of 8/10, meaning they would almost certainly have been published. If the peer reviewers can't be bothered to do the basics, then there is literally no point to peer review, which is fully independent of the author who uses or doesn't use AI tools.
  - smileybarry 10mo ago
    Peer reviewers can also use AI tools, which will hallucinate a "this seems fine" response.
  - amrocha 10mo ago
    If AI fraud is good at avoiding detection via peer review that doesn’t mean peer review is useless. If your unit tests don’t catch all errors it doesn’t mean unit tests are useless.
- sneak 10mo ago
  > it is a discourse used solely for the purpose of not doing anything and not addressing anything about the underlying problem Solely? Oh brother. In reality it’s the complete opposite. It exists to highlight the actual source of the problem, as both industries/practitioners using AI professionally and safely, and communities with very high rates of gun ownership and exceptionally low rates of gun violence exist. It isn’t the tools. It’s the social circumstances of the people with access to the tools. That’s the point. The tools are inanimate. You can use them well or use them badly. The existence of the tools does not make humans act badly.
b00ty4breakfast 10mo ago
maybe the hammer factory should be held responsible for pumping out so many poorly calibrated hammer
- venturecruelty 10mo ago
  No, because this would cost tens of jobs and affect someone's profits, which are sacrosanct. Obviously the market wants exploding hammers, or else people wouldn't buy them. I am very smart.
- SauntSolaire 10mo ago
  The obvious solution in this scenario is.. to just buy a different hammer. And in the case of AI, either review its output, or simply don't use it. No one has a gun to your head forcing you to use this product (and poorly at that). It's quite telling that, even in this basic hypothetical, your first instinct is to gesture vaguely in the direction of governmental action, rather than expect any agency at the level of the individual.
  - b00ty4breakfast 10mo ago
    >It's quite telling that, even in this basic hypothetical, your first instinct is to gesture vaguely in the direction of governmental action, rather than expect any agency at the level of the individual. When "individuals" (which is a funny way to refer to the global generative AI zeitgeist currently in full binge-mode that is encouraging and enabling this kind of behavior) refuse to regulate themselves, they have to be encouraged through external pressures to do so. Industry is so far up it's own ass wrt AI that all it can see is shit, there is no chance in hell that they will self-regulate. They gladly and indiscriminately slurp up the digital effluent that is currently sliding out the colon of the generative AI super-organism. And, of course, these "individuals" are more than happy to share the consequences with the rest of the world without sharing too much of the corn that they're digging out of the shit. It does not behoove the rest of the world to not protect it's self-interest, to minimize the consequences of foolish and irresponsible generative AI usage and to make sure it gets it's fare share of the semi-digested golden kernels
constantcrying 10mo ago
Absolutely correct. The real issue is that these people can avoid punishment. If you do not care enough about your paper to even verify the existence of citations, then you obviously should not have a job as a scientist. Taking an academic who does something like that seriously, seem impossible. At best he is someone who is neglecting his most basic duties as an academic, at worst he is just a fraudster. In both cases he should be shunned and excluded.
SubiculumCode 10mo ago
Yeah seriously. Using an LLM to help find papers is fine. Then you read them. Then you use a tool like Zotero or manually add citations. I use Gemini Pro to identify useful papers that I might not yet have encountered before. But, even when asking to restrict itself to Pubmed resources, it's citations are wonky, citing three different version sources of the same paper (citations that don't say what they said they'd discuss). That said, these tools have substantially reduced hallucinations over the last year, and will just get better. It also helps if you can restrict it to reference already screened papers. Finally, I'd lke to say tthat if we want scientists to engage in good science, stop forcing them to spend a third of their time in a rat race for funding...it is ridiculously time consuming and wasteful of expertise.
- bossyTeacher 10mo ago
  The problem isn't whether they have more or less hallucinations. The problem is that they have them. And as long as they hallucinate, you have to deal with that. It doesn't really matter how you prompt, you can't prevent hallucinations from happening and without manual checking, eventually hallucinations will slip under the radar because the only difference between a real pattern and a hallucinated one is that one exists in the world and the other one doesn't. This is not something you can really counter with more LLMs either as it is a problem intrinsic to LLMs
  - SubiculumCode 10mo ago
    Humans also hallucinate. We have an error rate. Your argument makes little sense in absolutist terms.
    - bossyTeacher 10mo ago
      > Humans also hallucinate "LLM hallucinations" and hallucinations are essentially different. Human hallucinations are related to perceptual experiences not memory errors like in the case of LLMs. Humans with certain neurological conditions hallucinate. Humans with healthy brains don't. This habit of misapplying terms needs to stop. Humans are not backpropagation algorithms nor whatever random concept you read about in a comp sci book.
deleted 10mo ago
[deleted]
mk89 10mo ago
> we are tacitly endorsing it. We are, in fact, not tacitly but openly endorsing this, due to this AI everywhere madness. I am so looking forward to when some genius in some banks starts to use it to simplify code and suddenly I have 100000000 € on my bank account. :)
jgalt212 10mo ago
fair enough, but carpenters are not being beat over the head to use new-fangled probabilistic speed squares.
grey-area 10mo ago
Generative AI and the companies selling it with false promises and using it for real work absolutely are the problem.
acituan 10mo ago
> AI is not the problem, laziness and negligence is. As much as I agree with you that this is wrong, there is a danger in putting the onus just on the human. Whether due to competition or top down expectations, humans are and will be pressured to use AI tools alongside their work and produce more. Whereas the original idea was for AI to assist the human, as the expected velocity and consumption pressure increases humans are more and more turning into a mere accountability laundering scheme for machine output. When we blame just the human, we are doing exactly what this scheme wants us to do. Therefore we must also criticize all the systemic factors that puts pressure on reversal of AI‘s assistance into AI’s domination of human activity. So AI (not as a technology but as a product when shoved down the throats) is the problem.
- alexcdot 10mo ago
  Absolutely, expectations and tools given by management are a real problem. If management fires you because they are wrong about how good AI is, and you're right - at the end of the day, you're fired and the manager is in lalaland. People need to actually push the correct calibration of what these tools should be trusted to do, while also trying to work with what they have.
rdiddly 10mo ago
¿Por qué no los dos?
jval43 10mo ago
If a scientist just completely "made up" their references 10 years ago, that's a fraudster. Not just dishonesty but outright academic fraud. If a scientist does it now, they just blame it on AI. But the consequences should remain the same. This is not an honest mistake. People that do this - even once - should be banned for life. They put their name on the thing. But just like with plagiarism, falsifying data and academic cheating, somehow a large subset of people thinks it's okay to cheat and lie, and another subset gives them chance after chance to misbehave like they're some kind of children. But these are adults and anyone doing this simply lacks morals and will never improve. And yes, I've published in academia and I've never cheated or plagiarized in my life. That should not be a drawback.
calmworm 10mo ago
I don’t understand. You’re saying even with crappy tools one should be able to do the job the same as with well made tools?
- tedd4u 10mo ago
  Three and a half years ago nobody had ever used tools like this. It can't be a legitimate complaint for an author to say, "not my fault my citations are fake it's the fault of these tools" because until recently no such tools were available and the expectation was that all citations are real.
  - calmworm 10mo ago
    Then it’s just a poor analogy.
DonHopkins 10mo ago
Shouldn't there be a black list of people who get caught writing fraudulent papers?
- cindyllm 10mo ago
  [dead]
- theoldgreybeard 10mo ago
  Probably. Something like that is what I meant by “social consequences”. Perhaps there should be civil or criminal ones for more egregious cases.
nwallin 10mo ago
"Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can’t break."--Bruce Schneier There's a corollary here with LLMs, but I'm not pithy enough to phrase it well. Anyone can create something using LLMs that they, themselves, aren't skilled enough to spot the LLMs' hallucinations. Or something. LLMs are incredibly good at exploiting peoples' confirmation biases. If it "thinks" it knows what you believe/want, it will tell you what you believe/want. There does not exist a way to interface with LLMs that will not ultimately end in the LLM telling you exactly what you want to hear. Using an LLM in your process necessarily results in being told that you're right, even when you're wrong. Using an LLM necessarily results in it reinforcing all of your prior beliefs, regardless of whether those prior beliefs are correct. To an LLM, all hypotheses are true, it's just a matter of hallucinating enough evidence to satisfy the users' skepticism. I do not believe there exists a way to safely use LLMs in scientific processes. Period. If my belief is true, and ChatGPT has told me it's true, then yes, AI, the tool, is the problem, not the human using the tool.
- czl 10mo ago
  > I do not believe there exists a way to safely use LLMs in scientific processes. What about giving the LLM a narrowly scoped role as a hostile reviewer, while your job is to strengthen the write-up to address any valid objections it raises, plus any hallucinations or confusions it introduces? That’s similar to fuzz testing software to see what breaks or where the reasoning crashes. Used this way, the model isn’t a source of truth or a decision-maker. It’s a stress test for your argument and your clarity. Obviously it shouldn’t be the only check you do, but it can still be a useful tool in the broader validation process.
foxfired 10mo ago
I disagree. When the tool promises to do something, you end up trusting it to do the thing. When Tesla says their car is self driving, people trust them to self drive. Yes, you can blame the user for believing, but that's exactly what they were promised. > Why didn't the lawyer who used ChatGPT to draft legal briefs verify the case citations before presenting them to a judge? Why are developers raising issues on projects like cURL using LLMs, but not verifying the generated code before pushing a Pull Request? Why are students using AI to write their essays, yet submitting the result without a single read-through? They are all using LLMs as their time-saving strategy. [0] It's not laziness, its the feature we were promised. We can't keep saying everyone is holding it wrong. [0]: https://idiallo.com/blog/none-of-us-read-the-specs https://idiallo.com/blog/none-of-us-read-the-specs
- rolandog 10mo ago
  Very well put. You're promised Artificial Super Intelligence and shown a super cherry-picked promo and instead get an agent that can't hold its drool and needs constant hand-holding... it can't be both things at the same time, so... which is it?
stocksinsmocks 10mo ago
Trades also have self regulation. You can’t sell plumbing services or build houses without any experience or you get in legal trouble. If your workmanship is poor, you can be disciplined by the board even if the tool was at fault. I think fraudulent publications should be taken at least as seriously as badly installed toilets.
venturecruelty 10mo ago
"It's not a fentanyl problem, it's a people problem." "It's not a car infrastructure problem, it's a people problem." "It's not a food safety problem, it's a people problem." "It's not a lead paint problem, it's a people problem." "It's not an asbestos problem, it's a people problem." "It's not a smoking problem, it's a people problem."
- SauntSolaire 10mo ago
  What an absurd set of equivalences to make regarding a scientist's relationship to their own work. If an engineer provided this line of excuse to me, I wouldn't let them anywhere near a product again - a complete abdication of personal and professional responsibility.
psychoslave 10mo ago
I don't see much crappy power tool provider throwing billions in marketing and product placement to make them used everywhere.

RossBencina 10mo ago

No qualified carpenter expects to use a hammer to drill a hole.

gdulli 10mo ago

That's like saying guns aren't the problem, the desire to shoot is the problem. Okay, sure, but wanting something like a metal detector requires us to focus on the more tangible aspect that is the gun.

baxtr 10mo ago
If I gave you a gun would you start shooting people just because you had one?
- agentultra 10mo ago
  If I gave you a gun without a safety could you be the one to blame when it goes off because you weren’t careful enough? The problem with this analogy is that it makes no sense. LLMs aren’t guns. The problem with using them is that humans have to review the content for accuracy. And that gets tiresome because the whole point is that the LLM saves you time and effort doing it yourself. So naturally people will tend to stop checking and assume the output is correct, “because the LLM is so good.” Then you get false citations and bogus claims everywhere.
  - sigbottle 10mo ago
    Sorry, I'm not following the gun analogies at all But regardless, I thought the point was that... > The problem with using them is that humans have to review the content for accuracy. There are (at least) two humans in this equation. The publisher, and the reader. The publisher at least should do their due diligence, regardless of how "hard" it is (in this case, we literally just ask that you review your OWN CITATIONS that you insert into your paper). This is why we have accountability as a concept.
  - oceansweep 10mo ago
    Yes. That is absolutely the case. One of the Most popular handguns does not have a safety switch that must be toggled before firing. (Glock series handguns) If someone performs a negligent discharge, they are responsible, not Glock. It does have other safety mechanisms to prevent accidental fires not resulting from a trigger pull.
    - agentultra 10mo ago
      You seem to be getting hung up on the details of guns and missing the point that it’s a bad analogy. Another way LLMs are not guns: you don’t need a giant data centre owned by a mega corp to use your gun. Can’t do science because GlockGPT is down? Too bad I guess. Let’s go watch the paint dry. The reason I made it is because this is inherently how we designed LLMs. They will make bad citations and people need to be careful.
  - zdragnar 10mo ago
    > If I gave you a gun without a safety could you be the one to blame when it goes off because you weren’t careful enough? Absolutely. Many guns don't have safties. You don't load a round in the chamber unless you intend on using it. A gun going off when you don't intend is a negligent discharge. No ifs, ands or buts. The person in possession of the gun is always responsible for it.
    - bluGill 10mo ago
      > A gun going off when you don't intend is a negligent discharg false. A gun goes off when not intended too often to claim that. It has happned to me - I then took the gun to a qualified gunsmith for repairs. A gun they fires and hits anything you didn't intend to is negligent discharge even if you intended to shoot. Gun saftey is about assuming a gun that could possible fire will and ensuring nothing bad can happen. When looking at gun in a store (that you might want to buy) you aim it at an upper corner where even if it fires the odds of something bad resulting is the least lively to happen (it should be unloaded - and you may have checked, but you still aim there!) same with cat toy lazers - they should be safe to shine in an eye - but you still point in a safe direction.
- komali2 10mo ago
  Ok sure I'm down for this hypothetical. I will bring 50 random people in front of you, and you will hand all 50 of them loaded guns. Still feeling it?
  - bandofthehawk 10mo ago
    Ever been to a shooting range? It's basically a bunch of random people with loaded guns.
    - komali2 10mo ago
      That's not as random as letting me choose them! They had to be allowed onto the range, show ID, afford the gun, probably do a background check to get the gun unless they used a loophole (which usually requires some social capital). I'm proposing the true proposal of many guns rights advocates: anyone might have a gun. So let me choose the 50 and you give them guns! Why not?
- deleted 10mo ago
  [deleted]
- hipshaker 10mo ago
  If you look at gun violence in the U.S that is , speaking as a European, kind of what I see happening.
- gdulli 10mo ago
  That doesn't address my point at all but no, I'm not a violent or murderous person. And most people aren't. Many more people do, however, want to take shortcuts to get their work done with the least amount of effort possible.
  - SauntSolaire 10mo ago
    > Many more people do, however, want to take shortcuts to get their work done with the least amount of effort possible. Yes, and they are the ones responsible for the poor quality of work that results from that.
- raincole 10mo ago
  If the society rewarded me money and fame when I kill someone then I would. Why wouldn't I? Like it or not, in our society scientists' job is to churn out papers. Of course they'll use the most efficient way to churn out papers.
- intended 10mo ago
  The issue with this argument, for anyone who comes after, is not when you give a gun to a SINGLE person, and then ask them "would you do a bad thing". The issue is when you give EVERYONE guns, and then are surprised when enough people do bad things with them, to create externalities for everyone else. There is some sort of trip up when personal responsibility, and society wide behaviors, intersect. Sure most people will be reasonable, but the issue is often the cost of the number of irresponsible or outright bad actors.
- rcpt 10mo ago
  Probably not but, empirically, there are a lot of short tempered people who would.

deleted 10mo ago

[deleted]

TomatoCo 10mo ago

To continue the carpenter analogy, the issue with LLMs is that the shelf looks great but is structurally unsound. That it looks good on surface inspection makes it harder to tell that the person making it had no idea what they're doing.

embedding-shape 10mo ago
Regardless, if a carpenter is not validating their work before selling it, it's the same as if a researcher doesn't validate their citations before publishing. Neither of them have any excuses, and one isn't harder to detect than the other. It's just straight up laziness regardless.
- judofyr 10mo ago
  I think this is a bit unfair. The carpenters are (1) living in world where there’s an extreme focus on delivering as quicklyas possible, (2) being presented with a tool which is promised by prominent figures to be amazing, and (3) the tool is given at a low cost due to being subsidized. And yet, we’re not supposed to criticize the tool or its makers? Clearly there’s more problems in this world than «lazy carpenters»?
  - embedding-shape 10mo ago
    > And yet, we’re not supposed to criticize the tool or its makers? Exactly, they're not forcing anyone to use these things, but sometimes others (their managers/bosses) forced them to. Yet it's their responsibility for choosing the right tool for the right problem, like any other professional. If a carpenter shows up to put a roof yet their hammer or nail-gun can't actually put in nails, who'd you blame; the tool, the toolmaker or the carpenter?
    - judofyr 10mo ago
      > If a carpenter shows up to put a roof yet their hammer or nail-gun can't actually put in nails, who'd you blame; the tool, the toolmaker or the carpenter? I would be unhappy with the carpenter, yes. But if the toolmaker was constantly over-promising (lying?), lobbying with governments, pushing their tools into the hands of carpenters, never taking responsibility, then I would also criticize the toolmaker. It’s also a toolmaker’s responsibility to be honest about what the tool should be used for. I think it’s a bit too simplistic to say «AI is not the problem» with the current state of the industry.

CapitalistCartr 10mo ago

I'm an industrial electrician. A lot of poor electrical work is visible only to a fellow electrician, and sometimes only another industrial electrician. Bad technical work requires technical inspectors to criticize. Sometimes highly skilled ones.

andy99 10mo ago
I’ve reviewed a lot of papers, I don’t consider it the reviewers responsibility to manually verify all citations are real. If there was an unusual citation that was relied on heavily for the basis of the work, one would expect it to be checked. Things like broad prior work, you’d just assume it’s part of background. The reviewer is not a proofreader, they are checking the rigour and relevance of the work, which does not rest heavily on all of the references in a document. They are also assuming good faith.
- zdragnar 10mo ago
  This is half the basis for the replication crisis, no? Shady papers come out and people cite them endlessly with no critical thought or verification. After all, their grant covers their thesis, not their thesis plus all of the theses they cite.
- Aurornis 10mo ago
  > I don’t consider it the reviewers responsibility to manually verify all citations are real I guess this explains all those times over the years where I follow a citation from a paper and discover it doesn’t support what the first paper claimed.
- auggierose 10mo ago
  In short, a review has no objective value, it is just an obstacle to be gamed.
  - amanaplanacanal 10mo ago
    In theory, the review tries to determine if the conclusion reached actually follows from whatever data is provided. It assumes that everything is honest, it's just looking to see if there were mistakes made.

left-struck 10mo ago

It’s like the problem was there all along, all LLMs did was expose it more

criley2 10mo ago
https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Modern science is designed from the top to the bottom to produce bad results. The incentives are all mucked up. It's absolutely not surprising that AI is quickly becoming yet-another factor lowering quality.
theoldgreybeard 10mo ago
Yes, LLMs didnt create the problem they just accelerated it to a speed that beggars belief.

thaumasiotes 10mo ago

> If a scientist uses an LLM to write a paper with fabricated citations - that’s a crappy scientist. Really? Regardless of whether it's a good paper?

zwnow 10mo ago
How is it a good paper if the info in it cant be trusted lmao
- thaumasiotes 10mo ago
  Whether the information in the paper can be trusted is an entirely separate concern. Old Chinese mathematics texts are difficult to date because they often purport to be older than they are. But the contents are unaffected by this. There is a history-of-math problem, but there's no math problem.
  - zwnow 10mo ago
    Not really true nowadays. Stuff in whitepapers needs to be verifiable which is kinda difficult with hallucinations. Whether the students directly used LLMs or just read content online that was produced with them and cited after just shows how difficult these things made gathering information that's verifiable.
    - thaumasiotes 10mo ago
      > Stuff in whitepapers needs to be verifiable which is kinda difficult with hallucinations. That's... gibberish. Anything you can do to verify a paper, you can do to verify the same paper with all citations scrubbed. Whether the citations support the paper, or whether they exist at all, just doesn't have anything to do with what the paper says.
      - zwnow 10mo ago
        I dont think you know how whitepapers work then

hansmayer 10mo ago

Scientists who use LLMs to write a paper are crappy scientists indeed. They need to be held accountable, even ostracised by the scientific community. But something is missing from the picture. Why is it that they came up with this idea in the first place? Who could have been peddling the impression (not an outright lie - they are very careful) about LLMs being these almost sentient systems with emergent intelligence, alleviating all of your problems, blah blah blah. Where is the god damn cure for cancer the LLMs were supposed to invent? Who else is it that we need to keep accountable, scrutinised and ostracised for the ever-increasing mountains of AI-crap that is flooding not just the Internet content but now also penetrating into science, every day work, daily lives, conversations, etc. If someone released a tool that enabled and encouraged people to commit suicide in multiple instances that we know of by now, and we know since the infamous "plandemic" facebook trend that the tech bros are more than happy to tolerate worsening societal conditions in the name of their platform growth, who else do we need to keep accountable, scrutinise and ostracise as a society, I wonder?

the8472 10mo ago
> Where is the god damn cure for cancer the LLMs were supposed to invent? Assuming that cure is meant as hyperbole, how about https://www.biorxiv.org/content/10.1101/2025.04.14.648850v3 https://www.biorxiv.org/content/10.1101/2025.04.14.648850v3 ? AI models being used for bad purposes doesn't preclude them being used for good purposes.
- hansmayer 10mo ago
  ...No, it was not meant as a hyperbole, as we were literally being told that these models will be able to do all of our work. I won't settle for the bullshit incremental wins here and there we see occassionally - I attribute those essentially to the old 'infinite number of monkeys typing on the infinite number of typewriters producing "Crime and Peace". No. that's not it - we were promised a god damn revolution, no less. Again, where is the cure for cancer and post-scarcity society ? Where is the AGI we were promised for the 2025? Let's hold the ghouls promising all that accountable for a change.

Forgeties79 10mo ago

If my calculator gives me the wrong number 20% of the time yeah I should’ve identified the problem, but ideally, that wouldn’t have been sold to me as a functioning calculator in the first place.

imiric 10mo ago
Indeed. The narrative that this type of issue is entirely the responsibility of the user to fix is insulting, and blame deflection 101. It's not like these are new issues. They're the same ones we've experienced since the introduction of these tools. And yet the focus has always been to throw more data and compute at the problem, and optimize for fancy benchmarks, instead of addressing these fundamental problems. Worse still, whenever they're brought up users are blamed for "holding it wrong", or for misunderstanding how the tools work. I don't care. An "artificial intelligence" shouldn't be plagued by these issues.
- SauntSolaire 10mo ago
  > It's not like these are new issues. Exactly, that's why not verifying the output is even less defensible now than it ever has been - especially for professional scientists who are responsible for the quality of their own work.
  - Forgeties79 10mo ago
    If I have to constantly assess every single line done by an LLM then we are fast approaching a point where it’s no longer being helpful and I’m just grading homework for a C student. I’m not saying that isn’t what has to be done, but it kind of clashes with the whole “this will make you more productive” argument if you ask me
- Forgeties79 10mo ago
  > Worse still, whenever they're brought up users are blamed for "holding it wrong", or for misunderstanding how the tools work. I don't care. An "artificial intelligence" shouldn't be plagued by these issues. My feelings exactly, but you’re articulating it better than I typically do ha

belter 10mo ago

"...each of which were missed by 3-5 peer reviewers..." Its sloppy work all the way down...

cindyllm 10mo ago
[dead]

deleted 10mo ago

[deleted]

rectang 10mo ago

“X isn’t the problem, people are the problem.” — the age-old cry of industry resisting regulation.

codywashere 10mo ago
what regulation are you advocating for here?
- kibwen 10mo ago
  At the very least, authors who have been caught publishing proven fabrications should be barred by those journals from ever publishing in them again. Mind you, this is regardless of whether or not an LLM was involved.
  - JumpCrisscross 10mo ago
    > authors who have been caught publishing proven fabrications should be barred by those journals from ever publishing in them again This is too harsh. Instead, their papers should be required to disclose the transgression for a period of time, and their institution should have to disclose it publicly as well as to the government, students and donors whenever they ask them for money.
- rectang 10mo ago
  I’m not advocating, I’m making a high-level observation: Industry forever pushes for nil regulation and blames bad actors for damaging use. But we always have some regulation in the end. Even if certain firearms are legal to own, howitzers are not — although it still takes a “bad actor” to rain down death on City Hall. The same dynamic is at play with LLMs: “Don’t regulate us, punish bad actors! If you still have a problem, punish them harder!” Well yes, we will punish bad actors, but we will also go through a negotiation of how heavily to constrain the use of your technology.
  - codywashere 10mo ago
    so, what regulation do we need on LLMs? the person you originally responded to isn’t against regulation per their comment. I’m not against regulation. what’s the pitch for regulation of LLMs?

only-one1701 10mo ago

Absolutely brutal case of engineering brain here. Real "guns don't kill people, people kill people" stuff.

theoldgreybeard 10mo ago
If you were to wager a guess, what do you think my views on gun rights are?
- only-one1701 10mo ago
  Probably something equally as nuanced and correct as the statement I replied to!
  - theoldgreybeard 10mo ago
    You're projecting.
somehnguy 10mo ago
Your second statement is correct. What about it makes it “engineering brain”?
- rcpt 10mo ago
  If the blame were solely on the user then we'd see similar rates of deaths from gun violence in the US vs. other countries. But we don't, because users are influenced by the UX
- venturecruelty 10mo ago
  Somehow people don't kill people nearly as easily, or with as high of a frequency or social support, in places that don't have guns that are more accessible than healthcare. So weird.

raincole 10mo ago

Given we tacitly accepted replication crisis we'll definitely tacitly accept this.

jodleif 10mo ago

I find this to be a bit “easy”. There is such a thing as bad tools. If it is difficult to determine if the tool is good or bad i’d say some of the blame has to be put on the tool.

photochemsyn 10mo ago

Yeah, I can't imagine not being familiar with every single reference in the bibliography of a technical publication with one's name on it. It's almost as bad as those PIs who rely on lab techs and postdocs to generate research data using equipment that they don't understand the workings of - but then, I've seen that kind of thing repeatedly in research academia, along with actual fabrication of data in the name of getting another paper out the door, another PhD granted, etc. Unfortunately, a large fraction of academic fraud has historically been detected by sloppy data duplication, and with LLMs and similar image generation tools, data fabrication has never been easier to do or harder to detect.

nialv7 10mo ago

Ah, the "guns don't kill people, people kill people" argument. I mean sure, but having a tool that made fabrication so much easier has made the problem a lot worse, don't you think?

theoldgreybeard 10mo ago
Yes I do agree with you that having a tool that gives rocket fuel to a fraud engine should probably be regulated in some fashion. Tiered licensing, mandatory safety training, and weapon classification by law enforcement works really well for Canada’s gun regime, for example.

bigstrat2003 10mo ago

> If a carpenter builds a crappy shelf “because” his power tools are not calibrated correctly - that’s a crappy carpenter, not a crappy tool. It's both. The tool is crappy, and the carpenter is crappy for blindly trusting it. > AI is not the problem, laziness and negligence is. Similarly, both are a problem here. LLMs are a bad tool, and we should hold people responsible when they blindly trust this bad tool and get bad results.

Hammershaft 10mo ago

AI dramatically changes the perceived cost/benefit of laziness and negligence, which is leading to much more of it.

kklisura 10mo ago

> AI is not the problem, laziness and negligence is This reminds me about discourse about a gun problem in US, "guns don't kill people, people kill people", etc - it is a discourse used solely for the purpose of not doing anything and not addressing anything about the underlying problem. So no, you're wrong - AI IS THE PROBLEM.

Yoofie 10mo ago
No, the OP is right in this case. Did you read TFA? It was "peer reviewed". > Worryingly, each of these submissions has already been reviewed by 3-5 peer experts, most of whom missed the fake citation(s). This failure suggests that some of these papers might have been accepted by ICLR without any intervention. Some had average ratings of 8/10, meaning they would almost certainly have been published. If the peer reviewers can't be bothered to do the basics, then there is literally no point to peer review, which is fully independent of the author who uses or doesn't use AI tools.
- smileybarry 10mo ago
  Peer reviewers can also use AI tools, which will hallucinate a "this seems fine" response.
- amrocha 10mo ago
  If AI fraud is good at avoiding detection via peer review that doesn’t mean peer review is useless. If your unit tests don’t catch all errors it doesn’t mean unit tests are useless.
sneak 10mo ago
> it is a discourse used solely for the purpose of not doing anything and not addressing anything about the underlying problem Solely? Oh brother. In reality it’s the complete opposite. It exists to highlight the actual source of the problem, as both industries/practitioners using AI professionally and safely, and communities with very high rates of gun ownership and exceptionally low rates of gun violence exist. It isn’t the tools. It’s the social circumstances of the people with access to the tools. That’s the point. The tools are inanimate. You can use them well or use them badly. The existence of the tools does not make humans act badly.

b00ty4breakfast 10mo ago

maybe the hammer factory should be held responsible for pumping out so many poorly calibrated hammer

venturecruelty 10mo ago
No, because this would cost tens of jobs and affect someone's profits, which are sacrosanct. Obviously the market wants exploding hammers, or else people wouldn't buy them. I am very smart.
SauntSolaire 10mo ago
The obvious solution in this scenario is.. to just buy a different hammer. And in the case of AI, either review its output, or simply don't use it. No one has a gun to your head forcing you to use this product (and poorly at that). It's quite telling that, even in this basic hypothetical, your first instinct is to gesture vaguely in the direction of governmental action, rather than expect any agency at the level of the individual.
- b00ty4breakfast 10mo ago
  >It's quite telling that, even in this basic hypothetical, your first instinct is to gesture vaguely in the direction of governmental action, rather than expect any agency at the level of the individual. When "individuals" (which is a funny way to refer to the global generative AI zeitgeist currently in full binge-mode that is encouraging and enabling this kind of behavior) refuse to regulate themselves, they have to be encouraged through external pressures to do so. Industry is so far up it's own ass wrt AI that all it can see is shit, there is no chance in hell that they will self-regulate. They gladly and indiscriminately slurp up the digital effluent that is currently sliding out the colon of the generative AI super-organism. And, of course, these "individuals" are more than happy to share the consequences with the rest of the world without sharing too much of the corn that they're digging out of the shit. It does not behoove the rest of the world to not protect it's self-interest, to minimize the consequences of foolish and irresponsible generative AI usage and to make sure it gets it's fare share of the semi-digested golden kernels

constantcrying 10mo ago

Absolutely correct. The real issue is that these people can avoid punishment. If you do not care enough about your paper to even verify the existence of citations, then you obviously should not have a job as a scientist. Taking an academic who does something like that seriously, seem impossible. At best he is someone who is neglecting his most basic duties as an academic, at worst he is just a fraudster. In both cases he should be shunned and excluded.

SubiculumCode 10mo ago

Yeah seriously. Using an LLM to help find papers is fine. Then you read them. Then you use a tool like Zotero or manually add citations. I use Gemini Pro to identify useful papers that I might not yet have encountered before. But, even when asking to restrict itself to Pubmed resources, it's citations are wonky, citing three different version sources of the same paper (citations that don't say what they said they'd discuss). That said, these tools have substantially reduced hallucinations over the last year, and will just get better. It also helps if you can restrict it to reference already screened papers. Finally, I'd lke to say tthat if we want scientists to engage in good science, stop forcing them to spend a third of their time in a rat race for funding...it is ridiculously time consuming and wasteful of expertise.

bossyTeacher 10mo ago
The problem isn't whether they have more or less hallucinations. The problem is that they have them. And as long as they hallucinate, you have to deal with that. It doesn't really matter how you prompt, you can't prevent hallucinations from happening and without manual checking, eventually hallucinations will slip under the radar because the only difference between a real pattern and a hallucinated one is that one exists in the world and the other one doesn't. This is not something you can really counter with more LLMs either as it is a problem intrinsic to LLMs
- SubiculumCode 10mo ago
  Humans also hallucinate. We have an error rate. Your argument makes little sense in absolutist terms.
  - bossyTeacher 10mo ago
    > Humans also hallucinate "LLM hallucinations" and hallucinations are essentially different. Human hallucinations are related to perceptual experiences not memory errors like in the case of LLMs. Humans with certain neurological conditions hallucinate. Humans with healthy brains don't. This habit of misapplying terms needs to stop. Humans are not backpropagation algorithms nor whatever random concept you read about in a comp sci book.

deleted 10mo ago

[deleted]

mk89 10mo ago

> we are tacitly endorsing it. We are, in fact, not tacitly but openly endorsing this, due to this AI everywhere madness. I am so looking forward to when some genius in some banks starts to use it to simplify code and suddenly I have 100000000 € on my bank account. :)

jgalt212 10mo ago

fair enough, but carpenters are not being beat over the head to use new-fangled probabilistic speed squares.

grey-area 10mo ago

Generative AI and the companies selling it with false promises and using it for real work absolutely are the problem.

acituan 10mo ago

> AI is not the problem, laziness and negligence is. As much as I agree with you that this is wrong, there is a danger in putting the onus just on the human. Whether due to competition or top down expectations, humans are and will be pressured to use AI tools alongside their work and produce more. Whereas the original idea was for AI to assist the human, as the expected velocity and consumption pressure increases humans are more and more turning into a mere accountability laundering scheme for machine output. When we blame just the human, we are doing exactly what this scheme wants us to do. Therefore we must also criticize all the systemic factors that puts pressure on reversal of AI‘s assistance into AI’s domination of human activity. So AI (not as a technology but as a product when shoved down the throats) is the problem.

alexcdot 10mo ago
Absolutely, expectations and tools given by management are a real problem. If management fires you because they are wrong about how good AI is, and you're right - at the end of the day, you're fired and the manager is in lalaland. People need to actually push the correct calibration of what these tools should be trusted to do, while also trying to work with what they have.

rdiddly 10mo ago

¿Por qué no los dos?

jval43 10mo ago

If a scientist just completely "made up" their references 10 years ago, that's a fraudster. Not just dishonesty but outright academic fraud. If a scientist does it now, they just blame it on AI. But the consequences should remain the same. This is not an honest mistake. People that do this - even once - should be banned for life. They put their name on the thing. But just like with plagiarism, falsifying data and academic cheating, somehow a large subset of people thinks it's okay to cheat and lie, and another subset gives them chance after chance to misbehave like they're some kind of children. But these are adults and anyone doing this simply lacks morals and will never improve. And yes, I've published in academia and I've never cheated or plagiarized in my life. That should not be a drawback.

calmworm 10mo ago

I don’t understand. You’re saying even with crappy tools one should be able to do the job the same as with well made tools?

tedd4u 10mo ago
Three and a half years ago nobody had ever used tools like this. It can't be a legitimate complaint for an author to say, "not my fault my citations are fake it's the fault of these tools" because until recently no such tools were available and the expectation was that all citations are real.
- calmworm 10mo ago
  Then it’s just a poor analogy.

DonHopkins 10mo ago

Shouldn't there be a black list of people who get caught writing fraudulent papers?

cindyllm 10mo ago
[dead]
theoldgreybeard 10mo ago
Probably. Something like that is what I meant by “social consequences”. Perhaps there should be civil or criminal ones for more egregious cases.

nwallin 10mo ago

"Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can’t break."--Bruce Schneier There's a corollary here with LLMs, but I'm not pithy enough to phrase it well. Anyone can create something using LLMs that they, themselves, aren't skilled enough to spot the LLMs' hallucinations. Or something. LLMs are incredibly good at exploiting peoples' confirmation biases. If it "thinks" it knows what you believe/want, it will tell you what you believe/want. There does not exist a way to interface with LLMs that will not ultimately end in the LLM telling you exactly what you want to hear. Using an LLM in your process necessarily results in being told that you're right, even when you're wrong. Using an LLM necessarily results in it reinforcing all of your prior beliefs, regardless of whether those prior beliefs are correct. To an LLM, all hypotheses are true, it's just a matter of hallucinating enough evidence to satisfy the users' skepticism. I do not believe there exists a way to safely use LLMs in scientific processes. Period. If my belief is true, and ChatGPT has told me it's true, then yes, AI, the tool, is the problem, not the human using the tool.

czl 10mo ago
> I do not believe there exists a way to safely use LLMs in scientific processes. What about giving the LLM a narrowly scoped role as a hostile reviewer, while your job is to strengthen the write-up to address any valid objections it raises, plus any hallucinations or confusions it introduces? That’s similar to fuzz testing software to see what breaks or where the reasoning crashes. Used this way, the model isn’t a source of truth or a decision-maker. It’s a stress test for your argument and your clarity. Obviously it shouldn’t be the only check you do, but it can still be a useful tool in the broader validation process.

foxfired 10mo ago

I disagree. When the tool promises to do something, you end up trusting it to do the thing. When Tesla says their car is self driving, people trust them to self drive. Yes, you can blame the user for believing, but that's exactly what they were promised. > Why didn't the lawyer who used ChatGPT to draft legal briefs verify the case citations before presenting them to a judge? Why are developers raising issues on projects like cURL using LLMs, but not verifying the generated code before pushing a Pull Request? Why are students using AI to write their essays, yet submitting the result without a single read-through? They are all using LLMs as their time-saving strategy. [0] It's not laziness, its the feature we were promised. We can't keep saying everyone is holding it wrong. [0]: https://idiallo.com/blog/none-of-us-read-the-specs https://idiallo.com/blog/none-of-us-read-the-specs

rolandog 10mo ago
Very well put. You're promised Artificial Super Intelligence and shown a super cherry-picked promo and instead get an agent that can't hold its drool and needs constant hand-holding... it can't be both things at the same time, so... which is it?

stocksinsmocks 10mo ago

Trades also have self regulation. You can’t sell plumbing services or build houses without any experience or you get in legal trouble. If your workmanship is poor, you can be disciplined by the board even if the tool was at fault. I think fraudulent publications should be taken at least as seriously as badly installed toilets.

venturecruelty 10mo ago

"It's not a fentanyl problem, it's a people problem." "It's not a car infrastructure problem, it's a people problem." "It's not a food safety problem, it's a people problem." "It's not a lead paint problem, it's a people problem." "It's not an asbestos problem, it's a people problem." "It's not a smoking problem, it's a people problem."

SauntSolaire 10mo ago
What an absurd set of equivalences to make regarding a scientist's relationship to their own work. If an engineer provided this line of excuse to me, I wouldn't let them anywhere near a product again - a complete abdication of personal and professional responsibility.

psychoslave 10mo ago

I don't see much crappy power tool provider throwing billions in marketing and product placement to make them used everywhere.