9 ms·

How many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?

LunaSea 7mo ago
It doesn't matter anymore. LLMs just revealed what a decadent society we have setup for ourselves worldwide.
zthrowaway 7mo ago
Do we see this a lot in the US? This seems to be more unique to India.
- duskdozer 7mo ago
  today: https://news.ycombinator.com/item?id=47231189 https://news.ycombinator.com/item?id=47231189
- tw04 7mo ago
  It’s happening A LOT in the US too. Mainstream media just doesn’t seem to find it that newsworthy. https://arstechnica.com/tech-policy/2026/02/randomly-quoting-ray-bradbury-did-not-save-lawyer-from-losing-case-over-ai-errors/ https://arstechnica.com/tech-policy/2026/02/randomly-quoting...
- malshe 7mo ago
  From the article: > In October, two federal judges in the US were called out for the use of AI tools which led to errors in their rulings. In June 2025, the High Court of England and Wales warned lawyers not to use AI-generated case material after a series of cases cited fictitious or partially made up rulings.
coffeefirst 7mo ago
It’s worse than that. We’re hearing about the lawyers and Ars Technica because the consequences are public and the errors are egregious. It’s likely happening to everyone.
duskdozer 7mo ago
I'm continually amazed at how much faith people have in them. I guess since they can sound like people and output really authoritative and confident text it just overrides any skepticism subconsciously?
- pjc50 7mo ago
  The advertising campaign is incredible.
- ben_w 7mo ago
  Much as I like them, I do frequently remind myself of two things: 1) https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans 2) https://archive.org/details/nextgen-issue-26 https://archive.org/details/nextgen-issue-26 as an example of how in the 90s we has rapid cycles of a new tech (3d graphics) astounding us with how realistic each new generation was compared to the previous one, and forgetting with each new (game engine) how we'd said the same and felt the same about (graphics) we now regarded as pathetic. So yes, they do sound "authoritative and confident text it just overrides any skepticism subconsciously", but you shouldn't be amazed, we've always been like this.
- moron4hire 7mo ago
  It's mind boggling how much people claim to like LLMs when you would never design any other piece of software to operate like LLMs do. Designing a system that interact with the user through natural text creates an awful experience. It slows down every interaction as you dig through all the prose to get to the key information. It turns every computer interaction into a school math word problem.
- PunchyHamster 7mo ago
  Yes, just as with politicians. And LLMs have been thoroughly tuned to appear that
- direwolf20 7mo ago
  https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect
Latty 7mo ago
It doesn't matter, because any process that seems right most of the time but occasionally is wrong in subtle, hard to spot ways is basically a machine to lull people into not checking, so stuff will always slip through. It's just like the cars driving themselves but you need to be able to jump in if there is a mistake, humans are not going to react as fast as if they were driving, because they aren't going to be engaged, and no one can stay as engaged as they were when they were doing it themselves. We need to stop pretending we can tell people they "just" need to check things from LLMs for accuracy, it's a process that inevitably leads to people not checking and things slipping through. Pretending it's the people's fault when essentially everyone using it would eventually end up doing that is stupid and won't solve the core problem.
- voidUpdate 7mo ago
  Probably worth including a "bibliography" section of citations that can be automatically checked that they actually exist then
  - lazide 7mo ago
    Not enough - you’d also need to check that they say/mean what is being implied. Which is a real problem.
    - mminer237 7mo ago
      To be fair, that's a problem with human authors too. Wikipedia is really well-cited, but it's common to check a citation and find it only says half of what a sentence does, while the rest seemingly has no basis in fact. Judges are supposed to actually read the citations to not only confirm the case exists and says what's being claimed, but often to also compare & contrast the situations to ensure that principle is applicable to the case at hand.
      - lazide 7mo ago
        Yup. The issue with LLMs are not that any specific thing it is doing is unique. Rather that it does it in previously unimaginable volume, scale, and accessibility.
YeGoblynQueenne 7mo ago
What kind of AI is this that you constantly need a human to check its job? Do you think Jean-Luc Piccard had to constantly check the output of the Enterprise computer? No he didn't. If AI is not better than humans, then what the heck is the point? You might as well just use humans.
AJ007 7mo ago
This whole thing is silly, LLMs can automate reference validation. If someone is a lawyer, accountant, doctor, teacher, surgeon, engineer etc, and is regurgitating answers that were pumped out with with GPT-5-extra-low or whatever mediocre throttled model they are using, they should just be fired and de-credentialed. Right now this is easy. The real problem is ahead: 99.999% of future content that exists will be made using generative AI. For many people using Facebook, Instagram, TikTok, or some other non-sequential, engagement weighted feed, 50%+ of the content they consume today is fake. As that stuff spreads in to modern culture it's going to be an endless battle to keep it out of stuff that should not be publishing fake content (e.g. the New York Times or Wall Street Journal; excluding scientific journals who seem to abandoned validation and basic statistics a long time ago.) Much of the future value and profit margins might just be in valid data?
- raincole 7mo ago
  > Right now this is easy. Easy? In the US you need house impeachment to fire a judge. In some countries judges are completely immune unless they are sentenced for crimes.
  - voidUpdate 7mo ago
    Do you need impeachment to fire a lawyer, accountant, doctor, teacher, surgeon or engineer?
    - raincole 7mo ago
      Nope, and the article is about a judge. What's the point to incentive lawyers to carefully verify their references when they know the judge has no incentive to read them and can just make shit up anyway?
  - mminer237 7mo ago
    To fire a federal judge. Local judges, which are the vast majority, can be fired by their colleagues or replaced in elections.
- 7mo ago
probably_wrong 7mo ago
Just this week I tracked down the citations of a scientific paper (whose authors could very well be here) where 25% of the citations were made up and 50% of the remaining ones were wrong, taking ArXiv papers and citing them as belonging to (say) IJCLR. It's not just lawyers.

LunaSea 7mo ago

It doesn't matter anymore. LLMs just revealed what a decadent society we have setup for ourselves worldwide.

zthrowaway 7mo ago

Do we see this a lot in the US? This seems to be more unique to India.

duskdozer 7mo ago
today: https://news.ycombinator.com/item?id=47231189 https://news.ycombinator.com/item?id=47231189
tw04 7mo ago
It’s happening A LOT in the US too. Mainstream media just doesn’t seem to find it that newsworthy. https://arstechnica.com/tech-policy/2026/02/randomly-quoting-ray-bradbury-did-not-save-lawyer-from-losing-case-over-ai-errors/ https://arstechnica.com/tech-policy/2026/02/randomly-quoting...
malshe 7mo ago
From the article: > In October, two federal judges in the US were called out for the use of AI tools which led to errors in their rulings. In June 2025, the High Court of England and Wales warned lawyers not to use AI-generated case material after a series of cases cited fictitious or partially made up rulings.

coffeefirst 7mo ago

It’s worse than that. We’re hearing about the lawyers and Ars Technica because the consequences are public and the errors are egregious. It’s likely happening to everyone.

duskdozer 7mo ago

I'm continually amazed at how much faith people have in them. I guess since they can sound like people and output really authoritative and confident text it just overrides any skepticism subconsciously?

pjc50 7mo ago
The advertising campaign is incredible.
ben_w 7mo ago
Much as I like them, I do frequently remind myself of two things: 1) https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans 2) https://archive.org/details/nextgen-issue-26 https://archive.org/details/nextgen-issue-26 as an example of how in the 90s we has rapid cycles of a new tech (3d graphics) astounding us with how realistic each new generation was compared to the previous one, and forgetting with each new (game engine) how we'd said the same and felt the same about (graphics) we now regarded as pathetic. So yes, they do sound "authoritative and confident text it just overrides any skepticism subconsciously", but you shouldn't be amazed, we've always been like this.
moron4hire 7mo ago
It's mind boggling how much people claim to like LLMs when you would never design any other piece of software to operate like LLMs do. Designing a system that interact with the user through natural text creates an awful experience. It slows down every interaction as you dig through all the prose to get to the key information. It turns every computer interaction into a school math word problem.
PunchyHamster 7mo ago
Yes, just as with politicians. And LLMs have been thoroughly tuned to appear that
direwolf20 7mo ago
https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect

Latty 7mo ago

It doesn't matter, because any process that seems right most of the time but occasionally is wrong in subtle, hard to spot ways is basically a machine to lull people into not checking, so stuff will always slip through. It's just like the cars driving themselves but you need to be able to jump in if there is a mistake, humans are not going to react as fast as if they were driving, because they aren't going to be engaged, and no one can stay as engaged as they were when they were doing it themselves. We need to stop pretending we can tell people they "just" need to check things from LLMs for accuracy, it's a process that inevitably leads to people not checking and things slipping through. Pretending it's the people's fault when essentially everyone using it would eventually end up doing that is stupid and won't solve the core problem.

voidUpdate 7mo ago
Probably worth including a "bibliography" section of citations that can be automatically checked that they actually exist then
- lazide 7mo ago
  Not enough - you’d also need to check that they say/mean what is being implied. Which is a real problem.
  - mminer237 7mo ago
    To be fair, that's a problem with human authors too. Wikipedia is really well-cited, but it's common to check a citation and find it only says half of what a sentence does, while the rest seemingly has no basis in fact. Judges are supposed to actually read the citations to not only confirm the case exists and says what's being claimed, but often to also compare & contrast the situations to ensure that principle is applicable to the case at hand.
    - lazide 7mo ago
      Yup. The issue with LLMs are not that any specific thing it is doing is unique. Rather that it does it in previously unimaginable volume, scale, and accessibility.

YeGoblynQueenne 7mo ago

What kind of AI is this that you constantly need a human to check its job? Do you think Jean-Luc Piccard had to constantly check the output of the Enterprise computer? No he didn't. If AI is not better than humans, then what the heck is the point? You might as well just use humans.

AJ007 7mo ago

This whole thing is silly, LLMs can automate reference validation. If someone is a lawyer, accountant, doctor, teacher, surgeon, engineer etc, and is regurgitating answers that were pumped out with with GPT-5-extra-low or whatever mediocre throttled model they are using, they should just be fired and de-credentialed. Right now this is easy. The real problem is ahead: 99.999% of future content that exists will be made using generative AI. For many people using Facebook, Instagram, TikTok, or some other non-sequential, engagement weighted feed, 50%+ of the content they consume today is fake. As that stuff spreads in to modern culture it's going to be an endless battle to keep it out of stuff that should not be publishing fake content (e.g. the New York Times or Wall Street Journal; excluding scientific journals who seem to abandoned validation and basic statistics a long time ago.) Much of the future value and profit margins might just be in valid data?

raincole 7mo ago
> Right now this is easy. Easy? In the US you need house impeachment to fire a judge. In some countries judges are completely immune unless they are sentenced for crimes.
- voidUpdate 7mo ago
  Do you need impeachment to fire a lawyer, accountant, doctor, teacher, surgeon or engineer?
  - raincole 7mo ago
    Nope, and the article is about a judge. What's the point to incentive lawyers to carefully verify their references when they know the judge has no incentive to read them and can just make shit up anyway?
- mminer237 7mo ago
  To fire a federal judge. Local judges, which are the vast majority, can be fired by their colleagues or replaced in elections.
7mo ago

probably_wrong 7mo ago

Just this week I tracked down the citations of a scientific paper (whose authors could very well be here) where 25% of the citations were made up and 50% of the remaining ones were wrong, taking ArXiv papers and citing them as belonging to (say) IJCLR. It's not just lawyers.