6 ms·

California issues fine over lawyer's ChatGPT fabrications

167 points by geox 1y ago

lordnacho 1y ago
This is why there are certain jobs AI can never take: we are wired for humans to be responsible. Even though a pilot can do a lot of his work via autopilot, we need a human to be accountable. For the pilot, that means sitting in the plane. But there are plenty of other jobs, mostly high-earning experts, where we need to be able to place responsibility on a person. For those jobs, the upside is that the tool will still be available for the expert to use and capture the benefits from. This lawyer fabricating his filings is going to be among the first in a bunch of related stories: devs who check in code they don't understand, doctors diagnosing people without looking, scientists skipping their experiments, and more.
- unshavedyak 1y ago
  > This is why there are certain jobs AI can never take You're thinking too linearly imo. Your examples are where AI will "take", just perhaps not entirely replace. Ie if liability is the only thing stopping them from being replaced, what's stopping them from simply assuming more liability? Why can't one lawyer assume the liability of ten lawyers?
  - lordnacho 1y ago
    Then there will still be lawyers. More productive, higher income lawyers. Just like with a lot of other jobs that got more productive.
  - observationist 1y ago
    People who think like this cannot be convinced; they're unaware of the acceleration of the rate of progress, and it won't change until they clash with reality. Don't waste your time and energy trying to convince them. They don't understand how to calibrate their model of the world with the shape of future changes. The gap between people who've been paying attention and those who haven't is going to increase, and the difficulty in explaining what's coming is going to keep rising, because humans don't do well with nonlinearities. The robots are here. The AI is here. The future is now, it's just not evenly distributed, and by the time you've finished arguing or explaining to someone what's coming, it'll have already passed, and something even weirder will be hurtling towards us even faster than whatever they just integrated. Sometime in the near future, there won't be much for people to do but stand by in befuddled amazement and hope the people who set this all in motion knew what they were doing (because if we're not doing that, we're all toast anyway.)
- pjc50 1y ago
  The book https://en.wikipedia.org/wiki/The_Unaccountability_Machine https://en.wikipedia.org/wiki/The_Unaccountability_Machine introduces the term "accountability sink", which is very useful for these discussions. Increasingly complicated systems generate these voids, where ultimately no human can be singled out or held responsible. AI offers an incredible caveat emptor tradeoff: you can get a lot more done more quickly, so long as you don't care about the quality of the work, and cannot hold anyone responsible for that quality.
gdulli 1y ago
Where could lawyers be learning this behavior? https://www.theguardian.com/us-news/2025/apr/24/california-bar-exam-ai https://www.theguardian.com/us-news/2025/apr/24/california-b...
abeppu 1y ago
> In recent weeks, she’s documented three instances of judges citing fake legal authority in their decisions. So lawyers use it, judges use it ... have we seen evidence of lawmakers submitting AI-generated language in bills or amendments?
- deleted 1y ago
  [deleted]
- dylan604 1y ago
  I mean, we've seen laws that were written by lobbyists with zero changes. Does it matter if it was AI generated or not at that point? The congress critters are not rewriting what they've been told to do if they've even read it after being told what to do.
- milesvp 1y ago
  I know a lawyer who almost took a job in state government where one of the primary duties was to make sure that the punctuation in the bills going through the state legislature were correct and accurate. For her, part of the appeal of the job, was that it would allow her to subtly alter the meaning of a bill being presented. Apparently it is a non trivial skill to be able to determine how judges are likely to rule on cases due to say, the presence, or the absence of an oxford comma. There was an entire team dedicated to this work, and the hours were insane when the legislature was in session. She ended up not taking the job because of the downsides associated with moving to the capital, so I don't know more about the job. I'd be curious how much AI has changed what that team does now. Certainly, they still would want to meticulously look at every character, but it is certainly possible that AI has gotten better at analyzing the "average" ruling, which might make the job a little easier. What I know about law though, is that it's often defined by the non average ruling, that there's sort of a fractal nature to it, and it's the unusual cases that often forever shape future interpretations of a given law. Unusual scenarios are something that LLMs generally struggle with, and add to that the need to creatively come up with scenarios that might further distort the bill, and I'd expect LLMs to be patently bad at creating laws. So while, I have no doubt that legislators (and lobbyists) are using AI to draft bills, I am positive that there is still a lot of work that goes into refining bills, and we're probably not seeing straight vibe drafting.
ratelimitsteve 1y ago
The fact that they interviewed someone named "Charlotin" for this article is a coincidence I can't ignore
FlamingMoe 1y ago
$10k is not a high enough fine to dissuade this behavior. Large law firms will just approach this the same way that big tech accepts GDPR fines as part of the cost of doing business.
amoshebb 1y ago
I'm numb to it after many "EU fines Householdnamecorp a zillion doubloons" type headlines, but using "historic fine" to describe $10k to a lawyer feels odd.
- dylan604 1y ago
  expecting the same level of fine to an individual person as opposed to a faceless corp really shows how numb you must be. for an attorney to be fined that much is not normal. TFA even shows example of higher fines issued to law firms, while still not as high as your zillion doubloons hyberbole it still shows the distinction between individual and s/corporation/law firm/. EU fines have been progressively getting higher especially for repeat offenders. it would be unwise to expect different in legal matters
  - amoshebb 1y ago
    His website makes him look like the owner of a law firm, although I think it's just him? I'm not expecting the same number, but... california issues bigger fines for watering lawns or buying illegal fireworks. For a lawyer, a fine order of magnitude smaller than "hiring a paralegal" is less "historic" and more "cost of doing business, don't get caught"
    - dylan604 1y ago
      California issues higher fines for littering than for abandoning an animal on the highway. It's listed right there on the highway signs as you enter the state. It stood out when I saw it for the first time. Fines are arbitrary numbers set by some people not necessarily knowing about other fines for other offenses.
- grues-dinner 1y ago
  > The fine appears to be the largest issued over AI fabrications by a California court This is a bit like all the stats like "this is appears to be an unprecedented majority in the last 10 years in a Vermont county starting with G for elections held on the 4th when no candidate is from Oklahoma". Lots of things are historic but that doesn't necessarily mean they're impressive overall. More interesting is how many of these cases have already been tried such that this isn't "historic" for being the first one decided.
jsbisviewtiful 1y ago
Doesn't even feel controversial. LLMs hallucinate and in law that's not acceptable. Increase the fine though to really punish those doing this.
- abirch 1y ago
  Hopefully the fines grow exponentially for repeat offenders. Seems like now, lawyers can use AI to DDOS the other counsel.
donatj 1y ago
$10,000? That's a slap on the wrist. I don't say this lightly, this should have been jail time for someone. You're making a mockery of our most sacred institutions.
- dylan604 1y ago
  for first offenses for something like this you'd suggest jail time? hope you find excuses to skip your next jury summons. however, it is typical in jury selection to be asked by the defense if you'd be able to agree with a minimum sentence while the prosecutors like to ask if you'd be able to agree to the maximum. personally was asked if I could agree to 99 years for someone's first offense of GTA. I said no, and was dismissed. Sounds like you'd have said yes.
  - quickthrowman 1y ago
    For someone who had to attend 6+ years of school and had to pass a professional licensing exam with ethics questions? Yes, I do. $10,000 is one week of billable hours at $250/hr. Do you think a Civil Engineer (PE) should be held liable if they vibe engineered a bridge using an LLM without reviewing the output? For this hypothetical, let’s assume an inspector caught the issue before the bridge was in use, but it would’ve collapsed had the inspector not noticed.
    - dylan604 1y ago
      No single civil engineer designs a bridge though now do they? So the premise of your retort is just way off here. No bridge plan is made without reviews after one person presses print on the plotter. Even the construction company hired to build the bridge will review the plans before they break ground. If someone is building a bridge on their private property and hires their nephew, that's on them. An actual civil project, nope, I reject your premise outright.
      - quickthrowman 1y ago
        I would wager that plenty of bridges are designed by a single engineer, most bridges are not massive 8-lane highway bridges but small bridges in municipalities. A single person can design a building, why not a bridge? P.S. I sell and run commercial construction work
jerlam 1y ago
I searched for the lawyer's name in the state bar association, they've been practicing for over 13 years. Even has electrical engineering in their background.
- anigbrowl 1y ago
  And?
SilverElfin 1y ago
> He thinks it is unrealistic to expect lawyers to stop using AI. It’s become an important tool just as online databases largely replaced law libraries and, until AI systems stop hallucinating fake information, he suggests lawyers who use AI to proceed with caution. I think this is a good reason for fines to not be incredibly big. People are using AI all the time. There will be a growing period until they learn of its limitations.
- dylan604 1y ago
  The slope of the learning curve can be adjusted though with the level of fines being a big adjustment. Only I'd suggest not the first time for any one lawyer per firm, but per incident for each firm.
- jmonty900 1y ago
  [dead]
sidewndr46 1y ago
$10,000, wow what a shocker! I don't know how anyone who can afford to live in California could ever expect to pay such a fine. I expect the lawyer will soon have to declare bankruptcy.
- bluGill 1y ago
  Most lawyers don't make great income. This could be substantial for him if he has to pay it personally.
- noisy_boy 1y ago
  It the first of its kind and I would wager is more of a warning shot. 10k on a case may not be much but tens of thousands multiplied by hundreds of cases won't be negligible for a lawyer or a small firm. Not to mention the prospect of loss of case and reputation and consequently future business due to such actions.
jerf 1y ago
Do lawyers and judges not by now have software that turns all these citations into hyperlinks into some relevant database? Software that would also flag a citation as not having a referent? Surely this exists and is expensive but in wide usage...? It's not a large step after that to verify that a quote actually exists in the cited document, though I can see how perhaps that was not something that was necessary up to this point. I have to think the window on this being even slightly viable is going to close quickly. When you ship something to a judge and their copy ends up festooned with "NO REFERENT" symbols it's not going to go well for you.
- KittenInABox 1y ago
  Part of an issue is that there's already in existence a lot of manual entry and a lot of small/regional courts with a lot of specificity for individual jurisdictions. Unification of standards is a long way away, I mean, tech hasn't even done it
- freejazz 1y ago
  >Do lawyers and judges not by now have software that turns all these citations into hyperlinks into some relevant database? Software that would also flag a citation as not having a referent? Surely this exists and is expensive but in wide usage...? Why would I pay for software what I could do with my own eyes in 2 minutes?
  - Dylan16807 1y ago
    Two minutes per what? Two minutes per citation is a huge time waste. Two minutes per filing full of citations is unrealistically fast and also adds up.
    - freejazz 1y ago
      It's not two minutes per citation and the notion that something that is "very expensive" but otherwise doesn't have a use case unless you're using AI in the first place is a ridiculous proposition. To call what I wrote a "huge time waste" seems to really miss the point. The citations are already hyperlinks into "some relevant database". Attorneys figured that out when they started publishing judicial opinions in reporters (books). I take the citation string and it gives me the decision. Wow! > Two minutes per filing full of citations is unrealistically fast and also adds up Also adds up to what? The amount of time it takes to do your job? Vs not doing it and just letting AI rip? Oh I see, people here think that paying another service to fix the poor job the "do your job for you" AI service did makes sense? I don't think so. But then again I'm not the one wondering if lawyers keep cases stored in "relevant databases" out of what could only be sheer ignorance (I bet AI could get that one right). Never heard of Westlaw or LexisNexis?
alwa 1y ago
> Mostafavi told CalMatters he wrote the appeal and then used ChatGPT to try and improve it. He said that he didn’t know it would add case citations or make things up. He thinks it is unrealistic to expect lawyers to stop using AI. [...] “In the meantime we’re going to have some victims, we’re going to have some damages, we’re going to have some wreckages,” he said. “I hope this example will help others not fall into the hole. I’m paying the price.” Wow. Seems like he really took the lesson to heart. We're so helpless in the face of LLM technology that "having some victims, having some damages" (rather than reading what you submit to the court) is the inevitable price of progress in the legal profession? 21 of 23 citations are fake, and so is whatever reasoning they purport to support, and that's casually "adding some citations"? I sometimes use tools that do things I don't expect, but usually I'd like to think I notice when I check their work... if there were 2 citations when I started, and 23 when I finished, I'd like to think I'd notice.
- bawolff 1y ago
  > He thinks it is unrealistic to expect lawyers to stop using AI Sure. Its also unrealistic to expect nobody to murder anyone. That's why we invented jail.
- OptionOfT 1y ago
  > He thinks it is unrealistic to expect lawyers to stop using AI. I disagree. It worked until now, and using AI is clearly doing more harm than good, especially in situations where you hire an expert to help you. Remember, a lawyer is someone who actually has passed a bar exam, and with that there is an understanding that whatever they sign, they validate as correct. The fact that they used AI here actually isn't the worst. The fact that they blindly signed it afterwards is a sign that they are unfit to be a lawyer. We can make the argument that this might be pushed from upper management, but remember, the license is personal. So it's not that they can hide behind such a mandate. It's the same discussions I'm having with colleagues about using AI to generate code, or to review code. At a certain moment there is pressure to go faster, and stuff gets committed without a human touching it. Until that software ends up on your glucose pump, or the system used to radiate your brain tumor.
  - 1y ago
nerdjon 1y ago
This is just incredibly defeatist on everyone talking here. Here we have irrefutable evidence of just how bad the result of using AI would be and yet... the response is just that we need to accept that there is going to be damage but keep using it?!? This isn't a tech company that "needs" to keep pushing AI because investors seem to think it is the only path of the future. There is absolutely zero reason to keep trying to shoehorn this tech in places it clearly doesn't belong. We don't need to accept anything here. Just don't use it... why is that such a hard concept.
- jihadjihad 1y ago
  It's not (just) defeatist, it's fatalistic. And I agree. There can be cultural underpinnings here, too--the attorney is CA-based, and though LA is distinct from SF, I wonder if there isn't a thread of the "move fast and break things" ethos showing up as well.
- mullingitover 1y ago
  > Here we have irrefutable evidence of just how bad the result of using AI would be It's not just the result of using AI, it's the result of failing to vet the information he was providing the court. The same thing could've happened if he hired a paralegal from Fiverr to write his pleadings and didn't check their work. It's like saying that because he typed it on a computer, it's the computers that are the problem, and we shouldn't keep using them. We're already at least a year past AI tools having the ability to perform grounding (Lexis+ from LexisNexis, as I cited on another comment in this post, for example), so this whole fiasco is already something from a bygone era.
- freejazz 1y ago
  I'm a lawyer and I do not use AI. I was given a product test of an AI legal solution and it was terrible.
- themafia 1y ago
  Whenever an "AI" article is posted here the comments are heavily astroturfed.
- s1artibartfast 1y ago
elpakal 1y ago
A lot of both defeatist and overly optimistic pro AI comments in here. Having built legaltech and interfaced heavily with attorneys over the last 5 years of my career, I will say that there is a wide spectrum of experience, ethics and intelligence in the field. Blindly copying output from anything and submitting it to the court seems like a mind boggling move, it doesn’t really make a difference if it was AI or Google or Bing or Thompson Reuters. This attorney is not representative of the greater population and probably had it coming imho. There is definitely benefit to using language models correctly in law, but they are different than most users in that their professional reputation is at stake wrt the output being created and the risk of adoption is always going to be greater for them.
buildbot 1y ago
Everyone is somewhat missing the point here that the California bar is making. They don't care if you use an AI or a Llama spitting at a board of letters to assemble your case, you are responsible for what you submit to the court. This is just a bad lawyer who probably didn't check their work in many other cases, and AI just enabled that bad behavior further.
- stockresearcher 1y ago
  The American Bar Association, as well as every state bar association, published guidance on GenAI usage at least a year ago. The existing legal and ethical responsibilities go beyond just being responsible for hallucinations. Client information privacy, accurately tracking time and billing only for time actually spent on the case, etc. As time goes on, it becomes less and less defensible to see this stuff.
- pdabbadabba 1y ago
  This. Lawyers can use AI tools. But an attorney is ultimately responsible for everything that goes out the door. So they had better check the output from their AI tool carefully. In most firms (at least most good ones) someone would check every citation that goes out the door, even if it was written by an experienced attorney, so it speaks volumes that someone would fail to check citations generated by an LLM.
ElijahLynn 1y ago
Does anyone have a source of what the citations were? I didn't see them mentioned in the article.
- SloopJon 1y ago
  The article links to the opinion[1], which notes more than once that "the quoted language does not appear anywhere in the opinion," and "Goldstine appears to be a fabricated case." I don't know whether it's easy to get a copy of the complaint in question. [1] https://www4.courts.ca.gov/opinions/documents/B331918.PDF https://www4.courts.ca.gov/opinions/documents/B331918.PDF
rdtsc 1y ago
Wonder what the State Bar of CA would have to say about this: https://apps.calbar.ca.gov/attorney/Licensee/Detail/282372 https://apps.calbar.ca.gov/attorney/Licensee/Detail/282372 Doesn't seems like there isn't kind of disciplinary action. You can just make up stuff and if you're caught, pay some pocket change (in lawyer money level territory) and move on.
- patrickhogan1 1y ago
  Fines above $1k must be reported to state bar in CA. So they will know about this one.
us0r 1y ago
An OnlyFans case that is ongoing now the plaintiffs attorneys made a filling with i believe entirely fabricated case references: https://www.courtlistener.com/docket/68990373/nz-v-fenix-international-limited/?filed_after=&filed_before=&entry_gte=&entry_lte=&order_by=desc https://www.courtlistener.com/docket/68990373/nz-v-fenix-int...
- yieldcrv 1y ago
  I just read the initial complaint, what do you think about that case? Is there a community that wants disclosure of "chatter's" existence? It seems to be going the other way with AI personalities doing the chatting
- narrator 1y ago
  The hallucinations in legal briefs get really out of hand when the attorney wants to make an argument not supported by the case law. The LLM wants to do a good job defending a case, so it invents the legal precident, because otherwise it'd be impossible to make the argument credibly. This invites a rule 11 challenge from the other side where you claim the lawyer is so full of crap with his claim that he deserves sanction for not understanding the law and wasting everyone's time. What's interesting about the rules of civil procedure is that it has been built up over centuries to prevent all kinds of abuse by sneaky, clever, unscrupulous litigants. Most systems are not so hardened against bad faith actors like the legal system is and AI just thinks it can pathologically lie its way through cause most people trust somebody who sounds authoritative.
pessimizer 1y ago
AI constantly fabricates references. I found a question recently (accidentally) which would make it fabricate every answer. Otherwise, it tends to only fabricate about a third of them, and somehow miss dozens of others that it should easily find. After asking for recommendations, I always immediately ask it if any are hallucinations. It then tells me a bunch of them are, then goes "Would you like more information about how LLMs "hallucinate," and for us to come up with an architecture that could reduce or eliminate the problem?" No, fake dude, I just want real books instead of imaginary ones, not to hear about your problems. detail: The question was to find a book that examined how the work of a short order cook is done in detail, or any book with a section covering this. I started the question by mentioning that I already had Fast Foods and Short Order Cooking(1984) by Popper et al. and that was the best I had found so far. It gave me about a half dozen great hallucinations. You can try the question and see how it works for you. They're so dumb. Our economy is screwed.
mullingitover 1y ago
It's fun to point and laugh in this scenario where the attorney just threw slop at the court. However, these stories won't be around long. Tools like Lexis+ from LexisNexis[1] now offer grounding, so it won't be as simple to bust people cutting corners in the future because these prevent the hallucinations. We're now closer to the real Butlerian Jihad when we see pro se plaintiffs/defendants winning cases regularly. [1] https://www.lexisnexis.com/blogs/en-au/insights/hallucination-free-linked-legal-citations https://www.lexisnexis.com/blogs/en-au/insights/hallucinatio...
- JambalayaJimbo 1y ago
  How will pro se defendants afford LexisNexis??
epa 1y ago
$10,000 is nothing. Should be $200,000+.
- deleted 1y ago
  [deleted]
dmitrygr 1y ago
The fact that the lawyer is unrepentant and says that it is simply too convenient and "there will be some victims, tough shit", basically, means the fine was a zero or two too short. Deterrence needs to hurt!
jqpabc123 1y ago
There you have it --- proof that lawyers love AI --- in more ways than this one example illustrates. Using a tool that is widely known to be flawed to provide any sort of professional service (legal, medical, accounting, engineering, banking, etc.) is pretty much a textbook definition of negligence. And lawyers just love negligence.
fooker 1y ago
LLMs are just not the technology for 'search' and cite references, no matter how 'agentic' you make it. They are great for a large number of other things, like generating code for example. We are so focused trying to shoehorn LLMs to do something it is fundamentally unsuitable for that we are probably missing the discovery of the technology that would solve this.
narrator 1y ago
The legal system is not an API you can spam. You must have credentials that involve passing a very tough exam to even submit to it and going to school for 3 years. If you make a submission that is not fully syntax and reference checked you can face a $10,000 fine and suspension of your license to submit to it. You do not waste the legal system's time with AI slop. I have worked in litigation and if you submit some litigation to a court clerk that is not perfect you won't even get a reply that it's no good. They'll just throw your brief in the trash without looking at it and never call you back.
stewBedasso 1y ago
free bro
BrenBarn 1y ago
$10k is a slap on the wrist. He should be disbarred.
smsm42 1y ago
Just 10k? And this is presented as historically huge fine? I don't know about thst particulat lawyer, but I heard lawyers charging multiple hundreds of dollars per hour, so that's what, less than week's wages? What kind of crappy deterrent is that? He should have been banned from court for a couple of years at least, if he repeats - forever.

167 points by geox 1y ago

This is why there are certain jobs AI can never take: we are wired for humans to be responsible. Even though a pilot can do a lot of his work via autopilot, we need a human to be accountable. For the pilot, that means sitting in the plane. But there are plenty of other jobs, mostly high-earning experts, where we need to be able to place responsibility on a person. For those jobs, the upside is that the tool will still be available for the expert to use and capture the benefits from. This lawyer fabricating his filings is going to be among the first in a bunch of related stories: devs who check in code they don't understand, doctors diagnosing people without looking, scientists skipping their experiments, and more.

unshavedyak 1y ago
> This is why there are certain jobs AI can never take You're thinking too linearly imo. Your examples are where AI will "take", just perhaps not entirely replace. Ie if liability is the only thing stopping them from being replaced, what's stopping them from simply assuming more liability? Why can't one lawyer assume the liability of ten lawyers?
- lordnacho 1y ago
  Then there will still be lawyers. More productive, higher income lawyers. Just like with a lot of other jobs that got more productive.
- observationist 1y ago
  People who think like this cannot be convinced; they're unaware of the acceleration of the rate of progress, and it won't change until they clash with reality. Don't waste your time and energy trying to convince them. They don't understand how to calibrate their model of the world with the shape of future changes. The gap between people who've been paying attention and those who haven't is going to increase, and the difficulty in explaining what's coming is going to keep rising, because humans don't do well with nonlinearities. The robots are here. The AI is here. The future is now, it's just not evenly distributed, and by the time you've finished arguing or explaining to someone what's coming, it'll have already passed, and something even weirder will be hurtling towards us even faster than whatever they just integrated. Sometime in the near future, there won't be much for people to do but stand by in befuddled amazement and hope the people who set this all in motion knew what they were doing (because if we're not doing that, we're all toast anyway.)
pjc50 1y ago
The book https://en.wikipedia.org/wiki/The_Unaccountability_Machine https://en.wikipedia.org/wiki/The_Unaccountability_Machine introduces the term "accountability sink", which is very useful for these discussions. Increasingly complicated systems generate these voids, where ultimately no human can be singled out or held responsible. AI offers an incredible caveat emptor tradeoff: you can get a lot more done more quickly, so long as you don't care about the quality of the work, and cannot hold anyone responsible for that quality.

gdulli 1y ago

Where could lawyers be learning this behavior? https://www.theguardian.com/us-news/2025/apr/24/california-bar-exam-ai https://www.theguardian.com/us-news/2025/apr/24/california-b...

abeppu 1y ago

> In recent weeks, she’s documented three instances of judges citing fake legal authority in their decisions. So lawyers use it, judges use it ... have we seen evidence of lawmakers submitting AI-generated language in bills or amendments?

deleted 1y ago
[deleted]
dylan604 1y ago
I mean, we've seen laws that were written by lobbyists with zero changes. Does it matter if it was AI generated or not at that point? The congress critters are not rewriting what they've been told to do if they've even read it after being told what to do.
milesvp 1y ago
I know a lawyer who almost took a job in state government where one of the primary duties was to make sure that the punctuation in the bills going through the state legislature were correct and accurate. For her, part of the appeal of the job, was that it would allow her to subtly alter the meaning of a bill being presented. Apparently it is a non trivial skill to be able to determine how judges are likely to rule on cases due to say, the presence, or the absence of an oxford comma. There was an entire team dedicated to this work, and the hours were insane when the legislature was in session. She ended up not taking the job because of the downsides associated with moving to the capital, so I don't know more about the job. I'd be curious how much AI has changed what that team does now. Certainly, they still would want to meticulously look at every character, but it is certainly possible that AI has gotten better at analyzing the "average" ruling, which might make the job a little easier. What I know about law though, is that it's often defined by the non average ruling, that there's sort of a fractal nature to it, and it's the unusual cases that often forever shape future interpretations of a given law. Unusual scenarios are something that LLMs generally struggle with, and add to that the need to creatively come up with scenarios that might further distort the bill, and I'd expect LLMs to be patently bad at creating laws. So while, I have no doubt that legislators (and lobbyists) are using AI to draft bills, I am positive that there is still a lot of work that goes into refining bills, and we're probably not seeing straight vibe drafting.

ratelimitsteve 1y ago

The fact that they interviewed someone named "Charlotin" for this article is a coincidence I can't ignore

FlamingMoe 1y ago

$10k is not a high enough fine to dissuade this behavior. Large law firms will just approach this the same way that big tech accepts GDPR fines as part of the cost of doing business.

amoshebb 1y ago

I'm numb to it after many "EU fines Householdnamecorp a zillion doubloons" type headlines, but using "historic fine" to describe $10k to a lawyer feels odd.

dylan604 1y ago
expecting the same level of fine to an individual person as opposed to a faceless corp really shows how numb you must be. for an attorney to be fined that much is not normal. TFA even shows example of higher fines issued to law firms, while still not as high as your zillion doubloons hyberbole it still shows the distinction between individual and s/corporation/law firm/. EU fines have been progressively getting higher especially for repeat offenders. it would be unwise to expect different in legal matters
- amoshebb 1y ago
  His website makes him look like the owner of a law firm, although I think it's just him? I'm not expecting the same number, but... california issues bigger fines for watering lawns or buying illegal fireworks. For a lawyer, a fine order of magnitude smaller than "hiring a paralegal" is less "historic" and more "cost of doing business, don't get caught"
  - dylan604 1y ago
    California issues higher fines for littering than for abandoning an animal on the highway. It's listed right there on the highway signs as you enter the state. It stood out when I saw it for the first time. Fines are arbitrary numbers set by some people not necessarily knowing about other fines for other offenses.
grues-dinner 1y ago
> The fine appears to be the largest issued over AI fabrications by a California court This is a bit like all the stats like "this is appears to be an unprecedented majority in the last 10 years in a Vermont county starting with G for elections held on the 4th when no candidate is from Oklahoma". Lots of things are historic but that doesn't necessarily mean they're impressive overall. More interesting is how many of these cases have already been tried such that this isn't "historic" for being the first one decided.

jsbisviewtiful 1y ago

Doesn't even feel controversial. LLMs hallucinate and in law that's not acceptable. Increase the fine though to really punish those doing this.

abirch 1y ago
Hopefully the fines grow exponentially for repeat offenders. Seems like now, lawyers can use AI to DDOS the other counsel.

donatj 1y ago

$10,000? That's a slap on the wrist. I don't say this lightly, this should have been jail time for someone. You're making a mockery of our most sacred institutions.

dylan604 1y ago
for first offenses for something like this you'd suggest jail time? hope you find excuses to skip your next jury summons. however, it is typical in jury selection to be asked by the defense if you'd be able to agree with a minimum sentence while the prosecutors like to ask if you'd be able to agree to the maximum. personally was asked if I could agree to 99 years for someone's first offense of GTA. I said no, and was dismissed. Sounds like you'd have said yes.
- quickthrowman 1y ago
  For someone who had to attend 6+ years of school and had to pass a professional licensing exam with ethics questions? Yes, I do. $10,000 is one week of billable hours at $250/hr. Do you think a Civil Engineer (PE) should be held liable if they vibe engineered a bridge using an LLM without reviewing the output? For this hypothetical, let’s assume an inspector caught the issue before the bridge was in use, but it would’ve collapsed had the inspector not noticed.
  - dylan604 1y ago
    No single civil engineer designs a bridge though now do they? So the premise of your retort is just way off here. No bridge plan is made without reviews after one person presses print on the plotter. Even the construction company hired to build the bridge will review the plans before they break ground. If someone is building a bridge on their private property and hires their nephew, that's on them. An actual civil project, nope, I reject your premise outright.
    - quickthrowman 1y ago
      I would wager that plenty of bridges are designed by a single engineer, most bridges are not massive 8-lane highway bridges but small bridges in municipalities. A single person can design a building, why not a bridge? P.S. I sell and run commercial construction work

jerlam 1y ago

I searched for the lawyer's name in the state bar association, they've been practicing for over 13 years. Even has electrical engineering in their background.

anigbrowl 1y ago
And?

SilverElfin 1y ago

> He thinks it is unrealistic to expect lawyers to stop using AI. It’s become an important tool just as online databases largely replaced law libraries and, until AI systems stop hallucinating fake information, he suggests lawyers who use AI to proceed with caution. I think this is a good reason for fines to not be incredibly big. People are using AI all the time. There will be a growing period until they learn of its limitations.

dylan604 1y ago
The slope of the learning curve can be adjusted though with the level of fines being a big adjustment. Only I'd suggest not the first time for any one lawyer per firm, but per incident for each firm.
jmonty900 1y ago
[dead]

sidewndr46 1y ago

$10,000, wow what a shocker! I don't know how anyone who can afford to live in California could ever expect to pay such a fine. I expect the lawyer will soon have to declare bankruptcy.

bluGill 1y ago
Most lawyers don't make great income. This could be substantial for him if he has to pay it personally.
noisy_boy 1y ago
It the first of its kind and I would wager is more of a warning shot. 10k on a case may not be much but tens of thousands multiplied by hundreds of cases won't be negligible for a lawyer or a small firm. Not to mention the prospect of loss of case and reputation and consequently future business due to such actions.

jerf 1y ago

Do lawyers and judges not by now have software that turns all these citations into hyperlinks into some relevant database? Software that would also flag a citation as not having a referent? Surely this exists and is expensive but in wide usage...? It's not a large step after that to verify that a quote actually exists in the cited document, though I can see how perhaps that was not something that was necessary up to this point. I have to think the window on this being even slightly viable is going to close quickly. When you ship something to a judge and their copy ends up festooned with "NO REFERENT" symbols it's not going to go well for you.

KittenInABox 1y ago
Part of an issue is that there's already in existence a lot of manual entry and a lot of small/regional courts with a lot of specificity for individual jurisdictions. Unification of standards is a long way away, I mean, tech hasn't even done it
freejazz 1y ago
>Do lawyers and judges not by now have software that turns all these citations into hyperlinks into some relevant database? Software that would also flag a citation as not having a referent? Surely this exists and is expensive but in wide usage...? Why would I pay for software what I could do with my own eyes in 2 minutes?
- Dylan16807 1y ago
  Two minutes per what? Two minutes per citation is a huge time waste. Two minutes per filing full of citations is unrealistically fast and also adds up.
  - freejazz 1y ago
    It's not two minutes per citation and the notion that something that is "very expensive" but otherwise doesn't have a use case unless you're using AI in the first place is a ridiculous proposition. To call what I wrote a "huge time waste" seems to really miss the point. The citations are already hyperlinks into "some relevant database". Attorneys figured that out when they started publishing judicial opinions in reporters (books). I take the citation string and it gives me the decision. Wow! > Two minutes per filing full of citations is unrealistically fast and also adds up Also adds up to what? The amount of time it takes to do your job? Vs not doing it and just letting AI rip? Oh I see, people here think that paying another service to fix the poor job the "do your job for you" AI service did makes sense? I don't think so. But then again I'm not the one wondering if lawyers keep cases stored in "relevant databases" out of what could only be sheer ignorance (I bet AI could get that one right). Never heard of Westlaw or LexisNexis?

alwa 1y ago

> Mostafavi told CalMatters he wrote the appeal and then used ChatGPT to try and improve it. He said that he didn’t know it would add case citations or make things up. He thinks it is unrealistic to expect lawyers to stop using AI. [...] “In the meantime we’re going to have some victims, we’re going to have some damages, we’re going to have some wreckages,” he said. “I hope this example will help others not fall into the hole. I’m paying the price.” Wow. Seems like he really took the lesson to heart. We're so helpless in the face of LLM technology that "having some victims, having some damages" (rather than reading what you submit to the court) is the inevitable price of progress in the legal profession? 21 of 23 citations are fake, and so is whatever reasoning they purport to support, and that's casually "adding some citations"? I sometimes use tools that do things I don't expect, but usually I'd like to think I notice when I check their work... if there were 2 citations when I started, and 23 when I finished, I'd like to think I'd notice.

bawolff 1y ago
> He thinks it is unrealistic to expect lawyers to stop using AI Sure. Its also unrealistic to expect nobody to murder anyone. That's why we invented jail.
OptionOfT 1y ago
> He thinks it is unrealistic to expect lawyers to stop using AI. I disagree. It worked until now, and using AI is clearly doing more harm than good, especially in situations where you hire an expert to help you. Remember, a lawyer is someone who actually has passed a bar exam, and with that there is an understanding that whatever they sign, they validate as correct. The fact that they used AI here actually isn't the worst. The fact that they blindly signed it afterwards is a sign that they are unfit to be a lawyer. We can make the argument that this might be pushed from upper management, but remember, the license is personal. So it's not that they can hide behind such a mandate. It's the same discussions I'm having with colleagues about using AI to generate code, or to review code. At a certain moment there is pressure to go faster, and stuff gets committed without a human touching it. Until that software ends up on your glucose pump, or the system used to radiate your brain tumor.
- 1y ago

nerdjon 1y ago

This is just incredibly defeatist on everyone talking here. Here we have irrefutable evidence of just how bad the result of using AI would be and yet... the response is just that we need to accept that there is going to be damage but keep using it?!? This isn't a tech company that "needs" to keep pushing AI because investors seem to think it is the only path of the future. There is absolutely zero reason to keep trying to shoehorn this tech in places it clearly doesn't belong. We don't need to accept anything here. Just don't use it... why is that such a hard concept.

jihadjihad 1y ago
It's not (just) defeatist, it's fatalistic. And I agree. There can be cultural underpinnings here, too--the attorney is CA-based, and though LA is distinct from SF, I wonder if there isn't a thread of the "move fast and break things" ethos showing up as well.
mullingitover 1y ago
> Here we have irrefutable evidence of just how bad the result of using AI would be It's not just the result of using AI, it's the result of failing to vet the information he was providing the court. The same thing could've happened if he hired a paralegal from Fiverr to write his pleadings and didn't check their work. It's like saying that because he typed it on a computer, it's the computers that are the problem, and we shouldn't keep using them. We're already at least a year past AI tools having the ability to perform grounding (Lexis+ from LexisNexis, as I cited on another comment in this post, for example), so this whole fiasco is already something from a bygone era.
freejazz 1y ago
I'm a lawyer and I do not use AI. I was given a product test of an AI legal solution and it was terrible.
themafia 1y ago
Whenever an "AI" article is posted here the comments are heavily astroturfed.
s1artibartfast 1y ago

elpakal 1y ago

A lot of both defeatist and overly optimistic pro AI comments in here. Having built legaltech and interfaced heavily with attorneys over the last 5 years of my career, I will say that there is a wide spectrum of experience, ethics and intelligence in the field. Blindly copying output from anything and submitting it to the court seems like a mind boggling move, it doesn’t really make a difference if it was AI or Google or Bing or Thompson Reuters. This attorney is not representative of the greater population and probably had it coming imho. There is definitely benefit to using language models correctly in law, but they are different than most users in that their professional reputation is at stake wrt the output being created and the risk of adoption is always going to be greater for them.

buildbot 1y ago

Everyone is somewhat missing the point here that the California bar is making. They don't care if you use an AI or a Llama spitting at a board of letters to assemble your case, you are responsible for what you submit to the court. This is just a bad lawyer who probably didn't check their work in many other cases, and AI just enabled that bad behavior further.

stockresearcher 1y ago
The American Bar Association, as well as every state bar association, published guidance on GenAI usage at least a year ago. The existing legal and ethical responsibilities go beyond just being responsible for hallucinations. Client information privacy, accurately tracking time and billing only for time actually spent on the case, etc. As time goes on, it becomes less and less defensible to see this stuff.
pdabbadabba 1y ago
This. Lawyers can use AI tools. But an attorney is ultimately responsible for everything that goes out the door. So they had better check the output from their AI tool carefully. In most firms (at least most good ones) someone would check every citation that goes out the door, even if it was written by an experienced attorney, so it speaks volumes that someone would fail to check citations generated by an LLM.

ElijahLynn 1y ago

Does anyone have a source of what the citations were? I didn't see them mentioned in the article.

SloopJon 1y ago
The article links to the opinion[1], which notes more than once that "the quoted language does not appear anywhere in the opinion," and "Goldstine appears to be a fabricated case." I don't know whether it's easy to get a copy of the complaint in question. [1] https://www4.courts.ca.gov/opinions/documents/B331918.PDF https://www4.courts.ca.gov/opinions/documents/B331918.PDF

rdtsc 1y ago

Wonder what the State Bar of CA would have to say about this: https://apps.calbar.ca.gov/attorney/Licensee/Detail/282372 https://apps.calbar.ca.gov/attorney/Licensee/Detail/282372 Doesn't seems like there isn't kind of disciplinary action. You can just make up stuff and if you're caught, pay some pocket change (in lawyer money level territory) and move on.

patrickhogan1 1y ago
Fines above $1k must be reported to state bar in CA. So they will know about this one.

us0r 1y ago

An OnlyFans case that is ongoing now the plaintiffs attorneys made a filling with i believe entirely fabricated case references: https://www.courtlistener.com/docket/68990373/nz-v-fenix-international-limited/?filed_after=&filed_before=&entry_gte=&entry_lte=&order_by=desc https://www.courtlistener.com/docket/68990373/nz-v-fenix-int...

yieldcrv 1y ago
I just read the initial complaint, what do you think about that case? Is there a community that wants disclosure of "chatter's" existence? It seems to be going the other way with AI personalities doing the chatting
narrator 1y ago
The hallucinations in legal briefs get really out of hand when the attorney wants to make an argument not supported by the case law. The LLM wants to do a good job defending a case, so it invents the legal precident, because otherwise it'd be impossible to make the argument credibly. This invites a rule 11 challenge from the other side where you claim the lawyer is so full of crap with his claim that he deserves sanction for not understanding the law and wasting everyone's time. What's interesting about the rules of civil procedure is that it has been built up over centuries to prevent all kinds of abuse by sneaky, clever, unscrupulous litigants. Most systems are not so hardened against bad faith actors like the legal system is and AI just thinks it can pathologically lie its way through cause most people trust somebody who sounds authoritative.

pessimizer 1y ago

AI constantly fabricates references. I found a question recently (accidentally) which would make it fabricate every answer. Otherwise, it tends to only fabricate about a third of them, and somehow miss dozens of others that it should easily find. After asking for recommendations, I always immediately ask it if any are hallucinations. It then tells me a bunch of them are, then goes "Would you like more information about how LLMs "hallucinate," and for us to come up with an architecture that could reduce or eliminate the problem?" No, fake dude, I just want real books instead of imaginary ones, not to hear about your problems. detail: The question was to find a book that examined how the work of a short order cook is done in detail, or any book with a section covering this. I started the question by mentioning that I already had Fast Foods and Short Order Cooking(1984) by Popper et al. and that was the best I had found so far. It gave me about a half dozen great hallucinations. You can try the question and see how it works for you. They're so dumb. Our economy is screwed.

mullingitover 1y ago

It's fun to point and laugh in this scenario where the attorney just threw slop at the court. However, these stories won't be around long. Tools like Lexis+ from LexisNexis[1] now offer grounding, so it won't be as simple to bust people cutting corners in the future because these prevent the hallucinations. We're now closer to the real Butlerian Jihad when we see pro se plaintiffs/defendants winning cases regularly. [1] https://www.lexisnexis.com/blogs/en-au/insights/hallucination-free-linked-legal-citations https://www.lexisnexis.com/blogs/en-au/insights/hallucinatio...

JambalayaJimbo 1y ago
How will pro se defendants afford LexisNexis??

epa 1y ago

$10,000 is nothing. Should be $200,000+.

deleted 1y ago
[deleted]

dmitrygr 1y ago

The fact that the lawyer is unrepentant and says that it is simply too convenient and "there will be some victims, tough shit", basically, means the fine was a zero or two too short. Deterrence needs to hurt!

jqpabc123 1y ago

There you have it --- proof that lawyers love AI --- in more ways than this one example illustrates. Using a tool that is widely known to be flawed to provide any sort of professional service (legal, medical, accounting, engineering, banking, etc.) is pretty much a textbook definition of negligence. And lawyers just love negligence.

fooker 1y ago

LLMs are just not the technology for 'search' and cite references, no matter how 'agentic' you make it. They are great for a large number of other things, like generating code for example. We are so focused trying to shoehorn LLMs to do something it is fundamentally unsuitable for that we are probably missing the discovery of the technology that would solve this.

narrator 1y ago

The legal system is not an API you can spam. You must have credentials that involve passing a very tough exam to even submit to it and going to school for 3 years. If you make a submission that is not fully syntax and reference checked you can face a $10,000 fine and suspension of your license to submit to it. You do not waste the legal system's time with AI slop. I have worked in litigation and if you submit some litigation to a court clerk that is not perfect you won't even get a reply that it's no good. They'll just throw your brief in the trash without looking at it and never call you back.

stewBedasso 1y ago

free bro

BrenBarn 1y ago

$10k is a slap on the wrist. He should be disbarred.

smsm42 1y ago

Just 10k? And this is presented as historically huge fine? I don't know about thst particulat lawyer, but I heard lawyers charging multiple hundreds of dollars per hour, so that's what, less than week's wages? What kind of crappy deterrent is that? He should have been banned from court for a couple of years at least, if he repeats - forever.