28 ms·

Lawyers blame ChatGPT for tricking them into citing bogus case law

150 points by glitcher 3y ago

sys42590 3y ago
When you start giving ChatGPT a plugin to query LexisNexis and do proper citiations (as Bing Chat does), then things get interesting. Unfortunately Lexis's API fees are currently quite steep, so only very wealthy law firms will be able to afford to use such a service in the short term.
- RugnirViking 3y ago
  You'll still run into the same problem these guys did (i.e. not checking the citations). It's easy enough to just search to see if a thing it tells you is real.
  - plorg 3y ago
    ...Until search is subsumed by chatbots and all you can access is a commercial statistical model's rendering of the truth, digested from ever more abstract renderings of the primary sources, which probably exist somewhere but are impractical to find under the mountain of re-written LLM spam.
  - SparkyMcUnicorn 3y ago
    I'm currently building something for a client that does this. 1. Searches for relevant documents. 2. Generate a response based on the found documents, using temperature=0 and a prompt that instructs the response to include a citation reference in a specific format. 3. Display the result, linking directly to the sources of the citations, and a warning on any that don't actually exist (which hasn't happened yet).
omginternets 3y ago
The cynic in me wonders if this isn’t part of a plan to create a legal precedent banning AI from handling legal disputes. Think about it: the legal profession is possibly one of the most threatened by the development of AI models. What better way to secure the professional future of the long tail of lawyers and paralegals?
- ImPostingOnHN 3y ago
  Occam's razor would suggest it's just an incompetent lawyer, rather than an evil genius lawyer playing 3d chess while perfectly appearing to be an incompetent lawyer
  - AnimalMuppet 3y ago
    Not in this article, and I something I heard second-hand, so take it with some salt: The lawyer wasn't admitted to Federal court, so he signed his partner's name on the filing. That's an incompetent lawyer, and a dishonest one. It's not 3D chess, it's just "sleazy lawyers gonna sleaze".
  - ArnoVW 3y ago
    Nit : you probably mean Hanlon's razor https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor
    - ImPostingOnHN 3y ago
      Thank you! How embarrassing :)
    - rideontime 3y ago
      Both would work in this case.
      - ilyt 3y ago
        Which is usually the case tbh.
- lennoff 3y ago
  How can you ban AI? It's literally designed to produce text that's indistinguishable from text that was produced by a living human being.
- pixl97 3y ago
  The particular issue here is you believe that this lawyer just isn't dumber than a bag of hammers. There is no conspiracy needed. People are dumb. This is why when you see warning labels all over some item, "Big Label" didn't do that, no some dumbass got their tallywacker ripped off by their Easy Bake Oven. Now everyone has to deal with 10 stickers on the item and a dictionary sized book of what you can and can't do with it.
voakbasda 3y ago
If they were able to be tricked by ChatGPT, they are definitely not good at being lawyers. Trying to blame the AI is like trying to blame MS Word for offering an inappropriate homonym when spell checking. The computer did not put the citations in front of the judge.
- mysterydip 3y ago
  They should ask themselves why we should pay for lawyers at all if we can ask ChatGPT the same things and get answers of the same quality.
  - singlow 3y ago
    So there are some bad lawyers therefore all lawyers are useless?
    - mysterydip 3y ago
      No, I'm saying if they think that it's to blame for wrong info and not themselves, then why should anyone pay for a lawyer instead of asking ChatGPT on their own?
  - rank0 3y ago
    These dudes are getting in trouble specifically for the shit answers spat out of chatGPT. Or yeah you could just yolo your freedom away and represent yourself using a mathematical expression with zero motivations, perception, or emotion.
    - mysterydip 3y ago
      You missed what I was saying (or I communicated it poorly). I'm not suggesting people use ChatGPT for their lawyer. They're saying it's ChatGPT's fault that they provided wrong info. If they provide no value (no liability for the service provided, and no filtering/vetting of ChatGPT answers from their lawyer expertise), then why would people pay their fee?
cowmix 3y ago
[flagged]
- butler14 3y ago
  Great anecdote, thanks for sharing. Forcing ChatGPT to go back and sense check its overarching initial interpretation is such a good way of doing things in these types of uses cases.
- COGlory 3y ago
  This comment is settinh of my ChatGPT alarm bells. Something about the sentence structure and adjective usage.
  - weego 3y ago
    Their recent comments have a few overly glowing and slightly unrealistic scenarios using ChatGPT.
  - jkea 3y ago
    Last two paragraphs starting with "however" and "ultimately" SCREAM chatGPT generated to me
    - cowmix 3y ago
      As least I didn't add "I hope this response finds you well."
  - cowmix 3y ago
    Heh, your spidey senses are correct. I actually wrote the response and then ran it though ChatGPT for clean up. I do agree, it actually reads weird over my original draft.
- pyuser583 3y ago
  Did you write this comment with ChatGPT's help? It's a helpful and well written post. I'm not trying to insult it. But I'm curious. The text seems to say "AI", and I'm wondering if I'm seeing things not there.
300bps 3y ago
Just once it would be refreshing for someone who gets caught doing something wrong to say, "Wow, ya got me. I'm sorry. Not for doing it, I'm only sorry I got caught. I really thought I'd skate right by on this one. Legitimately won't do it again. Or if I do I'll proof-read it better at least. Please let me know the punishment and I'll accept it."
- gizmo686 3y ago
  That happens all the time. It just isn't newsworthy.
- Avicebron 3y ago
  I mean isn't the punishment his livelihood? And whoever's livelihood when someone looks under the hood and they realize a lot of people (not just lawyers) are doing this all over the place? And everyone thinks they're too smart to get caught...until they aren't..
  - pixl97 3y ago
    Yep, admitting would be immediate disbarment. With the proper amount of bullshit you might just be able to escape disbarment.
    - jamesliudotcc 3y ago
      I used to be a lawyer. While I never got in trouble with the disciplinary commission, I did keep up with what they were up to. And once, I was involved in a bankruptcy case where they were a party (the disbarred lawyer filed bankruptcy, and it was a whole mess). My sense is that these are serious people who do not put up with BS. BS will only make it worse.
BaculumMeumEst 3y ago
“There’s simply no way we could have known these were bogus cases.” the lawyers are quoted as saying. They are currently using Bard to help draft a lawsuit against OpenAI, claiming the company knowingly misrepresents the capabilities of their technology.
- IIAOPSW 3y ago
  The judge is using bingChat to write the decisions anyway.
bioemerl 3y ago
OpenAI wanted to preach safety. I think we hold them liable for literally everything, everything Chat GPT does or says, until the model is open and they can argue that they have no control over it. They wanted this liability, they accepted this liability, they said they'd keep it safe and they haven't. It's on them.
midnitewarrior 3y ago
"How dare you allow me to use my laziness against myself ChatGPT!"
xbar 3y ago
Misrepresenting wholly fabricated case law to a judge out of sheer incompetence has been grounds for disbarment in at most US States for over a century. ChatGPT told me that. It might be true.
- kkielhofner 3y ago
  Bingo. With or without ChatGPT this is easily malpractice and potentially even fraud. My (cynical) guess is not only are they lazy and sloppy, they likely (fraudulently) over billed their clients in terms of representing billable hours spent drafting this lawsuit. They're almost certainly billing their clients (or in the case of contingency, keeping the cut they normally would) as though humans (with high hourly rates) are drafting this while having ChatGPT generate their supposed work product in seconds. If you remove ChatGPT from the picture and look at this as if it was actually drafted by them the fraud argument strengthens. They essentially made up case law and citations that artificially (fraudulently) improves their argument before the court. At a minimum it's grossly incompetent and when you consider my prior paragraph it strengthens the fraud angle, as they likely skimmed over the generated ChatGPT results and submitted it because it (again, artificially) strengthens their case. It seems as though ChatGPT (with whatever prompts they used) was more than happy to prioritize pleasing them vs actually being accurate. They may as well have prompted ChatGPT with "I'm a lawyer and tell me anything you need to so I can win this case and take home the money". It's a disgrace. What a mess - these lawyers should be disbarred and investigated for what is also likely fraudulent billing practices at minimum.
  - dylan604 3y ago
    >They're almost certainly billing their clients If they sent that bill in an envelope with a stamp and placed it in the mail...sounds very familiar. Maybe they should ask ChatGPT what are the possible outcomes of me using the answers you provide
    - kkielhofner 3y ago
      Hahahaha I almost included a reference to “The Firm”. But seriously it would be wire and/or mail fraud.
DtNZNkLN 3y ago
The lawyer said: “I did not comprehend that ChatGPT could fabricate cases." I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter... I'm surprised that this one case is getting so much attention because there must be so many instances of people using false information they got from ChatGPT.
- boredumb 3y ago
  I think the case is getting the attention because it's not just some one spouting off online, it's a lawyer bumping into the legal system with false information that would otherwise be a massive legal no-no and they are trying to scape-goat it onto the new shiny software.
  - Avicebron 3y ago
    But they might have seen the shiny new software touted by "those smart AI guys as being revolutionary and passing the Bar!" and they don't hang out on HN all day so to them that's like someone saying "this bridge is sturdy!" and they walk over it without realizing they should really go over the nuts and bolts of it like a civil engineer to really be sure
    - taberiand 3y ago
      I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. It's not that they need to inspect every nut and bolt of the bridge, they just need to not walk over the bridge - or at least, not immediately start driving unreasonably heavy loads across it.
      - Avicebron 3y ago
        >I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. Are we talking about lawyers or the AI researchers?, because they certainly want to portray themselves as a modern day Prometheus
theknocker 3y ago
[dead]
plorg 3y ago
A lawyer is certainly at fault if they do not fact check the material they present at trial. But the conmen who are selling ChatGPT and the like are extremely irresponsible for the way they sell LLMs as magical AI that arrives at factually correct answers by reasoning rather than the consequence of the law of large numbers applied to stochastic text generation.
- ignite 3y ago
  They literally warn you every time you log in that the results may not be accurate.
- Ukv 3y ago
  > the conmen who are selling ChatGPT and the like are extremely irresponsible for the way they sell LLMs as magical AI that arrives at factually correct answers ChatGPT has a pop-up on first use, a warning at the top of each chat, a warning below the chat bar, and a section in the FAQ explaining that it can generate nonsense and can't verify facts, provide references, or complete lookups. There is probably more OpenAI could do, like detect attempts to generate false references and add a warning in red to that chat message - since it seems there are still people taking its hallucinations as fact (although if there's hundreds of millions of users, maybe only a tiny fraction), but I don't think this is a fair characterization.
zzzeek 3y ago
comments here are like "what dumb lawyers". Sure OK. But what can we say here about "GPT-4 passed the bar exam!" and how useful is that data, given that this does not imply GPT-4 has the actual skills of a human lawyer.
- ekam 3y ago
  The lawyer here was probably not smart enough to distinguish between 3.5 and 4. Haven't seen anything to indicate this was the output of GPT-4
  - zzzeek 3y ago
    oh ok! so if he used gpt-4, still might have been illegal but the output would have been perfect. good to know
    - ekam 3y ago
      Your reading comprehension seems to be as good as that of the lawyer here. You asked what this says about the claim that GPT-4 was good enough to pass the bar. I didn't say anything about GPT-4's quality or the legality here, only that we cannot assume this was the output of GPT-4 without evidence it was given that more people overwhelmingly use the default 3.5-turbo.
      - deleted 3y ago
        [deleted]
  - BeetleB 3y ago
    I've had Bing Chat (GPT4) hallucinate research studies. It even generated links to them in citation indexes. The authors existed. They do research in the area, so the title was very plausible. But the paper didn't exist.
seydor 3y ago
Why don't they sue it?
John23832 3y ago
People confuse creating a response that makes sense in a language (which LLM’s are designed to do) with coveying facts and truths in a language (which LLM’s are not designed to do). LLM’s are revolutionary because they provide a more fluent interface to data… but that doesn’t not mean that the data is correct. Especially not in the early phases. For most people any sufficiently advance technology is indistinguishable from magic. The average joes think that this is magic.
jcranmer 3y ago
While the lawyers blamed ChatGPT, the totality of the circumstances seem to indicate that they're less than honest in doing so. There is a live-tweet of the hearing here: https://twitter.com/innercitypress/status/1666838526762139650 https://twitter.com/innercitypress/status/166683852676213965..., and you can follow along with the lawyerly cringe there. Okay, lawyer #1 (LoDuca, the one on the case in the first place) appears to have played essentially no role in the entire case; his entire purpose appears to be effectively a sockpuppet for lawyer #2 (Schwartz), as LoDuca was admitted to federal court and Schwartz was not. He admits to not having read the things Schwartz asked him to file, as well as the "WTF?" missives that came back. He lied to the court about when he was going to be on vacation, because that is when Schwartz was on vacation. But other than doing nothing when he was supposed to do something (supposed to do a lot of somethings), he is otherwise uninvolved in the shenanigans. So everything happened because of Schwartz, but before we get to this part, let me fill in relevant background information. The client is suing an airline for an injury governed by the Montreal Protocol. Said airline went bankrupt, and when that happened, the lawyer dismissed the lawsuit, only to refile it when the airline emerged from bankruptcy. This was a mistake; dismissing-and-refiling means the second case is outside the statute of limitations. The airline filed a motion to dismiss because, well, outside statute of limitations, and it is Schwartz's response that is at the center of this controversy. What appears to have happened is that there is no case law to justify why the case shouldn't be dismissed. Schwartz used chatGPT to try to come up with case law [1] for the argument. He claims in the hearing that he treated it like a search engine, and didn't understand that it could come up with fake arguments. But those claims I'm skeptical of, because even if he's using chatGPT to search for cases, he clearly isn't reading them. When the airline basically said "uh, we can't find these cases," Schwartz responded by providing fake cases from chatGPT where alarm bells should be ringing saying "SOMETHING IS HORRIBLY, HORRIBLY WRONG." The purported cases in the reply had blindingly obvious flaws that ought to have made you realize something was up before you're off the first page. It is only when the judge turns around and issues the order to show cause that the lawyers attempt to start coming clean. But wait, there's more! The response was improperly notarized: it had the wrong month. So the judge asked them to provide the original document before signature to justify why it wasn't notary fraud. And, uh, there's a clear OCR error (compare last page of [2] and [3]). When we get to these parts of in the hearing, Schwartz's responses aren't encouraging. Schwartz tries to dodge the issue of why he was citing cases he didn't read. Believing his responses of why he thought the cases were unpublished ("F.3d means Federal district, third department") really requires you to assume he is an incompetent lawyer at best. The inconsistencies in the affidavit are glossed over, and the stories don't entirely add up. It seems like a minor issue, but it does really give the impression that even with all the attention on them right now, they're still being less than candid with the court. The attorneys for Schwartz are trying hard to frame it as a "he didn't know what he was getting into with chatGPT, it's not his fault," but honestly, it really does strike me that he knew what he was getting into and somehow thought he wouldn't get caught. [1] His conversation can be found here: https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.46.1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57..., it's one of the affidavits in the case. [2] https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.29.0_1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57... [3] https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.44.1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
IIAOPSW 3y ago
Thats weird. I tried really hard to convince it that there's such case law establishing a legal category of "praiseworthy homicide" and it refused to believe me. I thought it was overttrained / patched on all law related applications.
anon25783 3y ago
I'm so confused. Why do the lawyers not simply check to see if the references are real or not? How hard is it to look through an LLM's output and do a quick search to see if any of the laws or cases mentioned in it are in fact unsubstantiated?
- phren0logy 3y ago
  Regardless of how hard it is to check the references (which is not really all that hard), it's the job of a lawyer.
- Ekaros 3y ago
  Sure use whatever tool to find suitable cases supporting you. But also verify that those cases support your position. Better not give ammo to your opponent.
- chrononaut 3y ago
  At best an LLM in this case should serve as a "pointer" or "hint" to references. It's clear one would still need validate the entirety of the case and the arguments put forward from them.
- mike_hearn 3y ago
  Because people aren't used to the idea that computers will make things up. I don't see why anyone is surprised by what the lawyer did here and for sure there is a lot more of this going on than is getting into the public. People spend their whole lives thinking of machines as logical and correct, but lacking in creativity. Suddenly generative AI comes along and totally upends this. If you're an HN reader and see in depth discussions of LLMs every day then of course it's obvious that these things can make up citations, but if you primary experience of the tech industry is search engines that have never made up fake results ever, then why would you even imagine it's possible?
bastardoperator 3y ago
They saw ChatGPT passed the bar in the 90th percentile and thought they were on easy street. What a dumb way to lose your law license.
- seydor 3y ago
  but they can use GPT4 to regain it
bequanna 3y ago
Would it have been that difficult for the lawyers to actually check the case law ChatGPT cited? Seems like pure laziness.
hackerfactor1 3y ago
A poor workman blames his tools.
causi 3y ago
Blaming ChatGPT for making stuff up is like blaming sex dice after you followed their instructions to "spank" + "hair".
6gvONxR4sf7o 3y ago
I don't understand how they messed this up so bad. They say they didn't know it could hallucinate and that they thought it was just like any other search engine. But it seems like even if it worked like they thought, they'd still have fucked up? If it's just like a normal person, if that person isn't a lawyer, you wouldn't ask them to do your lawyery work. I'd hope this lawyer doesn't his kids to do his work for him. If it's just like a normal search engine, we all know how much bullshit, spam, and misinformation there is on the internet (mostly written by normal good old fashioned humans!). So that wouldn't have been trustworthy either! There's no way this kind of thing is excusable.
elforce002 3y ago
This is interesting. How long untill someone gets sick because he/she was following what chatGPT told him/her to do? Medical advice? Political misinformation? How things will unfold this decade? Banning chatgpt from certain topics (medicine, law, etc...)? This decade will be really interesting indeed.
- ilyt 3y ago
  People did that with googling symptoms for a long time now
rank0 3y ago
Disbarment is appropriate here. Why not blame your laptop manufacturer for creating the hardware you used to file your fraudulent court documents?
- deleted 3y ago
  [deleted]
- CPLX 3y ago
  No it’s not. You’ll get old one day. It’s pretty challenging to keep track of what’s real and what isn’t possible with technology. The guy apologized and said he thought it was a search engine. He should definitely face some sanctions but someone had to learn this lesson in public the hard way for word to spread.
  - code_runner 3y ago
    These exemptions we are passing out are insane. This guy claims he thought it was a search engine? Lawyers use specific case law databases and they SHOULD approach any new tools with healthy skepticism. And if he googled it and cited a fake case would it be better? Why wouldn’t you vet the information.
    - CPLX 3y ago
      Yeah it was bad. Disbarment is equivalent to taking away his entire life's work. Two minute hate on the internet every day is fun and all but it's not that severe.
      - Ekaros 3y ago
        He failed his duty to both client and the court. This was wilful negligence. This is like surgeon doing wrong procedure he just invented on patient.
      - vkou 3y ago
        > Disbarment is equivalent to taking away his entire life's work. 'Lawyer' is the kind of profession which actually holds its practitioners to a standard, because the system falls apart when they stop behaving honestly. This wasn't an oopsie-daisies, this was dishonesty, followed by further dishonesty, when they supplied bogus references for these cases.
deleted 3y ago
[deleted]
summerlight 3y ago
Oh god, I suppose a part of the reason why those lawyers are well paid is because if something goes wrong then they're going to be responsible...
ineedasername 3y ago
>[to the judge on behalf of Schwartz] Mr. Schwartz, someone who barely does federal research, chose to use this new technology. That’s a horrible excuse. I’m not a lawyer and don’t do caselaw research on any sort of regular basis but I have still poked around a bit when something strikes my interest. Compared to google they’re clunky and have poor matching, but I don’t remember it taking me more than half an hour or so to figure out which system would have the case I want and have to drill down to find it. ChatGPT was giving the the lawyer the (made up) case. It should really be a trivial task to find it in a caselaw database. Heck if I was the lawyer I would really really want to find the full text case! Who knows what broader context or additional nuggets of useful information it might have for my current client’s issue? I would not be surprised if he went looking, couldn’t find it easily, and just said “whatever it has to be there somewhere and I can get by without the entire thing”
meghan_rain 3y ago
in this thread (and all threads on this topic): angry armchair legalists trying to stick it to The Man!! by pretending neglegient homicide is the same as premeditated murder
jameslk 3y ago
Given that they're attorneys, we know what their next course of action may be. I've noticed that ChatGPT's new onboarding includes a screen with a big disclaimer where there wasn't one before. I can only assume that it may be related to cases like these.
isp 3y ago
Related: - "A man sued Avianca Airline – his lawyer used ChatGPT" (13 days ago, 174 points, 139 comments): https://news.ycombinator.com/item?id=36095352 https://news.ycombinator.com/item?id=36095352 - "Lawyer who used ChatGPT faces penalty for made up citations" (1 day ago, 106 points, 128 comments): https://news.ycombinator.com/item?id=36242462 https://news.ycombinator.com/item?id=36242462 Original court documents: https://www.courtlistener.com/docket/63107798/mata-v-avianca-inc/ https://www.courtlistener.com/docket/63107798/mata-v-avianca...
activiation 3y ago
Surely CGPT can tell them how to get out of this one.
pengaru 3y ago
It makes sense in a Dunning-Kruger way that lawyers this dumb would consider ChatGPT qualified for their purposes.
RagnarD 3y ago
The lawyers are either idiots, totally dishonest, or both. OpenAI should make ChatGPT off-limits to lawyers in their TOS.
ww520 3y ago
Boy, that's elementary level excuse. Your honor, my dog ate my homework.
cainxinth 3y ago
“I’m an idiot, my bad.” is rarely a useful defense when you have a fiduciary duty to your client and work in tightly regulated field.

150 points by glitcher 3y ago

When you start giving ChatGPT a plugin to query LexisNexis and do proper citiations (as Bing Chat does), then things get interesting. Unfortunately Lexis's API fees are currently quite steep, so only very wealthy law firms will be able to afford to use such a service in the short term.

RugnirViking 3y ago
You'll still run into the same problem these guys did (i.e. not checking the citations). It's easy enough to just search to see if a thing it tells you is real.
- plorg 3y ago
  ...Until search is subsumed by chatbots and all you can access is a commercial statistical model's rendering of the truth, digested from ever more abstract renderings of the primary sources, which probably exist somewhere but are impractical to find under the mountain of re-written LLM spam.
- SparkyMcUnicorn 3y ago
  I'm currently building something for a client that does this. 1. Searches for relevant documents. 2. Generate a response based on the found documents, using temperature=0 and a prompt that instructs the response to include a citation reference in a specific format. 3. Display the result, linking directly to the sources of the citations, and a warning on any that don't actually exist (which hasn't happened yet).

omginternets 3y ago

The cynic in me wonders if this isn’t part of a plan to create a legal precedent banning AI from handling legal disputes. Think about it: the legal profession is possibly one of the most threatened by the development of AI models. What better way to secure the professional future of the long tail of lawyers and paralegals?

ImPostingOnHN 3y ago
Occam's razor would suggest it's just an incompetent lawyer, rather than an evil genius lawyer playing 3d chess while perfectly appearing to be an incompetent lawyer
- AnimalMuppet 3y ago
  Not in this article, and I something I heard second-hand, so take it with some salt: The lawyer wasn't admitted to Federal court, so he signed his partner's name on the filing. That's an incompetent lawyer, and a dishonest one. It's not 3D chess, it's just "sleazy lawyers gonna sleaze".
- ArnoVW 3y ago
  Nit : you probably mean Hanlon's razor https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor
  - ImPostingOnHN 3y ago
    Thank you! How embarrassing :)
  - rideontime 3y ago
    Both would work in this case.
    - ilyt 3y ago
      Which is usually the case tbh.
lennoff 3y ago
How can you ban AI? It's literally designed to produce text that's indistinguishable from text that was produced by a living human being.
pixl97 3y ago
The particular issue here is you believe that this lawyer just isn't dumber than a bag of hammers. There is no conspiracy needed. People are dumb. This is why when you see warning labels all over some item, "Big Label" didn't do that, no some dumbass got their tallywacker ripped off by their Easy Bake Oven. Now everyone has to deal with 10 stickers on the item and a dictionary sized book of what you can and can't do with it.

voakbasda 3y ago

If they were able to be tricked by ChatGPT, they are definitely not good at being lawyers. Trying to blame the AI is like trying to blame MS Word for offering an inappropriate homonym when spell checking. The computer did not put the citations in front of the judge.

mysterydip 3y ago
They should ask themselves why we should pay for lawyers at all if we can ask ChatGPT the same things and get answers of the same quality.
- singlow 3y ago
  So there are some bad lawyers therefore all lawyers are useless?
  - mysterydip 3y ago
    No, I'm saying if they think that it's to blame for wrong info and not themselves, then why should anyone pay for a lawyer instead of asking ChatGPT on their own?
- rank0 3y ago
  These dudes are getting in trouble specifically for the shit answers spat out of chatGPT. Or yeah you could just yolo your freedom away and represent yourself using a mathematical expression with zero motivations, perception, or emotion.
  - mysterydip 3y ago
    You missed what I was saying (or I communicated it poorly). I'm not suggesting people use ChatGPT for their lawyer. They're saying it's ChatGPT's fault that they provided wrong info. If they provide no value (no liability for the service provided, and no filtering/vetting of ChatGPT answers from their lawyer expertise), then why would people pay their fee?

cowmix 3y ago

[flagged]

butler14 3y ago
Great anecdote, thanks for sharing. Forcing ChatGPT to go back and sense check its overarching initial interpretation is such a good way of doing things in these types of uses cases.
COGlory 3y ago
This comment is settinh of my ChatGPT alarm bells. Something about the sentence structure and adjective usage.
- weego 3y ago
  Their recent comments have a few overly glowing and slightly unrealistic scenarios using ChatGPT.
- jkea 3y ago
  Last two paragraphs starting with "however" and "ultimately" SCREAM chatGPT generated to me
  - cowmix 3y ago
    As least I didn't add "I hope this response finds you well."
- cowmix 3y ago
  Heh, your spidey senses are correct. I actually wrote the response and then ran it though ChatGPT for clean up. I do agree, it actually reads weird over my original draft.
pyuser583 3y ago
Did you write this comment with ChatGPT's help? It's a helpful and well written post. I'm not trying to insult it. But I'm curious. The text seems to say "AI", and I'm wondering if I'm seeing things not there.

300bps 3y ago

Just once it would be refreshing for someone who gets caught doing something wrong to say, "Wow, ya got me. I'm sorry. Not for doing it, I'm only sorry I got caught. I really thought I'd skate right by on this one. Legitimately won't do it again. Or if I do I'll proof-read it better at least. Please let me know the punishment and I'll accept it."

gizmo686 3y ago
That happens all the time. It just isn't newsworthy.
Avicebron 3y ago
I mean isn't the punishment his livelihood? And whoever's livelihood when someone looks under the hood and they realize a lot of people (not just lawyers) are doing this all over the place? And everyone thinks they're too smart to get caught...until they aren't..
- pixl97 3y ago
  Yep, admitting would be immediate disbarment. With the proper amount of bullshit you might just be able to escape disbarment.
  - jamesliudotcc 3y ago
    I used to be a lawyer. While I never got in trouble with the disciplinary commission, I did keep up with what they were up to. And once, I was involved in a bankruptcy case where they were a party (the disbarred lawyer filed bankruptcy, and it was a whole mess). My sense is that these are serious people who do not put up with BS. BS will only make it worse.

BaculumMeumEst 3y ago

“There’s simply no way we could have known these were bogus cases.” the lawyers are quoted as saying. They are currently using Bard to help draft a lawsuit against OpenAI, claiming the company knowingly misrepresents the capabilities of their technology.

IIAOPSW 3y ago
The judge is using bingChat to write the decisions anyway.

bioemerl 3y ago

OpenAI wanted to preach safety. I think we hold them liable for literally everything, everything Chat GPT does or says, until the model is open and they can argue that they have no control over it. They wanted this liability, they accepted this liability, they said they'd keep it safe and they haven't. It's on them.

midnitewarrior 3y ago

"How dare you allow me to use my laziness against myself ChatGPT!"

xbar 3y ago

Misrepresenting wholly fabricated case law to a judge out of sheer incompetence has been grounds for disbarment in at most US States for over a century. ChatGPT told me that. It might be true.

kkielhofner 3y ago
Bingo. With or without ChatGPT this is easily malpractice and potentially even fraud. My (cynical) guess is not only are they lazy and sloppy, they likely (fraudulently) over billed their clients in terms of representing billable hours spent drafting this lawsuit. They're almost certainly billing their clients (or in the case of contingency, keeping the cut they normally would) as though humans (with high hourly rates) are drafting this while having ChatGPT generate their supposed work product in seconds. If you remove ChatGPT from the picture and look at this as if it was actually drafted by them the fraud argument strengthens. They essentially made up case law and citations that artificially (fraudulently) improves their argument before the court. At a minimum it's grossly incompetent and when you consider my prior paragraph it strengthens the fraud angle, as they likely skimmed over the generated ChatGPT results and submitted it because it (again, artificially) strengthens their case. It seems as though ChatGPT (with whatever prompts they used) was more than happy to prioritize pleasing them vs actually being accurate. They may as well have prompted ChatGPT with "I'm a lawyer and tell me anything you need to so I can win this case and take home the money". It's a disgrace. What a mess - these lawyers should be disbarred and investigated for what is also likely fraudulent billing practices at minimum.
- dylan604 3y ago
  >They're almost certainly billing their clients If they sent that bill in an envelope with a stamp and placed it in the mail...sounds very familiar. Maybe they should ask ChatGPT what are the possible outcomes of me using the answers you provide
  - kkielhofner 3y ago
    Hahahaha I almost included a reference to “The Firm”. But seriously it would be wire and/or mail fraud.

DtNZNkLN 3y ago

The lawyer said: “I did not comprehend that ChatGPT could fabricate cases." I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter... I'm surprised that this one case is getting so much attention because there must be so many instances of people using false information they got from ChatGPT.

boredumb 3y ago
I think the case is getting the attention because it's not just some one spouting off online, it's a lawyer bumping into the legal system with false information that would otherwise be a massive legal no-no and they are trying to scape-goat it onto the new shiny software.
- Avicebron 3y ago
  But they might have seen the shiny new software touted by "those smart AI guys as being revolutionary and passing the Bar!" and they don't hang out on HN all day so to them that's like someone saying "this bridge is sturdy!" and they walk over it without realizing they should really go over the nuts and bolts of it like a civil engineer to really be sure
  - taberiand 3y ago
    I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. It's not that they need to inspect every nut and bolt of the bridge, they just need to not walk over the bridge - or at least, not immediately start driving unreasonably heavy loads across it.
    - Avicebron 3y ago
      >I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. Are we talking about lawyers or the AI researchers?, because they certainly want to portray themselves as a modern day Prometheus

theknocker 3y ago

[dead]

plorg 3y ago

A lawyer is certainly at fault if they do not fact check the material they present at trial. But the conmen who are selling ChatGPT and the like are extremely irresponsible for the way they sell LLMs as magical AI that arrives at factually correct answers by reasoning rather than the consequence of the law of large numbers applied to stochastic text generation.

ignite 3y ago
They literally warn you every time you log in that the results may not be accurate.
Ukv 3y ago
> the conmen who are selling ChatGPT and the like are extremely irresponsible for the way they sell LLMs as magical AI that arrives at factually correct answers ChatGPT has a pop-up on first use, a warning at the top of each chat, a warning below the chat bar, and a section in the FAQ explaining that it can generate nonsense and can't verify facts, provide references, or complete lookups. There is probably more OpenAI could do, like detect attempts to generate false references and add a warning in red to that chat message - since it seems there are still people taking its hallucinations as fact (although if there's hundreds of millions of users, maybe only a tiny fraction), but I don't think this is a fair characterization.

zzzeek 3y ago

comments here are like "what dumb lawyers". Sure OK. But what can we say here about "GPT-4 passed the bar exam!" and how useful is that data, given that this does not imply GPT-4 has the actual skills of a human lawyer.

ekam 3y ago
The lawyer here was probably not smart enough to distinguish between 3.5 and 4. Haven't seen anything to indicate this was the output of GPT-4
- zzzeek 3y ago
  oh ok! so if he used gpt-4, still might have been illegal but the output would have been perfect. good to know
  - ekam 3y ago
    Your reading comprehension seems to be as good as that of the lawyer here. You asked what this says about the claim that GPT-4 was good enough to pass the bar. I didn't say anything about GPT-4's quality or the legality here, only that we cannot assume this was the output of GPT-4 without evidence it was given that more people overwhelmingly use the default 3.5-turbo.
    - deleted 3y ago
      [deleted]
- BeetleB 3y ago
  I've had Bing Chat (GPT4) hallucinate research studies. It even generated links to them in citation indexes. The authors existed. They do research in the area, so the title was very plausible. But the paper didn't exist.

seydor 3y ago

Why don't they sue it?

John23832 3y ago

People confuse creating a response that makes sense in a language (which LLM’s are designed to do) with coveying facts and truths in a language (which LLM’s are not designed to do). LLM’s are revolutionary because they provide a more fluent interface to data… but that doesn’t not mean that the data is correct. Especially not in the early phases. For most people any sufficiently advance technology is indistinguishable from magic. The average joes think that this is magic.

jcranmer 3y ago

While the lawyers blamed ChatGPT, the totality of the circumstances seem to indicate that they're less than honest in doing so. There is a live-tweet of the hearing here: https://twitter.com/innercitypress/status/1666838526762139650 https://twitter.com/innercitypress/status/166683852676213965..., and you can follow along with the lawyerly cringe there. Okay, lawyer #1 (LoDuca, the one on the case in the first place) appears to have played essentially no role in the entire case; his entire purpose appears to be effectively a sockpuppet for lawyer #2 (Schwartz), as LoDuca was admitted to federal court and Schwartz was not. He admits to not having read the things Schwartz asked him to file, as well as the "WTF?" missives that came back. He lied to the court about when he was going to be on vacation, because that is when Schwartz was on vacation. But other than doing nothing when he was supposed to do something (supposed to do a lot of somethings), he is otherwise uninvolved in the shenanigans. So everything happened because of Schwartz, but before we get to this part, let me fill in relevant background information. The client is suing an airline for an injury governed by the Montreal Protocol. Said airline went bankrupt, and when that happened, the lawyer dismissed the lawsuit, only to refile it when the airline emerged from bankruptcy. This was a mistake; dismissing-and-refiling means the second case is outside the statute of limitations. The airline filed a motion to dismiss because, well, outside statute of limitations, and it is Schwartz's response that is at the center of this controversy. What appears to have happened is that there is no case law to justify why the case shouldn't be dismissed. Schwartz used chatGPT to try to come up with case law [1] for the argument. He claims in the hearing that he treated it like a search engine, and didn't understand that it could come up with fake arguments. But those claims I'm skeptical of, because even if he's using chatGPT to search for cases, he clearly isn't reading them. When the airline basically said "uh, we can't find these cases," Schwartz responded by providing fake cases from chatGPT where alarm bells should be ringing saying "SOMETHING IS HORRIBLY, HORRIBLY WRONG." The purported cases in the reply had blindingly obvious flaws that ought to have made you realize something was up before you're off the first page. It is only when the judge turns around and issues the order to show cause that the lawyers attempt to start coming clean. But wait, there's more! The response was improperly notarized: it had the wrong month. So the judge asked them to provide the original document before signature to justify why it wasn't notary fraud. And, uh, there's a clear OCR error (compare last page of [2] and [3]). When we get to these parts of in the hearing, Schwartz's responses aren't encouraging. Schwartz tries to dodge the issue of why he was citing cases he didn't read. Believing his responses of why he thought the cases were unpublished ("F.3d means Federal district, third department") really requires you to assume he is an incompetent lawyer at best. The inconsistencies in the affidavit are glossed over, and the stories don't entirely add up. It seems like a minor issue, but it does really give the impression that even with all the attention on them right now, they're still being less than candid with the court. The attorneys for Schwartz are trying hard to frame it as a "he didn't know what he was getting into with chatGPT, it's not his fault," but honestly, it really does strike me that he knew what he was getting into and somehow thought he wouldn't get caught. [1] His conversation can be found here: https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.46.1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57..., it's one of the affidavits in the case. [2] https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.29.0_1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57... [3] https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.44.1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...

IIAOPSW 3y ago

Thats weird. I tried really hard to convince it that there's such case law establishing a legal category of "praiseworthy homicide" and it refused to believe me. I thought it was overttrained / patched on all law related applications.

anon25783 3y ago

I'm so confused. Why do the lawyers not simply check to see if the references are real or not? How hard is it to look through an LLM's output and do a quick search to see if any of the laws or cases mentioned in it are in fact unsubstantiated?

phren0logy 3y ago
Regardless of how hard it is to check the references (which is not really all that hard), it's the job of a lawyer.
Ekaros 3y ago
Sure use whatever tool to find suitable cases supporting you. But also verify that those cases support your position. Better not give ammo to your opponent.
chrononaut 3y ago
At best an LLM in this case should serve as a "pointer" or "hint" to references. It's clear one would still need validate the entirety of the case and the arguments put forward from them.
mike_hearn 3y ago
Because people aren't used to the idea that computers will make things up. I don't see why anyone is surprised by what the lawyer did here and for sure there is a lot more of this going on than is getting into the public. People spend their whole lives thinking of machines as logical and correct, but lacking in creativity. Suddenly generative AI comes along and totally upends this. If you're an HN reader and see in depth discussions of LLMs every day then of course it's obvious that these things can make up citations, but if you primary experience of the tech industry is search engines that have never made up fake results ever, then why would you even imagine it's possible?

bastardoperator 3y ago

They saw ChatGPT passed the bar in the 90th percentile and thought they were on easy street. What a dumb way to lose your law license.

seydor 3y ago
but they can use GPT4 to regain it

bequanna 3y ago

Would it have been that difficult for the lawyers to actually check the case law ChatGPT cited? Seems like pure laziness.

hackerfactor1 3y ago

A poor workman blames his tools.

causi 3y ago

Blaming ChatGPT for making stuff up is like blaming sex dice after you followed their instructions to "spank" + "hair".

6gvONxR4sf7o 3y ago

I don't understand how they messed this up so bad. They say they didn't know it could hallucinate and that they thought it was just like any other search engine. But it seems like even if it worked like they thought, they'd still have fucked up? If it's just like a normal person, if that person isn't a lawyer, you wouldn't ask them to do your lawyery work. I'd hope this lawyer doesn't his kids to do his work for him. If it's just like a normal search engine, we all know how much bullshit, spam, and misinformation there is on the internet (mostly written by normal good old fashioned humans!). So that wouldn't have been trustworthy either! There's no way this kind of thing is excusable.

elforce002 3y ago

This is interesting. How long untill someone gets sick because he/she was following what chatGPT told him/her to do? Medical advice? Political misinformation? How things will unfold this decade? Banning chatgpt from certain topics (medicine, law, etc...)? This decade will be really interesting indeed.

ilyt 3y ago
People did that with googling symptoms for a long time now

rank0 3y ago

Disbarment is appropriate here. Why not blame your laptop manufacturer for creating the hardware you used to file your fraudulent court documents?

deleted 3y ago
[deleted]
CPLX 3y ago
No it’s not. You’ll get old one day. It’s pretty challenging to keep track of what’s real and what isn’t possible with technology. The guy apologized and said he thought it was a search engine. He should definitely face some sanctions but someone had to learn this lesson in public the hard way for word to spread.
- code_runner 3y ago
  These exemptions we are passing out are insane. This guy claims he thought it was a search engine? Lawyers use specific case law databases and they SHOULD approach any new tools with healthy skepticism. And if he googled it and cited a fake case would it be better? Why wouldn’t you vet the information.
  - CPLX 3y ago
    Yeah it was bad. Disbarment is equivalent to taking away his entire life's work. Two minute hate on the internet every day is fun and all but it's not that severe.
    - Ekaros 3y ago
      He failed his duty to both client and the court. This was wilful negligence. This is like surgeon doing wrong procedure he just invented on patient.
    - vkou 3y ago
      > Disbarment is equivalent to taking away his entire life's work. 'Lawyer' is the kind of profession which actually holds its practitioners to a standard, because the system falls apart when they stop behaving honestly. This wasn't an oopsie-daisies, this was dishonesty, followed by further dishonesty, when they supplied bogus references for these cases.

deleted 3y ago

[deleted]

summerlight 3y ago

Oh god, I suppose a part of the reason why those lawyers are well paid is because if something goes wrong then they're going to be responsible...

ineedasername 3y ago

>[to the judge on behalf of Schwartz] Mr. Schwartz, someone who barely does federal research, chose to use this new technology. That’s a horrible excuse. I’m not a lawyer and don’t do caselaw research on any sort of regular basis but I have still poked around a bit when something strikes my interest. Compared to google they’re clunky and have poor matching, but I don’t remember it taking me more than half an hour or so to figure out which system would have the case I want and have to drill down to find it. ChatGPT was giving the the lawyer the (made up) case. It should really be a trivial task to find it in a caselaw database. Heck if I was the lawyer I would really really want to find the full text case! Who knows what broader context or additional nuggets of useful information it might have for my current client’s issue? I would not be surprised if he went looking, couldn’t find it easily, and just said “whatever it has to be there somewhere and I can get by without the entire thing”

meghan_rain 3y ago

in this thread (and all threads on this topic): angry armchair legalists trying to stick it to The Man!! by pretending neglegient homicide is the same as premeditated murder

jameslk 3y ago

Given that they're attorneys, we know what their next course of action may be. I've noticed that ChatGPT's new onboarding includes a screen with a big disclaimer where there wasn't one before. I can only assume that it may be related to cases like these.

isp 3y ago

Related: - "A man sued Avianca Airline – his lawyer used ChatGPT" (13 days ago, 174 points, 139 comments): https://news.ycombinator.com/item?id=36095352 https://news.ycombinator.com/item?id=36095352 - "Lawyer who used ChatGPT faces penalty for made up citations" (1 day ago, 106 points, 128 comments): https://news.ycombinator.com/item?id=36242462 https://news.ycombinator.com/item?id=36242462 Original court documents: https://www.courtlistener.com/docket/63107798/mata-v-avianca-inc/ https://www.courtlistener.com/docket/63107798/mata-v-avianca...

activiation 3y ago

Surely CGPT can tell them how to get out of this one.

pengaru 3y ago

It makes sense in a Dunning-Kruger way that lawyers this dumb would consider ChatGPT qualified for their purposes.

RagnarD 3y ago

The lawyers are either idiots, totally dishonest, or both. OpenAI should make ChatGPT off-limits to lawyers in their TOS.

ww520 3y ago

Boy, that's elementary level excuse. Your honor, my dog ate my homework.

cainxinth 3y ago

“I’m an idiot, my bad.” is rarely a useful defense when you have a fiduciary duty to your client and work in tightly regulated field.