12 ms·

A man sued Avianca Airline – his lawyer used ChatGPT

174 points by nlh 3y ago

fennecfoxy 3y ago
GPT4 hooked up to a legal DB and instructed that if no knowledge from the DB can be referenced to respond negatively would've solved this completely. Even a hard link to rewrite the prompt from "use provided knowledge to answer the question" to "no knowledge could be found, inform the user of this fact in a way that keeps the conversation fluid. Recommend that they search other sources themselves." would be more reliable. The problem, as always, isn't the tool. It's the tool using the tool.
- krimbus 3y ago
  Could we have certainty that an LLM would comply though? I can only imagine this happening with external constraints such as parsing the output to perform a DB check, so I'm curious.
  - fennecfoxy 3y ago
    Hmmm, I suppose we could, if there was a focus on it. However it's always interesting to watch how people react: * Traditional programming creates very strict functionality but is not flexible/is rigid. * Okay, so let's train models to act like biological life, which is super flexible but has a chance to make mistakes. But then people complain about the latter making mistakes. Realistically we can combine the two to ensure that what a model is saying is based on fact, though. Easiest way would be just for people to interact with an LLM to browse hard facts and then double check via sources. But relying on humans not to be lazy is just as good as relying on an ML model to act like traditional programming, so it's probably better to train our wild model with as many built-in constraints as possible and then decorate those with more traditionally programmed hard limits. It's really hard though because of the output data. We're going from a binary output of 1/0 for traditional programming to a continuous output of 0-1 for models; if I ask it to summarise something, whether it's correct or not is not only based on whether it summarised based only on the original text, but also down to the reviewer's individual biases and wants/needs as to whether the summarisation is sufficient. The continuous nature of an LLM's response does make it difficult to determine if it's sufficiently factually correct, though. Because if it's not repeating word for word, you need to be able to parse the output in order to check it...using another LLM (so compounding error...)
jaclaz 3y ago
Another thread where the poster linked to the relevant Court .pdf's: https://news.ycombinator.com/item?id=36092914 https://news.ycombinator.com/item?id=36092914 Particularly relevant is the affidavit in which the lawyer tries to explain to the Court what happened: https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.32.1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...
- lionkor 3y ago
  Hahaha that second link is brilliant. "Hm, maybe I should double check ChatGPTs output... Hey, ChatGPT, does your output make sense?" - "Yeah, my output definitely makes sense". "Are you sure?" - "Yeah". Well, then.
  - namaria 3y ago
    >Mr. Schwartz said that he had never used ChatGPT, and “therefore was unaware of the possibility that its content could be false.” Why did he think it was true, if he had never used it before then? "I found it online" levels of competence here.
    - vidarh 3y ago
      People are scarily willing to take eloquent text on trust if it says something that benefits them and sounds plausible. The biggest problem with ChatGPT is that it sounds like a smart adult that people want to give the benefit of the doubt because it agrees with them a lot instead of like a child that people would be more wary of trusting blindly.
      - namaria 3y ago
        This willingness most people show to jump on the bandwagon when hearing confident sounding language actually explains a lot of societal problems. Also a lot of workplace dynamics, and the career of a lot of very powerful and rich people.
Szpadel 3y ago
archive.org link https://web.archive.org/web/20230527153017/https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html/ https://web.archive.org/web/20230527153017/https://www.nytim...
- metadat 3y ago
  https://archive.today/pnEhz https://archive.today/pnEhz
m348e912 3y ago
This problem will be solved with LawGPT which will be a generative pre-trained transformer trained on a vast array of legal cases, rulings, and briefings and designed with accuracy in mind and free from hallucinations. It will cost $3,400/month for a legal firm the size of Levidow, Levidow & Oberman, P.C.
- ChuckNorris89 3y ago
  Still waiting for SaulGoodmanGPT
  - nostrademons 3y ago
    Same thing, but paid in Monero.
  - seanhunter 3y ago
    You don't want a criminal law AI... you want a criminal law AI.
  - fhd2 3y ago
    Would probably lead to an uptick in tourism for Belize.
  - hammock 3y ago
    Write an amicus brief for a Supreme Court case in the style of Saul Goodman UNITED STATES SUPREME COURT Saul Goodman's Legal Advocacy Clinic 123 Shady Lane Albuquerque, NM 87102 Phone: (555) 123-4567 Email: saulgoodman@lawyerextraordinaire.com May 27, 2023 JUSTICES OF THE SUPREME COURT Re: Amicus Curiae Brief - Case No. 12345 Dear Honorable Justices, I, Saul Goodman, attorney-at-law and mastermind of legal brilliance, on behalf of the humble but feisty David Rosencrantz, submit this amicus curiae brief to the esteemed Supreme Court. This Court, known for its unflinching pursuit of justice, now has the opportunity to rectify a grave miscarriage of justice perpetrated by the lower courts. SUMMARY OF ARGUMENT The case before this honorable Court presents a classic tale of a downtrodden individual, David Rosencrantz, pitted against the formidable Goliath that is the United States government. Rosencrantz, a simple taxi driver trying to earn an honest buck in this dog-eat-dog world, has been unjustly accused of tax evasion. However, this Amicus respectfully submits that the government's case is nothing but smoke and mirrors, an elaborate scheme to crush the dreams of an honest, hardworking citizen. ARGUMENT I. Violation of Due Process The lower courts, in their zealous pursuit of victory, trampled upon the sacred principles of due process. Mr. Rosencrantz was denied his fundamental right to a fair trial when the government's star witness, a notorious jailbird with a penchant for singing, was allowed to testify without any corroborating evidence. The government's reliance on the word of a jailhouse informant, who likely had ulterior motives, is an affront to the Constitution. II. Reasonable Doubt Ladies and gentlemen of the Court, this case is riddled with reasonable doubt like a cheap suit in a thrift store. The government's case rests on flimsy evidence and a convoluted web of speculation. The alleged tax discrepancies are based on questionable calculations and fuzzy math, all while conveniently ignoring legitimate deductions and expenses rightfully claimed by Mr. Rosencrantz. Reasonable doubt casts its long shadow on this entire proceeding, and it is the duty of this Court to bring it into the light. III. Selective Prosecution This Court has never turned a blind eye to injustice, and I implore you to scrutinize the government's motives. Mr. Rosencrantz, an unassuming member of society, has become the target of selective prosecution. Why, might you ask? It's because he dared to stand up against the establishment, questioning the powers that be. This case is not about taxes; it is about silencing dissent and punishing those who refuse to play by their rigged rules. CONCLUSION In conclusion, the government's case against David Rosencrantz reeks of a witch hunt, an attempt to crush the spirit of an ordinary citizen who dares to question authority. The violation of due process, the presence of reasonable doubt, and the specter of selective prosecution are grave injustices that cannot be ignored. This Court, armed with the sword of justice, must rise above the fog of deceit and deliver a resounding verdict in favor of Mr. Rosencrantz. For these reasons, I, Saul Goodman, implore this Court to right the wrongs committed by the lower courts and restore faith in the sanctity of our legal system. Respectfully submitted, Saul Goodman, Esq. Attorney for Amicus Curiae Saul Goodman's Legal Advocacy Clinic
babelfish 3y ago
Don’t use GPT for fact finding. Use it for synthesis when you already have the facts.
- _cenw 3y ago
  Or use it for fact discovery if you're not well-versed in a field. But always check if it hallucinated reasonable sounding gibberish.
  - ttul 3y ago
    Or use it to retrieve alleged facts from its own output, pass those alleged facts to another tool (like a legal search engine) for verification, and then get GPT to edit its output accordingly…
  - whatever1 3y ago
    The thing is that if you are not an expert in the field you cannot tell gibberish from legit facts. Especially if the writing style and grammar are top-notch. I know it is a bad bias, but we typically associate good and clear writing with legitimacy. Here we have chat GPT, that can do exactly that, but spit out complete bs.
    - hosh 3y ago
      One thing I like about phind.com is that it ties the specific assertions to the specific web page it made it from. That allows me to check into the sources. However, like all generative AI, it’s good at forming narratives, and not many people are aware how powerfully influential narrative frames are because people rarely step back to examine the frame itself.
      - ayewo 3y ago
        Yep. People rarely step back to examine how a narrative was framed. It is a cognitive bias [1] built into our brains. It takes a lot of mental effort to spot, which is why we don’t do it often. 1: https://en.wikipedia.org/wiki/Framing_effect_(psychology) https://en.wikipedia.org/wiki/Framing_effect_(psychology)
entwife 3y ago
I'm only peripherally involved in the law field, and I am aware that ChatGPT and similar do not consistently provide correct legal citations. It also gets things wrong some of the time; the only way to tell is to be an expert or to look up the facts yourself. Don't use ChatGPT for legal writing, nor for anything else that requires accuracy.
- mountainb 3y ago
  When it's correct, it can be a good search aid. But for a lot of things, it is just incorrect with high levels of confidence. You can also ask it Bluebook questions and it will often get the right answer. At other times, it will get the right answer but cite to the wrong rule (not that it matters that much). Another issue is that it can cite to the correct case, but misunderstand what it is citing to. You can be really specific and ask something like "what is the x-factor test from Doe v. Doe" and it will get three factors correct and invent the other two. The thing with law, though, is that there are often already many quick reference materials that have already been extensively published that will get you the answer you are looking for more quickly than you can get it through either search or a chat interface. Many state bar associations make available the equivalent of a "practice area in a box" full of checklists, templates, and other material geared towards making it possible for you to start working in that area almost immediately. I have had it be useful in course correcting my research in an unfamiliar area of law. I was wasting a lot of time reading secondary sources and cases that were not relevant to my problem because I knew nothing about that area of law and my search queries were just leading me in unproductive directions. ChatGPT pointed me towards a more relevant case that opened up the rest of my research for me using conventional tools like Westlaw. It saved me a lot of time. But I did not use it at all for the final work product and never used it blind without looking at a source.
  - visarga 3y ago
    That's right, generative model outputs are worthless unless checked by a human. You are using it right. For the moment I don't think there is any single domain where AI can work on its own, autonomy was reached in 0% of fields. That makes me think the removal of the human in the loop will take a long time. We are still safe, AI will be our sidekick. It's crazy how AI seems to progress at incredible speed and yet we don't get closer to full autonomy anywhere. It's as if we discover new problems at the same speed we are solving them. Just 5 years ago nobody would think hallucinations will become a central issue in AI, we might discover other unknown unknowns that hide in our future.
montroser 3y ago
> Mr. Schwartz said that he had never used ChatGPT, and “therefore was unaware of the possibility that its content could be false.” > He had, he told Judge Castel, even asked the program to verify that the cases were real. These two ideas are incompatible with each other. You can't claim that you didn't know to question the source, and then also that you questioned the source, even if it was done in the least effective possible manner.
- NeoTar 3y ago
  Perhaps logically incompatible, but within legal proceedings you are allowed to use what is called alternative pleading, or alternative defence. https://en.m.wikipedia.org/wiki/Alternative_pleading https://en.m.wikipedia.org/wiki/Alternative_pleading To quote Richard "Racehorse" Haynes in the Wikipedia article: "Say you sue me because you say my dog bit you. Well, now this is my defense: My dog doesn't bite. And second, in the alternative, my dog was tied up that night. And third, I don't believe you really got bit. And fourth, I don't have a dog." So, here the defence is: * I didn't believe the content could be false, * Even if it is legally determined that I (beyond a reasonable doubt) knew the the content could be false, I asked the program to verify that the cases were real. There are more details in the wikipedia article, but I believe this is legally valuable because a defendant is required to legally enter a defence and cannot easily change this.
  - codeflo 3y ago
    I don't even think it's logically incompatible. Isn't this just ((A → B) ∧ (¬A → B)) → B?
    - sowbug 3y ago
      Perhaps, but it can be confusing if the listener interprets it as a witness making a factual statement, leading to the rhetorical question "were you lying when you said A, or else when you said not-A?" For legal arguments, it's more like "we contend that you can't prove A, and even if you can, you can't also prove B, and both A and B must be proven for legal liability." Which most people can understand isn't inconsistent at all. That's why the legal & ethical guidelines spell it out.
shalalalaw 3y ago
We're a very tech forward law firm, and we're bullish on AI. The issue is that lawyers are traditionally tech illiterate, and they treat Gen AI like a search engine that puts results in narrative form. Realistically, I think AI generated motions and contracts are the future, and this instance will be looked at by every tech averse lawyer to try and stymie progress in the field. These lawyers deserve their sanctions for being so reckless with things they don't understand, but rather than take away that lawyers need to learn tech, lawyers will say tech is bad. I almost wish this was non-news so it wouldn't further push the legal industry into the past, but those clients were wronged and I guess people need to know what to beware of when hiring a lawyer. Personally, we get really good results from using AI, it's already present in all of our processes, but we tell it what to generate, rather than rely on it to know better.
- codeflo 3y ago
  It's not just lawyers who think that ChatGPT is a search engine. I've observed this many times in my vicinity, people from all walks of life think that Star Trek is here and computers now respond accurately to natural language queries. For non-techies, "just asking the computer" is so much more convenient than translating your question into traditional search queries. So I guarantee you that stuff like this is happening daily across all industries. Depending on the profession, people will lose money or get hurt as a result of someone blindly trusting this technology. I can't prove it, but statistically, that's basically a certainty. In my opinion, you can't overstate the importance of articles like this, which point out the limitations and highlight the dangers. I'm also against banning. But lay people need to be informed what ChatGPT is and is not, and OpenAI won't do it because they want to ride the hype train.
  - fennecfoxy 3y ago
    Idk it's pretty reliable even now if you chuck a vector db of "knowledge" in and inform the GPT in the overall prompt that it is must not go outside the bounds of knowledge provided with the user's query, and that if no hard knowledge is provided that it should respond along the lines of "I don't know" (or search the internet and parse results for them). I imagine at some point this behaviour will be built in. People are treating GPTs like they're knowledge bases, but they're not. The fact that they can answer simple or complex questions correctly is only a byproduct of the transformer being taught how to string language together. It's like an artist who learns to draw by making paintings of raccoons. You can't then ask them "what do raccoons like to eat?" or "what foods are poisonous to a raccoon?" just because they learnt to draw by painting raccoons. This is how people are treating GPTs atm. They believe in it because they ask the artist "what colour fur do raccoons have?" and because the artist can answer that correctly, they assume all other answers are factual.
SanderNL 3y ago
Lawyer uses completely new tech without so much as glancing over its properties and failure modes, also known as being completely incompetent. Breaking news: the results are not fantastic.
dpflan 3y ago
Events like this are needed to show to society the flaws of the technology and possible misuse. They can then lead to regulations or guidelines, or even lawsuits that can further define how and when this technology can be used.
naet 3y ago
In a thread on Reddit I saw some people who were using ChatGPT as their personal lawyer around some tech issues, and the community seemed to have absolute confidence it was providing good advice. I tried to comment that I wouldn't trust GPT as a legal reference and got downvoted into the deep negatives. I'm guessing we will see a lot more of these type of issues. ChatGPT is sometimes useful but sometimes full of misinformation and many people don't seem to be wary enough of the potential for error.
- misnome 3y ago
  Unpossible! I’m a 20x coder who spends hours a day writing regex, tried asking it a high-school level coding question, and it answered perfectly. I’m 100% sure that its just as accurate everywhere else.
wseqyrku 3y ago
They should've used GPT-4.
- wseqyrku 3y ago
  (The article does not say anything about this but I'd guess if they are as clueless as they claim to be by calling it "a source that has revealed itself to be unreliable", they might as well be using the free version without the knowledge of the latest version.)
  - codeflo 3y ago
    It's uncanny how precisely you've put your finger on what the actual issue is here.
hartator 3y ago
> Judge Castel said in an order that he had been presented with “an unprecedented circumstance,” a legal submission replete with “bogus judicial decisions, with bogus quotes and bogus internal citations.” He ordered a hearing for June 8 to discuss potential sanctions. Disbarment should be a no-brainer and a minimum. Just “Southern China Airlines” should have raised eyebrows. This lawyer has shown obvious disrespect to the court, the court time, and to his client.
- paul_f 3y ago
  The lawyer should lose his license. Imagine they had turned in this brief and follow up and ChatGPT had not been involved. Instant disbarment. ChatGPT is not an excuse for a professional to produce nonsense in front of the court. Bye bye.
  - psychphysic 3y ago
    Disbarment seems excessive to me. I don't think this is indictiative of a wider issue. Or likely to be repeated substantially. In terms of severity it's painfully foolish but then again ChatGPT is a totally new tool and a lot of people will be caught off guard. I am stunned the lawyer didn't at least look up the case notes or even prepare a pocket brief if he believed they were real but hard to find
    - unyttigfjelltol 3y ago
      Yeah, interestingly, being bad at lawyering isn't a typical reason for discipline. Discipline is integrity and process-based, things like stealing client funds and failing to communicate promptly and reasonably with the client. Arguably the leading goof was using a technology the lawyer didn't understand and failing to inform the client of the risks of using it. Between that and citing garbage precedent-- half the bar might be eligible for discipline on any given day. The judge might issue sanctions but bar discipline is a different ball of wax.
      - filoleg 3y ago
croes 3y ago
Could be a clever method to make programs like ChatGPT untrustworthy for potential clients.
julienchastang 3y ago
Related article: "End of the Billable Hour? Law Firms Get On Board With Artificial Intelligence, Lawyers start to use GPT-4 technology to do legal research, draft documents and analyze contracts" [0] Critical thinking skills are more important than ever in the age of AI. Used correctly, ChatGPT(4) can sometimes be a huge time saver, but you cannot believe all the bullshit it serves you. [0] https://www.wsj.com/articles/end-of-the-billable-hour-law-firms-get-on-board-with-artificial-intelligence-17ebd3f8 https://www.wsj.com/articles/end-of-the-billable-hour-law-fi...
isp 3y ago
I've been following this from legal circles Original court documents: https://www.courtlistener.com/docket/63107798/mata-v-avianca-inc/ https://www.courtlistener.com/docket/63107798/mata-v-avianca... The lawyer didn't only cite "bogus" cases, but when challenged attached entire "bogus" case contents hallucinated by ChatGPT (attachments to #29 on link above) In the second #32 affidavit, there are screenshots of ChatGPT itself! https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.32.1_1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57... A legendary example for the legal risks of hallucination in LLMs https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence) https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...
- _-____-_ 3y ago
  That the screenshots are from the mobile website for some reason makes this look even worse.
  - spondylosaurus 3y ago
    It does. It's like this lawyer is charging you his hourly rate just to prompt ChatGPT while he's on the toilet.
- asmithmd1 3y ago
  Thanks for the great context. The lawyer should be disbarred. He doubled down when he was caught, and then blamed chatGPT. What do you bet he was trying to settle really quickly to make this all go away. Here is the direct link to the chatGPT hallucination the lawyer filed in response to the judge's order to produce the actual text of the case: https://www.courtlistener.com/docket/63107798/29/1/mata-v-avianca-inc/ https://www.courtlistener.com/docket/63107798/29/1/mata-v-av...
  - BaseballPhysics 3y ago
    Did he "double down" or did he genuinely not understand that ChatGPT was making stuff up the whole time?
b1ue64 3y ago
https://archive.ph/tHuXv https://archive.ph/tHuXv
deleted 3y ago
[deleted]
squarefoot 3y ago
Could he sue his own lawyer now?
NelsonMinar 3y ago
OpenAI deserves some blame here. Their product UI does a very poor job conveying that it will just make things up when asked for specific information. And in the conversation there's no particular hint that the confidently stated "facts" are not in fact true. A large number of people get tripped up by this when first using ChatGPT. I did myself, it took me a day or two to realize the papers or web pages it was claiming existed didn't. It's pathological.
- scrollaway 3y ago
  This drawback is presented front and center every time you open ChatGPT. A lawyer of all people should have known better.
joshka 3y ago
@dang - the earlier article on reason.com is more thorough - perhaps merge these two posts? https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509 https://reason.com/volokh/2023/05/27/a-lawyers-filing-is-replete-with-citations-to-non-existent-cases-thanks-chatgpt/ https://reason.com/volokh/2023/05/27/a-lawyers-filing-is-rep...
acd 3y ago
I think it should be required to state if a written statement is written by a GPT. "This text section is written by GPT provided by company XYZ" Then humans would know that a particular part is written by an AI.
- GaggiX 3y ago
  >Then humans would know that a particular part is written by an AI. Humans would know if a part is written by an AI or not only if there is a way to enforce the rule.
Reason077 3y ago
"Estate of Durden v. KLM Royal Dutch Airlines" Tyler Durden? LOL, you couldn't make this stuff up. Unless you're ChatGPT, obviously.
adql 3y ago
Can't bullshit people who specialize at calling the other side's bullshit. Not that there can't be use for it,I'm sure it could work as great assisted search engine if it was made to cite sources.
Am4TIfIsER0ppos 3y ago
I can't wait for chatgpt equivalent of citogenesis (https://xkcd.com/978/ https://xkcd.com/978/) to happen.
simonw 3y ago
I tried pulling together a full timeline from the various documents, it's a fascinating story: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/
jhncls 3y ago
Even if the lawyer would have carefully checked and corrected ChatGPT's answers, isn't he going against his professional duty by submitting private information to a website that doesn't ensure confidentiality?
cratermoon 3y ago
I was able to coax ChatGPT into inventing an entirely fake trend in cocktails and distilling, complete with stories about newly-opening bars and made-up concoctions (including recipes) supposedly created and served at these bars.
andrewmg 3y ago
Some context: any litigator will have access to Westlaw or Lexis-Nexis to look up and verify cited authorities like cases. It’s considered bad practice, at best, to cite authorities that one has not reviewed—for example, case citations drawn from a treatise or article. As a practical matter, it is inconceivable to me that the attorney here, at least upon being ordered by the court to provide copies of the cases he cited, did not look them up in West or Lexis and see that they don’t exist. That he appears to have pressed on at that point, and asked ChatGPT to generate them—which would take some pointed prompting—was just digging his own hole. That, more than anything, may warrant professional discipline.
- nightowl_games 3y ago
  The precision with which you wrote this is refreshing. I'm guessing you're a lawyer? There are interesting parallels between lawyering and programming. I'm often surprised at how poorly many programmers write English. I think I'd have been a good lawyer.
dpifke 3y ago
Previous discussion: https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509
nunez 3y ago
I guess it will take someone dying by following ChatGPTs "advice" to make people think twice about using these LLMs!

174 points by nlh 3y ago

GPT4 hooked up to a legal DB and instructed that if no knowledge from the DB can be referenced to respond negatively would've solved this completely. Even a hard link to rewrite the prompt from "use provided knowledge to answer the question" to "no knowledge could be found, inform the user of this fact in a way that keeps the conversation fluid. Recommend that they search other sources themselves." would be more reliable. The problem, as always, isn't the tool. It's the tool using the tool.

krimbus 3y ago
Could we have certainty that an LLM would comply though? I can only imagine this happening with external constraints such as parsing the output to perform a DB check, so I'm curious.
- fennecfoxy 3y ago
  Hmmm, I suppose we could, if there was a focus on it. However it's always interesting to watch how people react: * Traditional programming creates very strict functionality but is not flexible/is rigid. * Okay, so let's train models to act like biological life, which is super flexible but has a chance to make mistakes. But then people complain about the latter making mistakes. Realistically we can combine the two to ensure that what a model is saying is based on fact, though. Easiest way would be just for people to interact with an LLM to browse hard facts and then double check via sources. But relying on humans not to be lazy is just as good as relying on an ML model to act like traditional programming, so it's probably better to train our wild model with as many built-in constraints as possible and then decorate those with more traditionally programmed hard limits. It's really hard though because of the output data. We're going from a binary output of 1/0 for traditional programming to a continuous output of 0-1 for models; if I ask it to summarise something, whether it's correct or not is not only based on whether it summarised based only on the original text, but also down to the reviewer's individual biases and wants/needs as to whether the summarisation is sufficient. The continuous nature of an LLM's response does make it difficult to determine if it's sufficiently factually correct, though. Because if it's not repeating word for word, you need to be able to parse the output in order to check it...using another LLM (so compounding error...)

jaclaz 3y ago

Another thread where the poster linked to the relevant Court .pdf's: https://news.ycombinator.com/item?id=36092914 https://news.ycombinator.com/item?id=36092914 Particularly relevant is the affidavit in which the lawyer tries to explain to the Court what happened: https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.32.1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...

lionkor 3y ago
Hahaha that second link is brilliant. "Hm, maybe I should double check ChatGPTs output... Hey, ChatGPT, does your output make sense?" - "Yeah, my output definitely makes sense". "Are you sure?" - "Yeah". Well, then.
- namaria 3y ago
  >Mr. Schwartz said that he had never used ChatGPT, and “therefore was unaware of the possibility that its content could be false.” Why did he think it was true, if he had never used it before then? "I found it online" levels of competence here.
  - vidarh 3y ago
    People are scarily willing to take eloquent text on trust if it says something that benefits them and sounds plausible. The biggest problem with ChatGPT is that it sounds like a smart adult that people want to give the benefit of the doubt because it agrees with them a lot instead of like a child that people would be more wary of trusting blindly.
    - namaria 3y ago
      This willingness most people show to jump on the bandwagon when hearing confident sounding language actually explains a lot of societal problems. Also a lot of workplace dynamics, and the career of a lot of very powerful and rich people.

Szpadel 3y ago

archive.org link https://web.archive.org/web/20230527153017/https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html/ https://web.archive.org/web/20230527153017/https://www.nytim...

metadat 3y ago
https://archive.today/pnEhz https://archive.today/pnEhz

m348e912 3y ago

This problem will be solved with LawGPT which will be a generative pre-trained transformer trained on a vast array of legal cases, rulings, and briefings and designed with accuracy in mind and free from hallucinations. It will cost $3,400/month for a legal firm the size of Levidow, Levidow & Oberman, P.C.

ChuckNorris89 3y ago
Still waiting for SaulGoodmanGPT
- nostrademons 3y ago
  Same thing, but paid in Monero.
- seanhunter 3y ago
  You don't want a criminal law AI... you want a criminal law AI.
- fhd2 3y ago
  Would probably lead to an uptick in tourism for Belize.
- hammock 3y ago
  Write an amicus brief for a Supreme Court case in the style of Saul Goodman UNITED STATES SUPREME COURT Saul Goodman's Legal Advocacy Clinic 123 Shady Lane Albuquerque, NM 87102 Phone: (555) 123-4567 Email: saulgoodman@lawyerextraordinaire.com May 27, 2023 JUSTICES OF THE SUPREME COURT Re: Amicus Curiae Brief - Case No. 12345 Dear Honorable Justices, I, Saul Goodman, attorney-at-law and mastermind of legal brilliance, on behalf of the humble but feisty David Rosencrantz, submit this amicus curiae brief to the esteemed Supreme Court. This Court, known for its unflinching pursuit of justice, now has the opportunity to rectify a grave miscarriage of justice perpetrated by the lower courts. SUMMARY OF ARGUMENT The case before this honorable Court presents a classic tale of a downtrodden individual, David Rosencrantz, pitted against the formidable Goliath that is the United States government. Rosencrantz, a simple taxi driver trying to earn an honest buck in this dog-eat-dog world, has been unjustly accused of tax evasion. However, this Amicus respectfully submits that the government's case is nothing but smoke and mirrors, an elaborate scheme to crush the dreams of an honest, hardworking citizen. ARGUMENT I. Violation of Due Process The lower courts, in their zealous pursuit of victory, trampled upon the sacred principles of due process. Mr. Rosencrantz was denied his fundamental right to a fair trial when the government's star witness, a notorious jailbird with a penchant for singing, was allowed to testify without any corroborating evidence. The government's reliance on the word of a jailhouse informant, who likely had ulterior motives, is an affront to the Constitution. II. Reasonable Doubt Ladies and gentlemen of the Court, this case is riddled with reasonable doubt like a cheap suit in a thrift store. The government's case rests on flimsy evidence and a convoluted web of speculation. The alleged tax discrepancies are based on questionable calculations and fuzzy math, all while conveniently ignoring legitimate deductions and expenses rightfully claimed by Mr. Rosencrantz. Reasonable doubt casts its long shadow on this entire proceeding, and it is the duty of this Court to bring it into the light. III. Selective Prosecution This Court has never turned a blind eye to injustice, and I implore you to scrutinize the government's motives. Mr. Rosencrantz, an unassuming member of society, has become the target of selective prosecution. Why, might you ask? It's because he dared to stand up against the establishment, questioning the powers that be. This case is not about taxes; it is about silencing dissent and punishing those who refuse to play by their rigged rules. CONCLUSION In conclusion, the government's case against David Rosencrantz reeks of a witch hunt, an attempt to crush the spirit of an ordinary citizen who dares to question authority. The violation of due process, the presence of reasonable doubt, and the specter of selective prosecution are grave injustices that cannot be ignored. This Court, armed with the sword of justice, must rise above the fog of deceit and deliver a resounding verdict in favor of Mr. Rosencrantz. For these reasons, I, Saul Goodman, implore this Court to right the wrongs committed by the lower courts and restore faith in the sanctity of our legal system. Respectfully submitted, Saul Goodman, Esq. Attorney for Amicus Curiae Saul Goodman's Legal Advocacy Clinic

babelfish 3y ago

Don’t use GPT for fact finding. Use it for synthesis when you already have the facts.

_cenw 3y ago
Or use it for fact discovery if you're not well-versed in a field. But always check if it hallucinated reasonable sounding gibberish.
- ttul 3y ago
  Or use it to retrieve alleged facts from its own output, pass those alleged facts to another tool (like a legal search engine) for verification, and then get GPT to edit its output accordingly…
- whatever1 3y ago
  The thing is that if you are not an expert in the field you cannot tell gibberish from legit facts. Especially if the writing style and grammar are top-notch. I know it is a bad bias, but we typically associate good and clear writing with legitimacy. Here we have chat GPT, that can do exactly that, but spit out complete bs.
  - hosh 3y ago
    One thing I like about phind.com is that it ties the specific assertions to the specific web page it made it from. That allows me to check into the sources. However, like all generative AI, it’s good at forming narratives, and not many people are aware how powerfully influential narrative frames are because people rarely step back to examine the frame itself.
    - ayewo 3y ago
      Yep. People rarely step back to examine how a narrative was framed. It is a cognitive bias [1] built into our brains. It takes a lot of mental effort to spot, which is why we don’t do it often. 1: https://en.wikipedia.org/wiki/Framing_effect_(psychology) https://en.wikipedia.org/wiki/Framing_effect_(psychology)

entwife 3y ago

I'm only peripherally involved in the law field, and I am aware that ChatGPT and similar do not consistently provide correct legal citations. It also gets things wrong some of the time; the only way to tell is to be an expert or to look up the facts yourself. Don't use ChatGPT for legal writing, nor for anything else that requires accuracy.

mountainb 3y ago
When it's correct, it can be a good search aid. But for a lot of things, it is just incorrect with high levels of confidence. You can also ask it Bluebook questions and it will often get the right answer. At other times, it will get the right answer but cite to the wrong rule (not that it matters that much). Another issue is that it can cite to the correct case, but misunderstand what it is citing to. You can be really specific and ask something like "what is the x-factor test from Doe v. Doe" and it will get three factors correct and invent the other two. The thing with law, though, is that there are often already many quick reference materials that have already been extensively published that will get you the answer you are looking for more quickly than you can get it through either search or a chat interface. Many state bar associations make available the equivalent of a "practice area in a box" full of checklists, templates, and other material geared towards making it possible for you to start working in that area almost immediately. I have had it be useful in course correcting my research in an unfamiliar area of law. I was wasting a lot of time reading secondary sources and cases that were not relevant to my problem because I knew nothing about that area of law and my search queries were just leading me in unproductive directions. ChatGPT pointed me towards a more relevant case that opened up the rest of my research for me using conventional tools like Westlaw. It saved me a lot of time. But I did not use it at all for the final work product and never used it blind without looking at a source.
- visarga 3y ago
  That's right, generative model outputs are worthless unless checked by a human. You are using it right. For the moment I don't think there is any single domain where AI can work on its own, autonomy was reached in 0% of fields. That makes me think the removal of the human in the loop will take a long time. We are still safe, AI will be our sidekick. It's crazy how AI seems to progress at incredible speed and yet we don't get closer to full autonomy anywhere. It's as if we discover new problems at the same speed we are solving them. Just 5 years ago nobody would think hallucinations will become a central issue in AI, we might discover other unknown unknowns that hide in our future.

montroser 3y ago

> Mr. Schwartz said that he had never used ChatGPT, and “therefore was unaware of the possibility that its content could be false.” > He had, he told Judge Castel, even asked the program to verify that the cases were real. These two ideas are incompatible with each other. You can't claim that you didn't know to question the source, and then also that you questioned the source, even if it was done in the least effective possible manner.

NeoTar 3y ago
Perhaps logically incompatible, but within legal proceedings you are allowed to use what is called alternative pleading, or alternative defence. https://en.m.wikipedia.org/wiki/Alternative_pleading https://en.m.wikipedia.org/wiki/Alternative_pleading To quote Richard "Racehorse" Haynes in the Wikipedia article: "Say you sue me because you say my dog bit you. Well, now this is my defense: My dog doesn't bite. And second, in the alternative, my dog was tied up that night. And third, I don't believe you really got bit. And fourth, I don't have a dog." So, here the defence is: * I didn't believe the content could be false, * Even if it is legally determined that I (beyond a reasonable doubt) knew the the content could be false, I asked the program to verify that the cases were real. There are more details in the wikipedia article, but I believe this is legally valuable because a defendant is required to legally enter a defence and cannot easily change this.
- codeflo 3y ago
  I don't even think it's logically incompatible. Isn't this just ((A → B) ∧ (¬A → B)) → B?
  - sowbug 3y ago
    Perhaps, but it can be confusing if the listener interprets it as a witness making a factual statement, leading to the rhetorical question "were you lying when you said A, or else when you said not-A?" For legal arguments, it's more like "we contend that you can't prove A, and even if you can, you can't also prove B, and both A and B must be proven for legal liability." Which most people can understand isn't inconsistent at all. That's why the legal & ethical guidelines spell it out.

shalalalaw 3y ago

We're a very tech forward law firm, and we're bullish on AI. The issue is that lawyers are traditionally tech illiterate, and they treat Gen AI like a search engine that puts results in narrative form. Realistically, I think AI generated motions and contracts are the future, and this instance will be looked at by every tech averse lawyer to try and stymie progress in the field. These lawyers deserve their sanctions for being so reckless with things they don't understand, but rather than take away that lawyers need to learn tech, lawyers will say tech is bad. I almost wish this was non-news so it wouldn't further push the legal industry into the past, but those clients were wronged and I guess people need to know what to beware of when hiring a lawyer. Personally, we get really good results from using AI, it's already present in all of our processes, but we tell it what to generate, rather than rely on it to know better.

codeflo 3y ago
It's not just lawyers who think that ChatGPT is a search engine. I've observed this many times in my vicinity, people from all walks of life think that Star Trek is here and computers now respond accurately to natural language queries. For non-techies, "just asking the computer" is so much more convenient than translating your question into traditional search queries. So I guarantee you that stuff like this is happening daily across all industries. Depending on the profession, people will lose money or get hurt as a result of someone blindly trusting this technology. I can't prove it, but statistically, that's basically a certainty. In my opinion, you can't overstate the importance of articles like this, which point out the limitations and highlight the dangers. I'm also against banning. But lay people need to be informed what ChatGPT is and is not, and OpenAI won't do it because they want to ride the hype train.
- fennecfoxy 3y ago
  Idk it's pretty reliable even now if you chuck a vector db of "knowledge" in and inform the GPT in the overall prompt that it is must not go outside the bounds of knowledge provided with the user's query, and that if no hard knowledge is provided that it should respond along the lines of "I don't know" (or search the internet and parse results for them). I imagine at some point this behaviour will be built in. People are treating GPTs like they're knowledge bases, but they're not. The fact that they can answer simple or complex questions correctly is only a byproduct of the transformer being taught how to string language together. It's like an artist who learns to draw by making paintings of raccoons. You can't then ask them "what do raccoons like to eat?" or "what foods are poisonous to a raccoon?" just because they learnt to draw by painting raccoons. This is how people are treating GPTs atm. They believe in it because they ask the artist "what colour fur do raccoons have?" and because the artist can answer that correctly, they assume all other answers are factual.

SanderNL 3y ago

Lawyer uses completely new tech without so much as glancing over its properties and failure modes, also known as being completely incompetent. Breaking news: the results are not fantastic.

dpflan 3y ago

Events like this are needed to show to society the flaws of the technology and possible misuse. They can then lead to regulations or guidelines, or even lawsuits that can further define how and when this technology can be used.

naet 3y ago

In a thread on Reddit I saw some people who were using ChatGPT as their personal lawyer around some tech issues, and the community seemed to have absolute confidence it was providing good advice. I tried to comment that I wouldn't trust GPT as a legal reference and got downvoted into the deep negatives. I'm guessing we will see a lot more of these type of issues. ChatGPT is sometimes useful but sometimes full of misinformation and many people don't seem to be wary enough of the potential for error.

misnome 3y ago
Unpossible! I’m a 20x coder who spends hours a day writing regex, tried asking it a high-school level coding question, and it answered perfectly. I’m 100% sure that its just as accurate everywhere else.

wseqyrku 3y ago

They should've used GPT-4.

wseqyrku 3y ago
(The article does not say anything about this but I'd guess if they are as clueless as they claim to be by calling it "a source that has revealed itself to be unreliable", they might as well be using the free version without the knowledge of the latest version.)
- codeflo 3y ago
  It's uncanny how precisely you've put your finger on what the actual issue is here.

hartator 3y ago

> Judge Castel said in an order that he had been presented with “an unprecedented circumstance,” a legal submission replete with “bogus judicial decisions, with bogus quotes and bogus internal citations.” He ordered a hearing for June 8 to discuss potential sanctions. Disbarment should be a no-brainer and a minimum. Just “Southern China Airlines” should have raised eyebrows. This lawyer has shown obvious disrespect to the court, the court time, and to his client.

paul_f 3y ago
The lawyer should lose his license. Imagine they had turned in this brief and follow up and ChatGPT had not been involved. Instant disbarment. ChatGPT is not an excuse for a professional to produce nonsense in front of the court. Bye bye.
- psychphysic 3y ago
  Disbarment seems excessive to me. I don't think this is indictiative of a wider issue. Or likely to be repeated substantially. In terms of severity it's painfully foolish but then again ChatGPT is a totally new tool and a lot of people will be caught off guard. I am stunned the lawyer didn't at least look up the case notes or even prepare a pocket brief if he believed they were real but hard to find
  - unyttigfjelltol 3y ago
    Yeah, interestingly, being bad at lawyering isn't a typical reason for discipline. Discipline is integrity and process-based, things like stealing client funds and failing to communicate promptly and reasonably with the client. Arguably the leading goof was using a technology the lawyer didn't understand and failing to inform the client of the risks of using it. Between that and citing garbage precedent-- half the bar might be eligible for discipline on any given day. The judge might issue sanctions but bar discipline is a different ball of wax.
    - filoleg 3y ago

croes 3y ago

Could be a clever method to make programs like ChatGPT untrustworthy for potential clients.

julienchastang 3y ago

Related article: "End of the Billable Hour? Law Firms Get On Board With Artificial Intelligence, Lawyers start to use GPT-4 technology to do legal research, draft documents and analyze contracts" [0] Critical thinking skills are more important than ever in the age of AI. Used correctly, ChatGPT(4) can sometimes be a huge time saver, but you cannot believe all the bullshit it serves you. [0] https://www.wsj.com/articles/end-of-the-billable-hour-law-firms-get-on-board-with-artificial-intelligence-17ebd3f8 https://www.wsj.com/articles/end-of-the-billable-hour-law-fi...

isp 3y ago

I've been following this from legal circles Original court documents: https://www.courtlistener.com/docket/63107798/mata-v-avianca-inc/ https://www.courtlistener.com/docket/63107798/mata-v-avianca... The lawyer didn't only cite "bogus" cases, but when challenged attached entire "bogus" case contents hallucinated by ChatGPT (attachments to #29 on link above) In the second #32 affidavit, there are screenshots of ChatGPT itself! https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.32.1_1.pdf https://storage.courtlistener.com/recap/gov.uscourts.nysd.57... A legendary example for the legal risks of hallucination in LLMs https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence) https://en.wikipedia.org/wiki/Hallucination_(artificial_inte...

_-____-_ 3y ago
That the screenshots are from the mobile website for some reason makes this look even worse.
- spondylosaurus 3y ago
  It does. It's like this lawyer is charging you his hourly rate just to prompt ChatGPT while he's on the toilet.
asmithmd1 3y ago
Thanks for the great context. The lawyer should be disbarred. He doubled down when he was caught, and then blamed chatGPT. What do you bet he was trying to settle really quickly to make this all go away. Here is the direct link to the chatGPT hallucination the lawyer filed in response to the judge's order to produce the actual text of the case: https://www.courtlistener.com/docket/63107798/29/1/mata-v-avianca-inc/ https://www.courtlistener.com/docket/63107798/29/1/mata-v-av...
- BaseballPhysics 3y ago
  Did he "double down" or did he genuinely not understand that ChatGPT was making stuff up the whole time?

b1ue64 3y ago

https://archive.ph/tHuXv https://archive.ph/tHuXv

deleted 3y ago

[deleted]

squarefoot 3y ago

Could he sue his own lawyer now?

NelsonMinar 3y ago

OpenAI deserves some blame here. Their product UI does a very poor job conveying that it will just make things up when asked for specific information. And in the conversation there's no particular hint that the confidently stated "facts" are not in fact true. A large number of people get tripped up by this when first using ChatGPT. I did myself, it took me a day or two to realize the papers or web pages it was claiming existed didn't. It's pathological.

scrollaway 3y ago
This drawback is presented front and center every time you open ChatGPT. A lawyer of all people should have known better.

joshka 3y ago

@dang - the earlier article on reason.com is more thorough - perhaps merge these two posts? https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509 https://reason.com/volokh/2023/05/27/a-lawyers-filing-is-replete-with-citations-to-non-existent-cases-thanks-chatgpt/ https://reason.com/volokh/2023/05/27/a-lawyers-filing-is-rep...

acd 3y ago

I think it should be required to state if a written statement is written by a GPT. "This text section is written by GPT provided by company XYZ" Then humans would know that a particular part is written by an AI.

GaggiX 3y ago
>Then humans would know that a particular part is written by an AI. Humans would know if a part is written by an AI or not only if there is a way to enforce the rule.

Reason077 3y ago

"Estate of Durden v. KLM Royal Dutch Airlines" Tyler Durden? LOL, you couldn't make this stuff up. Unless you're ChatGPT, obviously.

adql 3y ago

Can't bullshit people who specialize at calling the other side's bullshit. Not that there can't be use for it,I'm sure it could work as great assisted search engine if it was made to cite sources.

Am4TIfIsER0ppos 3y ago

I can't wait for chatgpt equivalent of citogenesis (https://xkcd.com/978/ https://xkcd.com/978/) to happen.

simonw 3y ago

I tried pulling together a full timeline from the various documents, it's a fascinating story: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/

jhncls 3y ago

Even if the lawyer would have carefully checked and corrected ChatGPT's answers, isn't he going against his professional duty by submitting private information to a website that doesn't ensure confidentiality?

cratermoon 3y ago

I was able to coax ChatGPT into inventing an entirely fake trend in cocktails and distilling, complete with stories about newly-opening bars and made-up concoctions (including recipes) supposedly created and served at these bars.

andrewmg 3y ago

Some context: any litigator will have access to Westlaw or Lexis-Nexis to look up and verify cited authorities like cases. It’s considered bad practice, at best, to cite authorities that one has not reviewed—for example, case citations drawn from a treatise or article. As a practical matter, it is inconceivable to me that the attorney here, at least upon being ordered by the court to provide copies of the cases he cited, did not look them up in West or Lexis and see that they don’t exist. That he appears to have pressed on at that point, and asked ChatGPT to generate them—which would take some pointed prompting—was just digging his own hole. That, more than anything, may warrant professional discipline.

nightowl_games 3y ago
The precision with which you wrote this is refreshing. I'm guessing you're a lawyer? There are interesting parallels between lawyering and programming. I'm often surprised at how poorly many programmers write English. I think I'd have been a good lawyer.

dpifke 3y ago

Previous discussion: https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509

nunez 3y ago

I guess it will take someone dying by following ChatGPTs "advice" to make people think twice about using these LLMs!