5 ms·

B.C. lawyer reprimanded for citing fake cases invented by ChatGPT

31 points by uladzislau 3y ago

ChrisArchitect 3y ago
Related: Lawyer fined for legal filings that included 'hallucinated' AI citations https://news.ycombinator.com/item?id=39491510 https://news.ycombinator.com/item?id=39491510
loregate 3y ago
Your honor, my source is that I made it the fuck up.
DoreenMichele 3y ago
"I did not intend to generate or refer to fictitious cases in this matter. That is clearly wrong and not something I would knowingly do," Ke wrote in her deposition. "I never had any intention to rely upon any fictitious authorities or to mislead the court." So beware what ChatGPT tells you. It might not be real and this can be a big problem in some situations.
- 255kb 3y ago
  Every time I see a company/startup claiming to do an AI about X or Y, I wonder how they can be trusted at all. What about all the hallucinations, false positives, etc.? Aside from some sort of tool that classify cases, or extract key points from an agreement, and talk in probabilities rather than asserting things I don't see a use for these. And I don't know if we will be able to 100% trust these tools one day.
  - DoreenMichele 3y ago
    We haven't figured out how to deal with human intelligence effectively. It's jumping the gun to imagine we can do AI well. /2 cents -- as if I need a "disclaimer" but the internet being what it is....
    - mdp2021 3y ago
      And yet Columbus made the egg sit firmly on the table. The "«we»" that includes the lawyer in question has so little to do with the "«we»" that may include Richard Feynman or Marvin Minsky (and further in the right directions), that you may want to refine your model past the problem of Porphyry and hyper-realism in the universals: that "«we»" is insubstantial. It is not some collective humanity that arrived to non-Euclidean geometry, but a small number of people that had the skills and dealt with the problem intensively.
      - deleted 3y ago
        [deleted]
      - DoreenMichele 3y ago
        I have absolutely no idea at all what you are saying to me. For clarification of some kind: I homeschooled my twice exceptional --"gifted and (learning) disabled" -- sons and for a time did pro bono professional work for a gifted organization in support of me being the teacher in a private school with two students for some years under California law and designing the custom curriculum for my special-needs sons so as to make it digestible without talking down to them and accommodating their big strengths and big weaknesses at the same time. My emails were seemingly "popular" for a time on a gifted homeschoolers list and you can blame their enthusiasm for my writing for my habit of blogging to this day, though I have no traction and it makes damn little money and is no doubt basically me wasting my life. Fortunately, we are all dust in the wind anyway, so I'm sure none of it really matters, but I have Opinions about the poor job we generally do with dealing with other humans, especially short ones under the legal age of majority. So maybe that gives you some context for my stupid comment. No, I don't care to talk further about any of this really, tbh.
        mdp2021 3y ago
        So I understand you experienced difficulties in dealing with developing intelligences, and you entailed that it should be difficult to create artificial ones. But nobody said it was easy (also because ideas like "to invent the lightbulb is not easy" in that form make little sense), and yet the "right" people - not "«we»" - will probably reach the goal, partially owing to a piece of the "«we»" and largely in spite of a large piece of the "«we»". Just like some achieved the lightbulb, and some (Columbus) achieved making an egg stand straight on the table while those nearby claimed it was impossible. Now, the problem in the original context was not about those people able to create a hammer, but about the people unable to use it. The problem here is not about creating intelligence but using it.
  - pjc50 3y ago
    > What about all the hallucinations, false positives, etc.? There's a very small disclaimer at the bottom that says all those things are your problem.
- Lio 3y ago
  "That is clearly wrong and not something I would knowingly do," Such a bullshit excuse. If you are using ChatGPT to do the donkey work for you then you are knowingly using it to cut corners. It's like turning up in court and saying "well I asked this postman Cliff at the bar I go to his opinion on the case...". She should loose here licence for pulling a stunt like that.
  - ben_w 3y ago
    It's sufficiently good that it fools people into thinking its not even corner-cutting. 85% of the time, the source code it gives me is correct (50% of the time it's mediocre, but still correct), and that's true even when it's surprised me by spewing out an entire single-file web app rather than just disparate functions I was expecting to have to copy-paste into a single file[0]; how good or bad varies wildly in a way that suggests behind the scenes changes like all software, but it's enough that I can easily believe someone sees it on a good day, is convinced it's always that good, and doesn't keep checking. [0] I'm not a web developer, don't bother to tell me there's a better way — my career path (for now) is iOS, where I use the LLM very differently anyway.
  - DoreenMichele 3y ago
    Law isn't my strong point. I have had two Law classes -- environmental law -- and made A minuses in spite of bothering to actually study. So this isn't really an area I feel genuinely qualified to suggest a course of action. But I don't think I have ever used ChatGPT and was blissfully ignorant for some months as to what that hodgepodge of letters even stood for and I'm inclined to be generally appalled by the whole thing. And I'm sure people simultaneously don't actually care about my opinion but will nonetheless not hesitate to hand me my head for saying anything. But if you are more qualified than me and feel strongly about this, perhaps you should do something about it -- something more than being cranky about it on the internet.
- quantified 3y ago
  Assuming you believe the lawyer. The fact that ChatGPT hallucinates citations has been very well publicized and this lawyer would need to have been under the proverbial rock to not know this. My money's on their simply not expecting to get caught.
  - DoreenMichele 3y ago
    I wasn't actually suggesting that I believe the lawyer, fwiw. Just that if you use ChatGPT, "caveat emptor" so to speak.
- justinclift 3y ago
  Directly related to this, there is the recently submitted project named like a famous award: https://news.ycombinator.com/item?id=39514192 https://news.ycombinator.com/item?id=39514192
bell-cot 3y ago
The judge believed the offending lawyer's plea of ignorant incompetence, and declined the other side's request for extra penalties for reprehensible conduct and abuse of process. But still ordered the offender to review all her previous work submitted to the court, to identify any previous submissions of fake cases... So a complete train wreck, that could re-open a bunch of settled cases. Or the offender and the "justice" system might quietly "decide" that they don't want to work hard at making themselves look worse, and just leave a bunch of wrongly-settled cases lie. IANAL, but it seems long past time for judges to start pretending that they actually care about the "truth and justice" stuff, and permabanning lawyers who pull this kind of crap from the practice of law.
- mdp2021 3y ago
  > it seems long past time Before that (yours is just an instance), we need a renewed discipline of individual assessment. Experience is much too full of "...How could you?!"s: it is increasingly important that all effort is spent to have them prevented.
rcbdev 3y ago
Despite what some in the tech space might think, I'm not sure I can lay the blame 100 % on the lawyer. It's true that this is definitely malpractice and they should have known better, it's also true that big tech for years overplayed their hand with the marketing of "AI" to laypeople and management, which has created completely unreasonable expectations. As long as OpenAI has a profit motive to make people believe ChatGPT is "right" more often than not they will never blatantly advertise that everything their product generates might be unverifiable garbage, especially in a way everyone will understand and that isn't just a liability footnote.
- bigstrat2003 3y ago
  Right. I think it's fair to ask whether a reasonable, non-technical person would even know that ChatGPT will hallucinate answers. That is widely known in the tech world (and even still you have people relying on it for work, which blows my mind), but it takes time for that knowledge to get out to the general public. That goes double when vendors are putting out advertising which misleads people (as you pointed out).
  - mdp2021 3y ago
    The last time I tried a LLM based chatbot, for testing purposes, was yesterday, and I had fully clear signals of dealing with a moron. First you have a matter of due diligence when using a tool - especially if you are past the age of twelve, especially if you are a professional -, due diligence that will inform you of the boundaries, but then you also have to have clean senses and be able to see things for what they are as they patently show to be such. It is like having empowered masses of people that in real "darwinian", selective contexts would die before being impactful.
- quantified 3y ago
  The legal peril for using ChatGPT has been well publicized already.
- xxs 3y ago
  so you are saying the lawyer has been conned? I'd not think twice about her not cross checking her sources.
happymellon 3y ago
This appears to be the same case, as reported a month ago. https://news.ycombinator.com/item?id=39115148 https://news.ycombinator.com/item?id=39115148
ifdefdebug 3y ago
They have to realize that the language model did it's job just fine. It produced a convincing piece of well-formed language that fits the context window. If they wanted knowledge instead, they should have used a knowledge model.
- mdp2021 3y ago
  > convincing piece of well-formed language that fits the context window I hope you do realize that "artificial conman" is hardly a goal.
  - miningape 3y ago
    I hope you do realise this is what it was trained to do, not be correct, not be inventive/creative, but to make text that _sounds_ good.
    - mdp2021 3y ago
      I very well realize it, I and I hope that re-reading again my original comment you will see that I am condemning that specifically.
  - graemep 3y ago
    If people are willing to pay for it, then it is.
    - mdp2021 3y ago
      Clearly that is what some seem to think, and it should be also clear that it has sinister sides.
  - rsynnott 3y ago
    ... I mean, given how much money is being spent on LLM-based startups at the moment, it certainly _seems_ to be a goal for a lot of people.
- pjc50 3y ago
mdp2021 3y ago
From instead https://globalnews.ca/news/10263897/fake-ai-cases-b-c-supreme-court/ https://globalnews.ca/news/10263897/fake-ai-cases-b-c-suprem... > Neither side would comment on Wednesday, but Lorne MacLean previously described the threat AI poses to the legal system as grave No, no, no, no, no, no, no. That people believe whatever is in front of them, and that such people are working as professionals instead of pivoting around a couch, is grave. That active morons are taken as a legitimate state of things, is grave. That due diligence was missed, is grave. And appeals in the category of > “Imagine yourself as a young lawyer … she was mortified” [her] lawyer told the court are grave.
rsynnott 3y ago
This seems to be happening quite a lot. Of course, it's particularly likely to be _caught_ early in law; the law is inherently adversarial, and it's a small enough world that people will be conscious that their opponent is the sort of lazy idiot prone to do this. Where I'd _really_ worry about it is in environments where there's an expectation that the 'expert' is operating in good faith. I'd expect over the next few years we'll see some deaths whose root cause is ultimately "the doctor/engineer believed the magic robot".

31 points by uladzislau 3y ago

Related: Lawyer fined for legal filings that included 'hallucinated' AI citations https://news.ycombinator.com/item?id=39491510 https://news.ycombinator.com/item?id=39491510

loregate 3y ago

Your honor, my source is that I made it the fuck up.

DoreenMichele 3y ago

"I did not intend to generate or refer to fictitious cases in this matter. That is clearly wrong and not something I would knowingly do," Ke wrote in her deposition. "I never had any intention to rely upon any fictitious authorities or to mislead the court." So beware what ChatGPT tells you. It might not be real and this can be a big problem in some situations.

255kb 3y ago
Every time I see a company/startup claiming to do an AI about X or Y, I wonder how they can be trusted at all. What about all the hallucinations, false positives, etc.? Aside from some sort of tool that classify cases, or extract key points from an agreement, and talk in probabilities rather than asserting things I don't see a use for these. And I don't know if we will be able to 100% trust these tools one day.
- DoreenMichele 3y ago
  We haven't figured out how to deal with human intelligence effectively. It's jumping the gun to imagine we can do AI well. /2 cents -- as if I need a "disclaimer" but the internet being what it is....
  - mdp2021 3y ago
    And yet Columbus made the egg sit firmly on the table. The "«we»" that includes the lawyer in question has so little to do with the "«we»" that may include Richard Feynman or Marvin Minsky (and further in the right directions), that you may want to refine your model past the problem of Porphyry and hyper-realism in the universals: that "«we»" is insubstantial. It is not some collective humanity that arrived to non-Euclidean geometry, but a small number of people that had the skills and dealt with the problem intensively.
    - deleted 3y ago
      [deleted]
    - DoreenMichele 3y ago
      I have absolutely no idea at all what you are saying to me. For clarification of some kind: I homeschooled my twice exceptional --"gifted and (learning) disabled" -- sons and for a time did pro bono professional work for a gifted organization in support of me being the teacher in a private school with two students for some years under California law and designing the custom curriculum for my special-needs sons so as to make it digestible without talking down to them and accommodating their big strengths and big weaknesses at the same time. My emails were seemingly "popular" for a time on a gifted homeschoolers list and you can blame their enthusiasm for my writing for my habit of blogging to this day, though I have no traction and it makes damn little money and is no doubt basically me wasting my life. Fortunately, we are all dust in the wind anyway, so I'm sure none of it really matters, but I have Opinions about the poor job we generally do with dealing with other humans, especially short ones under the legal age of majority. So maybe that gives you some context for my stupid comment. No, I don't care to talk further about any of this really, tbh.
      - mdp2021 3y ago
        So I understand you experienced difficulties in dealing with developing intelligences, and you entailed that it should be difficult to create artificial ones. But nobody said it was easy (also because ideas like "to invent the lightbulb is not easy" in that form make little sense), and yet the "right" people - not "«we»" - will probably reach the goal, partially owing to a piece of the "«we»" and largely in spite of a large piece of the "«we»". Just like some achieved the lightbulb, and some (Columbus) achieved making an egg stand straight on the table while those nearby claimed it was impossible. Now, the problem in the original context was not about those people able to create a hammer, but about the people unable to use it. The problem here is not about creating intelligence but using it.
- pjc50 3y ago
  > What about all the hallucinations, false positives, etc.? There's a very small disclaimer at the bottom that says all those things are your problem.
Lio 3y ago
"That is clearly wrong and not something I would knowingly do," Such a bullshit excuse. If you are using ChatGPT to do the donkey work for you then you are knowingly using it to cut corners. It's like turning up in court and saying "well I asked this postman Cliff at the bar I go to his opinion on the case...". She should loose here licence for pulling a stunt like that.
- ben_w 3y ago
  It's sufficiently good that it fools people into thinking its not even corner-cutting. 85% of the time, the source code it gives me is correct (50% of the time it's mediocre, but still correct), and that's true even when it's surprised me by spewing out an entire single-file web app rather than just disparate functions I was expecting to have to copy-paste into a single file[0]; how good or bad varies wildly in a way that suggests behind the scenes changes like all software, but it's enough that I can easily believe someone sees it on a good day, is convinced it's always that good, and doesn't keep checking. [0] I'm not a web developer, don't bother to tell me there's a better way — my career path (for now) is iOS, where I use the LLM very differently anyway.
- DoreenMichele 3y ago
  Law isn't my strong point. I have had two Law classes -- environmental law -- and made A minuses in spite of bothering to actually study. So this isn't really an area I feel genuinely qualified to suggest a course of action. But I don't think I have ever used ChatGPT and was blissfully ignorant for some months as to what that hodgepodge of letters even stood for and I'm inclined to be generally appalled by the whole thing. And I'm sure people simultaneously don't actually care about my opinion but will nonetheless not hesitate to hand me my head for saying anything. But if you are more qualified than me and feel strongly about this, perhaps you should do something about it -- something more than being cranky about it on the internet.
quantified 3y ago
Assuming you believe the lawyer. The fact that ChatGPT hallucinates citations has been very well publicized and this lawyer would need to have been under the proverbial rock to not know this. My money's on their simply not expecting to get caught.
- DoreenMichele 3y ago
  I wasn't actually suggesting that I believe the lawyer, fwiw. Just that if you use ChatGPT, "caveat emptor" so to speak.
justinclift 3y ago
Directly related to this, there is the recently submitted project named like a famous award: https://news.ycombinator.com/item?id=39514192 https://news.ycombinator.com/item?id=39514192

bell-cot 3y ago

The judge believed the offending lawyer's plea of ignorant incompetence, and declined the other side's request for extra penalties for reprehensible conduct and abuse of process. But still ordered the offender to review all her previous work submitted to the court, to identify any previous submissions of fake cases... So a complete train wreck, that could re-open a bunch of settled cases. Or the offender and the "justice" system might quietly "decide" that they don't want to work hard at making themselves look worse, and just leave a bunch of wrongly-settled cases lie. IANAL, but it seems long past time for judges to start pretending that they actually care about the "truth and justice" stuff, and permabanning lawyers who pull this kind of crap from the practice of law.

mdp2021 3y ago
> it seems long past time Before that (yours is just an instance), we need a renewed discipline of individual assessment. Experience is much too full of "...How could you?!"s: it is increasingly important that all effort is spent to have them prevented.

rcbdev 3y ago

Despite what some in the tech space might think, I'm not sure I can lay the blame 100 % on the lawyer. It's true that this is definitely malpractice and they should have known better, it's also true that big tech for years overplayed their hand with the marketing of "AI" to laypeople and management, which has created completely unreasonable expectations. As long as OpenAI has a profit motive to make people believe ChatGPT is "right" more often than not they will never blatantly advertise that everything their product generates might be unverifiable garbage, especially in a way everyone will understand and that isn't just a liability footnote.

bigstrat2003 3y ago
Right. I think it's fair to ask whether a reasonable, non-technical person would even know that ChatGPT will hallucinate answers. That is widely known in the tech world (and even still you have people relying on it for work, which blows my mind), but it takes time for that knowledge to get out to the general public. That goes double when vendors are putting out advertising which misleads people (as you pointed out).
- mdp2021 3y ago
  The last time I tried a LLM based chatbot, for testing purposes, was yesterday, and I had fully clear signals of dealing with a moron. First you have a matter of due diligence when using a tool - especially if you are past the age of twelve, especially if you are a professional -, due diligence that will inform you of the boundaries, but then you also have to have clean senses and be able to see things for what they are as they patently show to be such. It is like having empowered masses of people that in real "darwinian", selective contexts would die before being impactful.
quantified 3y ago
The legal peril for using ChatGPT has been well publicized already.
xxs 3y ago
so you are saying the lawyer has been conned? I'd not think twice about her not cross checking her sources.

happymellon 3y ago

This appears to be the same case, as reported a month ago. https://news.ycombinator.com/item?id=39115148 https://news.ycombinator.com/item?id=39115148

ifdefdebug 3y ago

They have to realize that the language model did it's job just fine. It produced a convincing piece of well-formed language that fits the context window. If they wanted knowledge instead, they should have used a knowledge model.

mdp2021 3y ago
> convincing piece of well-formed language that fits the context window I hope you do realize that "artificial conman" is hardly a goal.
- miningape 3y ago
  I hope you do realise this is what it was trained to do, not be correct, not be inventive/creative, but to make text that _sounds_ good.
  - mdp2021 3y ago
    I very well realize it, I and I hope that re-reading again my original comment you will see that I am condemning that specifically.
- graemep 3y ago
  If people are willing to pay for it, then it is.
  - mdp2021 3y ago
    Clearly that is what some seem to think, and it should be also clear that it has sinister sides.
- rsynnott 3y ago
  ... I mean, given how much money is being spent on LLM-based startups at the moment, it certainly _seems_ to be a goal for a lot of people.
pjc50 3y ago

mdp2021 3y ago

From instead https://globalnews.ca/news/10263897/fake-ai-cases-b-c-supreme-court/ https://globalnews.ca/news/10263897/fake-ai-cases-b-c-suprem... > Neither side would comment on Wednesday, but Lorne MacLean previously described the threat AI poses to the legal system as grave No, no, no, no, no, no, no. That people believe whatever is in front of them, and that such people are working as professionals instead of pivoting around a couch, is grave. That active morons are taken as a legitimate state of things, is grave. That due diligence was missed, is grave. And appeals in the category of > “Imagine yourself as a young lawyer … she was mortified” [her] lawyer told the court are grave.

rsynnott 3y ago

This seems to be happening quite a lot. Of course, it's particularly likely to be _caught_ early in law; the law is inherently adversarial, and it's a small enough world that people will be conscious that their opponent is the sort of lazy idiot prone to do this. Where I'd _really_ worry about it is in environments where there's an expectation that the 'expert' is operating in good faith. I'd expect over the next few years we'll see some deaths whose root cause is ultimately "the doctor/engineer believed the magic robot".