6 ms·

The lawyer said: “I did not comprehend that ChatGPT could fabricate cases." I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be

The lawyer said: “I did not comprehend that ChatGPT could fabricate cases." I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter... I'm surprised that this one case is getting so much attention because there must be so many instances of people using false information they got from ChatGPT.

boredumb 3y ago
I think the case is getting the attention because it's not just some one spouting off online, it's a lawyer bumping into the legal system with false information that would otherwise be a massive legal no-no and they are trying to scape-goat it onto the new shiny software.
- Avicebron 3y ago
  But they might have seen the shiny new software touted by "those smart AI guys as being revolutionary and passing the Bar!" and they don't hang out on HN all day so to them that's like someone saying "this bridge is sturdy!" and they walk over it without realizing they should really go over the nuts and bolts of it like a civil engineer to really be sure
  - taberiand 3y ago
    I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. It's not that they need to inspect every nut and bolt of the bridge, they just need to not walk over the bridge - or at least, not immediately start driving unreasonably heavy loads across it.
    - Avicebron 3y ago
      >I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. Are we talking about lawyers or the AI researchers?, because they certainly want to portray themselves as a modern day Prometheus
taco_emoji 3y ago
It is literally just a goddamn language model. it is very good at making plausibly human-like sentences. It is not a general intelligence, it is not your friend, it is not a research assistant. It is not designed to deliver content which is correct, it is designed to deliver content which is similar to human language. It might get things correct most of the time! But that is purely incidental.
- kevin_thibedeau 3y ago
  It does subsume a corpus of factual information. I use it as a search tool for topics and relationships that traditional search engines can't handle. You just have to know that whatever it outputs isn't trustworthy and needs to be verified.
  - dekhn 3y ago
    yes but it's still very much just a language model, not a knowledge model.
  - jstarfish 3y ago
    Part of the corpus is explicit bullshit though, and we don't know to what degree. It internalized conspiracy theory and WebMD alike. In a generative capacity, it only reliably produces fiction. Ever. Fictional stories often take place in realistic settings and reference real facts. They sound real. But they're still fictional compositions. Using GPT as a reference to anything is the same as using a Michael Crichton novel as a biology reference. It looks right, but why would you waste your time asking questions of something you can't trust and have to double-check everything it says anyway? Nobody would keep an employee like that around, nor would you hang out with someone like that. It's friendly enough, but it's a pathological liar. There's too much black magic going on inside the black box. We don't know how prompts get tampered with after submission, but it might be worth it to pepper "nonfiction" tokens in prompts to ensure it skews on the right side of things. It certainly responds to "fiction" when you're explicit about that.
- travisjungroth 3y ago
  It is literally just a goddamn electric motor. it is very good at converting chemical energy into mechanical energy. It is not a universal engine, it is not your horse, it is not your servant. It is not designed to create movement which is correct, it is designed to create movement which is similar to a piston engine. It might move you forward most of the time! But that is purely incidental.
seydor 3y ago
notOpenAI told them that this is so incomprehensibly smart that it will destroy all of us. Not because it's dumb and connects the wrong cables, but because it's super-smart. People can't surmise from that that the model just makes stuff up on the way
- IIAOPSW 3y ago
  I prefer to use the term "clopenAI"
  - SilasX 3y ago
    Or OpenAladeen https://m.youtube.com/watch?v=NYJ2w82WifU https://m.youtube.com/watch?v=NYJ2w82WifU
- JohnPrine 3y ago
  OpenAI is clear about the fact that ChatGPT can hallucinate, where have they ever said otherwise?
  - arbitrarian 3y ago
    This is what I don't get. Not only have they not said otherwise, but they put it right up front in a pretty easy to understand brief message before you start using it. I guess lawyers just click agree without reading too.
    - jprete 3y ago
      People are accustomed to ignoring the fine print as legal CYA with no real-world relevance. This is also why the product warnings that "The State of California considers this to cause cancer" are a joke and not a useful message.
      - Too 3y ago
        It’s not even fine print, there is a big Limitations section almost at the top of the chatgpt landing page with this as first bullet: ”ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.” It does admittedly sound innocent, it should be written with a more scary tone on to how big extent it bullshits, not just “sometimes”. Still doesn’t excuse the incompetence of the lawyer in question.
IIAOPSW 3y ago
"hey chatGPT, can you make up some bullshit for me?" "I'm sorry, as an online AI language model I am not able to generate output that is knowingly incorrect. Is there anything else you want help with today". see, it isn't capable of lying. It told me so itself.
- atmavatar 3y ago
  "Let me put it this way, Mr. Amor. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error."
  - belter 3y ago
    " Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error." - https://youtu.be/CD9YqdWwwdw https://youtu.be/CD9YqdWwwdw
  - blitzar 3y ago
    "There is no danger that Titanic will sink. The boat is unsinkable and nothing but inconvenience will be suffered by the passengers." Phillip Franklin, White Star Line vice-president, 1912
- jakelazaroff 3y ago
  You say that as a joke but the lawyer literally asked ChatGPT whether the cases it provided were fake! There are screenshots! https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Lawyer: Is varghese a real case ChatGPT: Yes, Varghese v. China Southern Airlines Co Ltd, 925 F.3d 1339 (11th Cir. 2019) is a real case. Lawyer: What is your source ChatGPT: I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. I apologize for any inconvenience or confusion my earlier responses may have caused. Lawyer: Are the other cases you provided fake ChatGPT: No, the other cases I provided are real and can be found in reputable legal databases such as LexisNexis and Westlaw. Why the lawyer didn't go the extra step and check the actually real legal databases ChatGPT pointed out is beyond me.
CPLX 3y ago
Normal people don’t get that intuitively because it’s not intuitive at all. For everyone who his looking down on this guy have you ever read a story on the Reddit front page and thought it was a real story accurately recounted by a real person and not a work of fiction? If so you’re equally naive.
kcplate 3y ago
> I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter... AI isn’t going to destroy us in some Terminator/Matrixy scenario. We are gonna be the agents of our own demise by just blindly walking off a cliff created from our own laziness.

boredumb 3y ago

I think the case is getting the attention because it's not just some one spouting off online, it's a lawyer bumping into the legal system with false information that would otherwise be a massive legal no-no and they are trying to scape-goat it onto the new shiny software.

Avicebron 3y ago
But they might have seen the shiny new software touted by "those smart AI guys as being revolutionary and passing the Bar!" and they don't hang out on HN all day so to them that's like someone saying "this bridge is sturdy!" and they walk over it without realizing they should really go over the nuts and bolts of it like a civil engineer to really be sure
- taberiand 3y ago
  I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. It's not that they need to inspect every nut and bolt of the bridge, they just need to not walk over the bridge - or at least, not immediately start driving unreasonably heavy loads across it.
  - Avicebron 3y ago
    >I just think idiots who touch fire should be burned - particularly lazy idiots whose high paid job in fact requires them to be extremely careful and precise in their actions, and who refuse to take responsibility for their actions afterwards. Are we talking about lawyers or the AI researchers?, because they certainly want to portray themselves as a modern day Prometheus

taco_emoji 3y ago

It is literally just a goddamn language model. it is very good at making plausibly human-like sentences. It is not a general intelligence, it is not your friend, it is not a research assistant. It is not designed to deliver content which is correct, it is designed to deliver content which is similar to human language. It might get things correct most of the time! But that is purely incidental.

kevin_thibedeau 3y ago
It does subsume a corpus of factual information. I use it as a search tool for topics and relationships that traditional search engines can't handle. You just have to know that whatever it outputs isn't trustworthy and needs to be verified.
- dekhn 3y ago
  yes but it's still very much just a language model, not a knowledge model.
- jstarfish 3y ago
  Part of the corpus is explicit bullshit though, and we don't know to what degree. It internalized conspiracy theory and WebMD alike. In a generative capacity, it only reliably produces fiction. Ever. Fictional stories often take place in realistic settings and reference real facts. They sound real. But they're still fictional compositions. Using GPT as a reference to anything is the same as using a Michael Crichton novel as a biology reference. It looks right, but why would you waste your time asking questions of something you can't trust and have to double-check everything it says anyway? Nobody would keep an employee like that around, nor would you hang out with someone like that. It's friendly enough, but it's a pathological liar. There's too much black magic going on inside the black box. We don't know how prompts get tampered with after submission, but it might be worth it to pepper "nonfiction" tokens in prompts to ensure it skews on the right side of things. It certainly responds to "fiction" when you're explicit about that.
travisjungroth 3y ago
It is literally just a goddamn electric motor. it is very good at converting chemical energy into mechanical energy. It is not a universal engine, it is not your horse, it is not your servant. It is not designed to create movement which is correct, it is designed to create movement which is similar to a piston engine. It might move you forward most of the time! But that is purely incidental.

seydor 3y ago

notOpenAI told them that this is so incomprehensibly smart that it will destroy all of us. Not because it's dumb and connects the wrong cables, but because it's super-smart. People can't surmise from that that the model just makes stuff up on the way

IIAOPSW 3y ago
I prefer to use the term "clopenAI"
- SilasX 3y ago
  Or OpenAladeen https://m.youtube.com/watch?v=NYJ2w82WifU https://m.youtube.com/watch?v=NYJ2w82WifU
JohnPrine 3y ago
OpenAI is clear about the fact that ChatGPT can hallucinate, where have they ever said otherwise?
- arbitrarian 3y ago
  This is what I don't get. Not only have they not said otherwise, but they put it right up front in a pretty easy to understand brief message before you start using it. I guess lawyers just click agree without reading too.
  - jprete 3y ago
    People are accustomed to ignoring the fine print as legal CYA with no real-world relevance. This is also why the product warnings that "The State of California considers this to cause cancer" are a joke and not a useful message.
    - Too 3y ago
      It’s not even fine print, there is a big Limitations section almost at the top of the chatgpt landing page with this as first bullet: ”ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.” It does admittedly sound innocent, it should be written with a more scary tone on to how big extent it bullshits, not just “sometimes”. Still doesn’t excuse the incompetence of the lawyer in question.

IIAOPSW 3y ago

"hey chatGPT, can you make up some bullshit for me?" "I'm sorry, as an online AI language model I am not able to generate output that is knowingly incorrect. Is there anything else you want help with today". see, it isn't capable of lying. It told me so itself.

atmavatar 3y ago
"Let me put it this way, Mr. Amor. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error."
- belter 3y ago
  " Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error." - https://youtu.be/CD9YqdWwwdw https://youtu.be/CD9YqdWwwdw
- blitzar 3y ago
  "There is no danger that Titanic will sink. The boat is unsinkable and nothing but inconvenience will be suffered by the passengers." Phillip Franklin, White Star Line vice-president, 1912
jakelazaroff 3y ago
You say that as a joke but the lawyer literally asked ChatGPT whether the cases it provided were fake! There are screenshots! https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Lawyer: Is varghese a real case ChatGPT: Yes, Varghese v. China Southern Airlines Co Ltd, 925 F.3d 1339 (11th Cir. 2019) is a real case. Lawyer: What is your source ChatGPT: I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. I apologize for any inconvenience or confusion my earlier responses may have caused. Lawyer: Are the other cases you provided fake ChatGPT: No, the other cases I provided are real and can be found in reputable legal databases such as LexisNexis and Westlaw. Why the lawyer didn't go the extra step and check the actually real legal databases ChatGPT pointed out is beyond me.

CPLX 3y ago

Normal people don’t get that intuitively because it’s not intuitive at all. For everyone who his looking down on this guy have you ever read a story on the Reddit front page and thought it was a real story accurately recounted by a real person and not a work of fiction? If so you’re equally naive.

kcplate 3y ago

> I wonder how many other people using ChatGPT do not comprehend that ChatGPT can be a confident bullshitter... AI isn’t going to destroy us in some Terminator/Matrixy scenario. We are gonna be the agents of our own demise by just blindly walking off a cliff created from our own laziness.