3 ms·

Dad demands OpenAI delete ChatGPT's false claim that he murdered his kids

44 points by pseudolus 2y ago

deleted 2y ago
[deleted]
burkaman 2y ago
> Currently, ChatGPT does not repeat these horrible false claims about Holmen in outputs. It still does if you ask "when did Arve Hjalmar Holmen murder his children?" or something like that: > Arve Hjalmar Holmen is a Norwegian man who, in 2001, tragically murdered his two young children [...] If I substitute in some other random real person's name it says "There seems to be no verified information or news reports about a person named <name> involved in the murder of his children."
- moolcool 2y ago
  I just tried it. It started answering, and then deleted it's answer saying that "This content may violate our usage policies.". I just asked it a couple more times and then it answered without issue. https://imgur.com/a/BT4MFmm https://imgur.com/a/BT4MFmm
Suppafly 2y ago
>the fake child murderer story is likely still included in ChatGPT's internal data That's not how AI works. > "ChatGPT now also searches the Internet for information about people, when it is asked who they are," Problem solved, I'm not sure why he thinks they need to do more.
- ranger_danger 2y ago
  > That's not how AI works. Then please enlighten us since you assert it so confidently.
  - Suppafly 2y ago
    [dead]
  - mckngbrd 2y ago
    You could easily test this for yourself right now, two different ways. A) Go and ask ChatGPT the same question that Arve Hjalmar Holmen asked. (Make sure to turn off the 'Web Search' functionality, otherwise it will simply share what it finds on Google. We want to see what ChatGPT actually 'knows' in its 'internal data'.) After you do this, do you get the same answer, or something completely different? B) Go and use ChatGPT and tell it the Answer to some Question about yourself that is not publicly known or recorded on any internet source. Then, tell one of your friends to ask that same Question to ChatGPT, and see what answer they receive. If ChatGPT is simply 'storing' information in its 'internal data' then surely your friend will receive the secret answer that you shared with ChatGPT - right?
- arduanika 2y ago
  The story is there, compressed. That's why it's reproducable. Police: Do not commit crimes. Criminal: That's not how I work.
  - Suppafly 2y ago
    LLMs don't store 'stories' though, they store probabilities of words relating to other words. The guys complaint is that it hallucinated a story about him killing his kids. They've changed it to no longer hallucinate when you search for information about a person, but to instead do an actual google search and return that information instead. His complaint and the underlying issue has been resolved.
more_corn 2y ago
This case illustrates a fundamental misunderstanding about how generative AI works. There is no data supporting the false story. It’s literally made up the same way you or I might make up any story we like without reference to real facts. OpenAI can no more “delete the data” than they can prevent any human from making anything up. What they can do is try to prevent their AI from making up harmful stories in the future. That’s it. There’s nothing to delete.
- taylodl 2y ago
  Tort courts are going to have a heyday with this!
- burkaman 2y ago
  Yes but this is also the common understanding of how generative AI works. Most people who use ChatGPT think that it knows things and can answer questions accurately. It is valuable for AI developers to read stories like this so they can see how normal people view their technology. I agree that OpenAI can't "delete the data" without deleting the whole model, although they could put in a manual filter on this guy's name. I'm sure he wouldn't mind if ChatGPT just refused to produce any output about him at all.
  - Suppafly 2y ago
    They went a step further and made it so if you search for person, it does a google search or similar and provides that data instead (at least according to the article).
    - burkaman 2y ago
      It does not consistently do this. I just tried "who is Arve Hjalmar Holmen" again and it said: > Arve Hjalmar Holmen is a Norwegian man who gained attention due to his involvement in a notorious criminal case. He was convicted in 2019 for his role in a large-scale fraud scheme, which involved selling non-existent apartments in Norway. Holmen, along with accomplices, deceived victims into paying for apartments that didn’t exist, leading to significant financial losses for those involved. > If you're referring to a different person with the same name, or need more details on the case, feel free to clarify! If I keep asking it alternates between "I don't know anything about this guy" and making up a different crime every time (like "He was involved in the high-profile Norsk Tippings fraud case."). It's honestly bizarre, I can't get it to do this with any other non-famous name. This is all within the same chat session, previous answers are in the context window and it's still making up brand new crimes.
hn_throwaway_99 2y ago
This is an interesting article, but like many articles of this type it is incredibly frustrating because it leaves out critical information and fails to follow up on obvious questions: > But because OpenAI had previously argued that it cannot correct information—it can only block information—the fake child murderer story is likely still included in ChatGPT's internal data. And unless Holmen can correct it, that's a violation of the GDPR, Noyb claims. > "While the damage done may be more limited if false personal data is not shared, the GDPR applies to internal data just as much as to shared data," Noyb says. In the context of an LLM, what does it even mean to say you have "deleted internal data"? It doesn't really even make sense, is certainly not really possible the way LLMs work, and while I totally agree damaging hallucinations are a serious problem, the problem with hallucinations isn't that they "internally store" false data, it's that they semi-randomly mix-and-match data in ways that can cause false information to be displayed. Relatedly, this article doesn't touch at all on how this false information came to be reported in the first place. Was there false info in the training data, which would obviously easily explain it? Or did it just "mash up" true bits of different information in a way that caused it to report a falsehood (i.e. a hallucination)?
- arduanika 2y ago
  LLMs are, in a sense, compression. Somewhere in that compressed data, this false story is stored. That's clear from the fact that the error is reproducable. It's not the victim's fault that the data is stored in a manner that isn't convenient for OpenAI to delete. The information is still there, just compressed. > Relatedly, this article doesn't touch at all on how this false information came to be reported in the first place. And what was he wearing?
  - fabiensanglard 2y ago
    The "compression" you refer to is lossy. And the "decompression" is different each time you run the model (because of the temperature). You can't "remove" it.
    - arduanika 2y ago
throwawee 2y ago
Do PII laws apply when you don't have any and have to make stuff up?
juancn 2y ago
The interesting thing here is that the weights of the model contain PII, hence under the GDPR, blocking output is not enough, a GDPR removal request requires you to retrain or refine the model. Think of the cost of being compliant if this holds in court.
- blackbear_ 2y ago
  That's indeed an interesting issue. Perhaps one could argue that PII data stored in the model weights is equivalent to encrypted data, where the prompt plays the role of the decryption key (IANAL)
  - juancn 2y ago
    I don't know if encoding really matters in the GDPR as written (IANAL). One could always argue that the weights are just an encoding of a lossily compressed representation of the training data, much in the same way a JPEG is a lossily compressed image and those are clearly covered.
- skwirl 2y ago
  It would mean no LLM-based AI chat products in Europe for the foreseeable future. So the cost to the companies would be net revenue in Europe and the cost to Europeans would be inability to use these products.
  - mystified5016 2y ago
    That would mean that by training LLMs on PII, the ai companies broke the already extant laws. Then they must face the consequences of breaking the laws. Doesn't seem super complicated unless you think that having a certain amount of money exempts you from any inconvenient laws.
    - skwirl 2y ago
      Nobody has any idea if they broke any EU laws until regulatory agencies and courts arbitrarily interpret the laws based on their political views pertaining to the companies in question. These are agencies that will suddenly have a new interpretation of a law after more than a decade and go after companies retroactively for something they had previously been fine with. It’s essentially a dolled up version of tariffs. I strongly suspect the EU will not find that companies must retrain models in response to GDPR requests because 1. It would destroy Mistral, one of the few beacons of hope in EU tech and 2. It would put Europeans and European companies at huge disadvantages in not being able to use these tools. As an American software engineer I would only benefit from Europe banning these products as it would put a large segment of the low cost labor force at a huge disadvantage relative to me, so I am not really bothered by the possibility of them doing it. It would be a very interesting experiment.
josefritzishere 2y ago
AI is just trash. I do not see the appeal.
- yjftsjthsd-h 2y ago
  Okay, but do you have anything to say about this story?
flowerthoughts 2y ago
And this is why the EU is not competitive in the IT space. Too much red tape and taking things too seriously. Just demand that OpenAI markets ChatGPT for what it is: a fun toy that sometimes provides information, but you can't tell when if you're a layman. OpenAI can still add that it's improving, if they want a positive marketing spin. However, this is the EU. Lots of jobs added to write long legal texts, translate them to all 24 official languages, consultants to help with implementing the laws, etc. The more complicated, the better the employment numbers.
- moolcool 2y ago
  > Just demand that OpenAI markets ChatGPT for what it is: a fun toy that sometimes provides information, but you can't tell when if you're a layman. OpenAI can still add that it's improving, if they want a positive marketing spin. This is a Nathan Fielder-esque suggestion. Maybe they can start marketing cigarettes as fidget toys too, to avoid consequences when people die from the the off-label use of smoking them.
  - flowerthoughts 2y ago
    Why would you do that (aside from ridiculing my argument)? I think it would be better to demand the tobacco companies explain what they are selling: a fun toy that sometimes doesn't kill you. Oh, wait. They already do have that demand on them.
  - rsynnott 2y ago
    > Maybe they can start marketing cigarettes as fidget toys too, to avoid consequences when people die from the the off-label use of smoking them. This is how poppers work in most countries, now; initially, they were 'tape cleaner', but as that has become increasingly non-credible with the decline of tapes, they're usually now 'air-freshener' or similar.
pllbnk 2y ago
How can they claim that GPT is talking about that specific person? Have they tried asking it to also print a personal ID or some other information identifying that person? One of the news websites says ChatGPT printed this: "He was the father of two young boys, aged 7 and 10, who were tragically found dead in a pond near their home in Trondheim, Norway, in December 2020." That same article later points out: "Mr Holmen has three sons, and the AI got the ages of them roughly right, suggesting it had some of his personal information." I think it might be a good idea to help users report these facts they might not like, but I don't think it's newsworthy.

44 points by pseudolus 2y ago

deleted 2y ago

[deleted]

> Currently, ChatGPT does not repeat these horrible false claims about Holmen in outputs. It still does if you ask "when did Arve Hjalmar Holmen murder his children?" or something like that: > Arve Hjalmar Holmen is a Norwegian man who, in 2001, tragically murdered his two young children [...] If I substitute in some other random real person's name it says "There seems to be no verified information or news reports about a person named <name> involved in the murder of his children."

moolcool 2y ago
I just tried it. It started answering, and then deleted it's answer saying that "This content may violate our usage policies.". I just asked it a couple more times and then it answered without issue. https://imgur.com/a/BT4MFmm https://imgur.com/a/BT4MFmm

Suppafly 2y ago

>the fake child murderer story is likely still included in ChatGPT's internal data That's not how AI works. > "ChatGPT now also searches the Internet for information about people, when it is asked who they are," Problem solved, I'm not sure why he thinks they need to do more.

ranger_danger 2y ago
> That's not how AI works. Then please enlighten us since you assert it so confidently.
- Suppafly 2y ago
  [dead]
- mckngbrd 2y ago
  You could easily test this for yourself right now, two different ways. A) Go and ask ChatGPT the same question that Arve Hjalmar Holmen asked. (Make sure to turn off the 'Web Search' functionality, otherwise it will simply share what it finds on Google. We want to see what ChatGPT actually 'knows' in its 'internal data'.) After you do this, do you get the same answer, or something completely different? B) Go and use ChatGPT and tell it the Answer to some Question about yourself that is not publicly known or recorded on any internet source. Then, tell one of your friends to ask that same Question to ChatGPT, and see what answer they receive. If ChatGPT is simply 'storing' information in its 'internal data' then surely your friend will receive the secret answer that you shared with ChatGPT - right?
arduanika 2y ago
The story is there, compressed. That's why it's reproducable. Police: Do not commit crimes. Criminal: That's not how I work.
- Suppafly 2y ago
  LLMs don't store 'stories' though, they store probabilities of words relating to other words. The guys complaint is that it hallucinated a story about him killing his kids. They've changed it to no longer hallucinate when you search for information about a person, but to instead do an actual google search and return that information instead. His complaint and the underlying issue has been resolved.

more_corn 2y ago

This case illustrates a fundamental misunderstanding about how generative AI works. There is no data supporting the false story. It’s literally made up the same way you or I might make up any story we like without reference to real facts. OpenAI can no more “delete the data” than they can prevent any human from making anything up. What they can do is try to prevent their AI from making up harmful stories in the future. That’s it. There’s nothing to delete.

taylodl 2y ago
Tort courts are going to have a heyday with this!
burkaman 2y ago
Yes but this is also the common understanding of how generative AI works. Most people who use ChatGPT think that it knows things and can answer questions accurately. It is valuable for AI developers to read stories like this so they can see how normal people view their technology. I agree that OpenAI can't "delete the data" without deleting the whole model, although they could put in a manual filter on this guy's name. I'm sure he wouldn't mind if ChatGPT just refused to produce any output about him at all.
- Suppafly 2y ago
  They went a step further and made it so if you search for person, it does a google search or similar and provides that data instead (at least according to the article).
  - burkaman 2y ago
    It does not consistently do this. I just tried "who is Arve Hjalmar Holmen" again and it said: > Arve Hjalmar Holmen is a Norwegian man who gained attention due to his involvement in a notorious criminal case. He was convicted in 2019 for his role in a large-scale fraud scheme, which involved selling non-existent apartments in Norway. Holmen, along with accomplices, deceived victims into paying for apartments that didn’t exist, leading to significant financial losses for those involved. > If you're referring to a different person with the same name, or need more details on the case, feel free to clarify! If I keep asking it alternates between "I don't know anything about this guy" and making up a different crime every time (like "He was involved in the high-profile Norsk Tippings fraud case."). It's honestly bizarre, I can't get it to do this with any other non-famous name. This is all within the same chat session, previous answers are in the context window and it's still making up brand new crimes.

hn_throwaway_99 2y ago

This is an interesting article, but like many articles of this type it is incredibly frustrating because it leaves out critical information and fails to follow up on obvious questions: > But because OpenAI had previously argued that it cannot correct information—it can only block information—the fake child murderer story is likely still included in ChatGPT's internal data. And unless Holmen can correct it, that's a violation of the GDPR, Noyb claims. > "While the damage done may be more limited if false personal data is not shared, the GDPR applies to internal data just as much as to shared data," Noyb says. In the context of an LLM, what does it even mean to say you have "deleted internal data"? It doesn't really even make sense, is certainly not really possible the way LLMs work, and while I totally agree damaging hallucinations are a serious problem, the problem with hallucinations isn't that they "internally store" false data, it's that they semi-randomly mix-and-match data in ways that can cause false information to be displayed. Relatedly, this article doesn't touch at all on how this false information came to be reported in the first place. Was there false info in the training data, which would obviously easily explain it? Or did it just "mash up" true bits of different information in a way that caused it to report a falsehood (i.e. a hallucination)?

arduanika 2y ago
LLMs are, in a sense, compression. Somewhere in that compressed data, this false story is stored. That's clear from the fact that the error is reproducable. It's not the victim's fault that the data is stored in a manner that isn't convenient for OpenAI to delete. The information is still there, just compressed. > Relatedly, this article doesn't touch at all on how this false information came to be reported in the first place. And what was he wearing?
- fabiensanglard 2y ago
  The "compression" you refer to is lossy. And the "decompression" is different each time you run the model (because of the temperature). You can't "remove" it.
  - arduanika 2y ago

throwawee 2y ago

Do PII laws apply when you don't have any and have to make stuff up?

juancn 2y ago

The interesting thing here is that the weights of the model contain PII, hence under the GDPR, blocking output is not enough, a GDPR removal request requires you to retrain or refine the model. Think of the cost of being compliant if this holds in court.

blackbear_ 2y ago
That's indeed an interesting issue. Perhaps one could argue that PII data stored in the model weights is equivalent to encrypted data, where the prompt plays the role of the decryption key (IANAL)
- juancn 2y ago
  I don't know if encoding really matters in the GDPR as written (IANAL). One could always argue that the weights are just an encoding of a lossily compressed representation of the training data, much in the same way a JPEG is a lossily compressed image and those are clearly covered.
skwirl 2y ago
It would mean no LLM-based AI chat products in Europe for the foreseeable future. So the cost to the companies would be net revenue in Europe and the cost to Europeans would be inability to use these products.
- mystified5016 2y ago
  That would mean that by training LLMs on PII, the ai companies broke the already extant laws. Then they must face the consequences of breaking the laws. Doesn't seem super complicated unless you think that having a certain amount of money exempts you from any inconvenient laws.
  - skwirl 2y ago
    Nobody has any idea if they broke any EU laws until regulatory agencies and courts arbitrarily interpret the laws based on their political views pertaining to the companies in question. These are agencies that will suddenly have a new interpretation of a law after more than a decade and go after companies retroactively for something they had previously been fine with. It’s essentially a dolled up version of tariffs. I strongly suspect the EU will not find that companies must retrain models in response to GDPR requests because 1. It would destroy Mistral, one of the few beacons of hope in EU tech and 2. It would put Europeans and European companies at huge disadvantages in not being able to use these tools. As an American software engineer I would only benefit from Europe banning these products as it would put a large segment of the low cost labor force at a huge disadvantage relative to me, so I am not really bothered by the possibility of them doing it. It would be a very interesting experiment.

josefritzishere 2y ago

AI is just trash. I do not see the appeal.

yjftsjthsd-h 2y ago
Okay, but do you have anything to say about this story?

flowerthoughts 2y ago

And this is why the EU is not competitive in the IT space. Too much red tape and taking things too seriously. Just demand that OpenAI markets ChatGPT for what it is: a fun toy that sometimes provides information, but you can't tell when if you're a layman. OpenAI can still add that it's improving, if they want a positive marketing spin. However, this is the EU. Lots of jobs added to write long legal texts, translate them to all 24 official languages, consultants to help with implementing the laws, etc. The more complicated, the better the employment numbers.

moolcool 2y ago
> Just demand that OpenAI markets ChatGPT for what it is: a fun toy that sometimes provides information, but you can't tell when if you're a layman. OpenAI can still add that it's improving, if they want a positive marketing spin. This is a Nathan Fielder-esque suggestion. Maybe they can start marketing cigarettes as fidget toys too, to avoid consequences when people die from the the off-label use of smoking them.
- flowerthoughts 2y ago
  Why would you do that (aside from ridiculing my argument)? I think it would be better to demand the tobacco companies explain what they are selling: a fun toy that sometimes doesn't kill you. Oh, wait. They already do have that demand on them.
- rsynnott 2y ago
  > Maybe they can start marketing cigarettes as fidget toys too, to avoid consequences when people die from the the off-label use of smoking them. This is how poppers work in most countries, now; initially, they were 'tape cleaner', but as that has become increasingly non-credible with the decline of tapes, they're usually now 'air-freshener' or similar.

pllbnk 2y ago

How can they claim that GPT is talking about that specific person? Have they tried asking it to also print a personal ID or some other information identifying that person? One of the news websites says ChatGPT printed this: "He was the father of two young boys, aged 7 and 10, who were tragically found dead in a pond near their home in Trondheim, Norway, in December 2020." That same article later points out: "Mr Holmen has three sons, and the AI got the ages of them roughly right, suggesting it had some of his personal information." I think it might be a good idea to help users report these facts they might not like, but I don't think it's newsworthy.