4 ms·

The LLM doesn't have secrets or other users' chats in it. Why would they put that in there?

ummonk 3y ago
If they use batching during inference (which they very probably do), then some kind of coding mistake of the sort that happened with this bug absolutely could result in leakage between chats.
- astrange 3y ago
  One thing that did happen is there was a bug in the website for a day that really did show you other user's chat history. IIRC some reporting confused this with "Samsung had some employees upload internal PDFs to ChatGPT" to produce the claim that ChatGPT was leaking internal Samsung information via training, which it wasn't.
_heimdall 3y ago
LLMs can have secrets if they were scraped in the training data. And how do we know definitively what is done with chat logs? The LLM model is a black box for OpenAI (they don't know what was learned or why it was learned), and OpenAI is a black box for users (we don't know what data they collect or how they use it).
- astrange 3y ago
  You can probe what was learned if you have access to the model; it'll tell you, especially if you do it before applying the safety features. A good heuristic for whether they would train user chats into the model is whether this makes any sense. But it doesn't; it's not valuable. They could be saying anything in there, it's likely private, and it's probably not truthful information. Presumably they do do something with responses you've marked thumbs up/thumbs down to, but there are ways of using those that aren't directly putting them in the training data. After all, that feedback isn't trustworthy either.
  - bpye 3y ago
    > You can probe what was learned if you have access to the model; it'll tell you Could that not also be hallucinated?
    - astrange 3y ago
      By probe, I mean observe the internal activations. There are methods that can suggest if it's hallucinating or not, and ones that can delete individual pieces of knowledge from the model.
  - _heimdall 3y ago
    > You can probe what was learned if you have access to the model; it'll tell you, especially if you do it before applying the safety features. Does that involve actually parsing the data itself, or effectively asking the model questions to see what was learned? If the data model itself can be parsed and analyzed directly by humans that is better than I realized. If its abstracted through an interpreter (I'm sure my terminology is off here) similar to the final GPT product then we still can't really see what was learned.
EchoChamberMan 3y ago
There are historical cases of just such a thing happening. Google warns you not to put important or private information in.
- astrange 3y ago
  That's because they have human evaluators read user prompts to decide where to make quality improvements.
  - EchoChamberMan 3y ago
    I'm pretty sure it was because people were seeing other people's data. Feel free to search for sources of you're curious.
  - EchoChamberMan 3y ago
    https://arstechnica.com/security/2024/01/ars-reader-reports-chatgpt-is-sending-him-conversations-from-unrelated-ai-users/ https://arstechnica.com/security/2024/01/ars-reader-reports-...

If they use batching during inference (which they very probably do), then some kind of coding mistake of the sort that happened with this bug absolutely could result in leakage between chats.

astrange 3y ago
One thing that did happen is there was a bug in the website for a day that really did show you other user's chat history. IIRC some reporting confused this with "Samsung had some employees upload internal PDFs to ChatGPT" to produce the claim that ChatGPT was leaking internal Samsung information via training, which it wasn't.

_heimdall 3y ago

LLMs can have secrets if they were scraped in the training data. And how do we know definitively what is done with chat logs? The LLM model is a black box for OpenAI (they don't know what was learned or why it was learned), and OpenAI is a black box for users (we don't know what data they collect or how they use it).

astrange 3y ago
You can probe what was learned if you have access to the model; it'll tell you, especially if you do it before applying the safety features. A good heuristic for whether they would train user chats into the model is whether this makes any sense. But it doesn't; it's not valuable. They could be saying anything in there, it's likely private, and it's probably not truthful information. Presumably they do do something with responses you've marked thumbs up/thumbs down to, but there are ways of using those that aren't directly putting them in the training data. After all, that feedback isn't trustworthy either.
- bpye 3y ago
  > You can probe what was learned if you have access to the model; it'll tell you Could that not also be hallucinated?
  - astrange 3y ago
    By probe, I mean observe the internal activations. There are methods that can suggest if it's hallucinating or not, and ones that can delete individual pieces of knowledge from the model.
- _heimdall 3y ago
  > You can probe what was learned if you have access to the model; it'll tell you, especially if you do it before applying the safety features. Does that involve actually parsing the data itself, or effectively asking the model questions to see what was learned? If the data model itself can be parsed and analyzed directly by humans that is better than I realized. If its abstracted through an interpreter (I'm sure my terminology is off here) similar to the final GPT product then we still can't really see what was learned.

EchoChamberMan 3y ago

There are historical cases of just such a thing happening. Google warns you not to put important or private information in.

astrange 3y ago
That's because they have human evaluators read user prompts to decide where to make quality improvements.
- EchoChamberMan 3y ago
  I'm pretty sure it was because people were seeing other people's data. Feel free to search for sources of you're curious.
- EchoChamberMan 3y ago
  https://arstechnica.com/security/2024/01/ars-reader-reports-chatgpt-is-sending-him-conversations-from-unrelated-ai-users/ https://arstechnica.com/security/2024/01/ars-reader-reports-...