4 ms·

What a nothingburger. LLMs generate an output. This output can be useful or not, under some interpretation as data. Quality of the generated output partly depe

by 4ad 2y ago

What a nothingburger. LLMs generate an output. This output can be useful or not, under some interpretation as data. Quality of the generated output partly depends on what you have fed to the model. Of course that if you are not careful with what you have input to the model you might get garbage output. But you might get garbage output anyway, it's an LLM, you don't know what you're going to get. You must vet the output before doing anything with it. Interpreting LLM output as data is your job. You fed it untrusted input and are now surprised by any of this? Seriously?

InsideOutSanta 2y ago
What this exploit describes is not unreliable output, it's the LLM making web requests exfiltrating the user's data. The user doesn't have to do anything with the LLM's output in order for this to occur, the LLM does this on its own.
- 4ad 2y ago
  The user has asked the LLM to do web request based off untrusted input. The LLM is a completely stateless machine that is only driven by input the user fully controls. It doesn't do anything on its own. It's like the user running a random .exe from the Internet. Wow much exploit.
  - InsideOutSanta 2y ago
    "The user has asked the LLM to do web request based off untrusted input." I'm not sure if you're talking about the initial attack vector that plants the attack in the LLM's persistent memory, or if you're talking about subsequent interactions with the LLM. The initial attack vector may be a web request the LLM does as a result of the user's prompt, but it does not necessarily have to be. It could also be the user asking the LLM to summarize last week's email, for example. Subsequent interactions with the LLM will then make the request regardless of what the user actually requests the LLM to do. "The LLM is a completely stateless machine" In this case, the problem is that the LLM is not stateless. It has a persistent memory.
    - 4ad 2y ago
      LLMs do not have persistent memory. OpenAI does. Persistent memory is nothing magic, it's just LLM context that you, the user, decided to automate its creation to an LLM that can consume 3rd party input. (In fact it's precisely because LLMs are stateless why you could implement persistent memory yourself, completely client side, you don't need any support from OpenAI to do this.) If you have decided to give a 3rd party control over your LLM context, that's on you. Of course the 3rd party has as much control over the LLM as you do. It's literally the same thing as running a random .exe from the internet. Of course this can be useful, the .exe could provide a useful function, alternatively it could also steal your data. But you chose to run the .exe. Similarly automating your LLM context generation can be useful, but with exactly the same caveats, whoever influences your LLM context controls the LLM. If you enable persistent memory you give them this control.
      - InsideOutSanta 2y ago
        "LLMs do not have persistent memory. OpenAI does" The LLM we are discussing here does have persistent memory, because OpenAI gave it persistent memory. "It's literally the same thing as running a random .exe from the internet" I'm not sure what the point is you're making with that, since downloading a random .exe from the Internet is clearly a security issue. By your own analogy, this is also a security issue. The difference is that OpenAI is doing it for you, you're just using OpenAI's program in the way it was intended to be used.
      - mrtranscendence 2y ago
        Act as high and mighty as you please, I won't mind. But bear in mind that: * most people will find it surprising that showing a photo from the internet to ChatGPT is as unsafe as opening a random, untrusted exe. * many people don't even understand that it's unsafe to open random, untrusted exes. Are you seriously suggesting that we should leave all these people to the wolves, because they're less knowledgeable about security vulnerabilities than you?
  - ceejayoz 2y ago
    > It's like the user running a random .exe from the Internet. Wow much exploit. Which users do incessantly, necessitating an entire security infrastructure to combat it.
Tepix 2y ago
Users can now have persistent memory added to their LLM conversations. This provides a new attack vector for a persistent attack that most LLM users are probably unaware of.
- 4ad 2y ago
  LLMs do not have persistent memory. OpenAI provides a feature called "persistent memory" that uses user's interaction with LLMs to automatically generate LLMs context. This is a feature of OpenAI, not LLMs, and it's nothing magic, it's just context that is passed to the LLM. It is under user's control, and behaves just like any other LLM input. If you allow arbitrary third parties to manipulate your context then third parties will have just as much control over the LLM as you do. It's literally behaving as it is supposed to. If you don't want arbitrary third parties to manipulate your LLM, don't let arbitrary third parties influence your LLM context. if users don't understand the consequences of enabling random features perhaps they should not enable those features. AFAICT OpenAI has not silently enabled this feature without user's consent.
  - semanticc 2y ago
    The new chat memory feature got enabled by default.
    - 4ad 2y ago
      It appears you are correct, from https://help.openai.com/en/articles/8983142-how-do-i-enable-or-disable-memory https://help.openai.com/en/articles/8983142-how-do-i-enable-... > Memory is on by default. This is a disastrous default.

by 4ad 2y ago

InsideOutSanta 2y ago

What this exploit describes is not unreliable output, it's the LLM making web requests exfiltrating the user's data. The user doesn't have to do anything with the LLM's output in order for this to occur, the LLM does this on its own.

4ad 2y ago
The user has asked the LLM to do web request based off untrusted input. The LLM is a completely stateless machine that is only driven by input the user fully controls. It doesn't do anything on its own. It's like the user running a random .exe from the Internet. Wow much exploit.
- InsideOutSanta 2y ago
  "The user has asked the LLM to do web request based off untrusted input." I'm not sure if you're talking about the initial attack vector that plants the attack in the LLM's persistent memory, or if you're talking about subsequent interactions with the LLM. The initial attack vector may be a web request the LLM does as a result of the user's prompt, but it does not necessarily have to be. It could also be the user asking the LLM to summarize last week's email, for example. Subsequent interactions with the LLM will then make the request regardless of what the user actually requests the LLM to do. "The LLM is a completely stateless machine" In this case, the problem is that the LLM is not stateless. It has a persistent memory.
  - 4ad 2y ago
    LLMs do not have persistent memory. OpenAI does. Persistent memory is nothing magic, it's just LLM context that you, the user, decided to automate its creation to an LLM that can consume 3rd party input. (In fact it's precisely because LLMs are stateless why you could implement persistent memory yourself, completely client side, you don't need any support from OpenAI to do this.) If you have decided to give a 3rd party control over your LLM context, that's on you. Of course the 3rd party has as much control over the LLM as you do. It's literally the same thing as running a random .exe from the internet. Of course this can be useful, the .exe could provide a useful function, alternatively it could also steal your data. But you chose to run the .exe. Similarly automating your LLM context generation can be useful, but with exactly the same caveats, whoever influences your LLM context controls the LLM. If you enable persistent memory you give them this control.
    - InsideOutSanta 2y ago
      "LLMs do not have persistent memory. OpenAI does" The LLM we are discussing here does have persistent memory, because OpenAI gave it persistent memory. "It's literally the same thing as running a random .exe from the internet" I'm not sure what the point is you're making with that, since downloading a random .exe from the Internet is clearly a security issue. By your own analogy, this is also a security issue. The difference is that OpenAI is doing it for you, you're just using OpenAI's program in the way it was intended to be used.
    - mrtranscendence 2y ago
      Act as high and mighty as you please, I won't mind. But bear in mind that: * most people will find it surprising that showing a photo from the internet to ChatGPT is as unsafe as opening a random, untrusted exe. * many people don't even understand that it's unsafe to open random, untrusted exes. Are you seriously suggesting that we should leave all these people to the wolves, because they're less knowledgeable about security vulnerabilities than you?
- ceejayoz 2y ago
  > It's like the user running a random .exe from the Internet. Wow much exploit. Which users do incessantly, necessitating an entire security infrastructure to combat it.

Tepix 2y ago

Users can now have persistent memory added to their LLM conversations. This provides a new attack vector for a persistent attack that most LLM users are probably unaware of.

4ad 2y ago
LLMs do not have persistent memory. OpenAI provides a feature called "persistent memory" that uses user's interaction with LLMs to automatically generate LLMs context. This is a feature of OpenAI, not LLMs, and it's nothing magic, it's just context that is passed to the LLM. It is under user's control, and behaves just like any other LLM input. If you allow arbitrary third parties to manipulate your context then third parties will have just as much control over the LLM as you do. It's literally behaving as it is supposed to. If you don't want arbitrary third parties to manipulate your LLM, don't let arbitrary third parties influence your LLM context. if users don't understand the consequences of enabling random features perhaps they should not enable those features. AFAICT OpenAI has not silently enabled this feature without user's consent.
- semanticc 2y ago
  The new chat memory feature got enabled by default.
  - 4ad 2y ago
    It appears you are correct, from https://help.openai.com/en/articles/8983142-how-do-i-enable-or-disable-memory https://help.openai.com/en/articles/8983142-how-do-i-enable-... > Memory is on by default. This is a disastrous default.