4 ms·

The user has asked the LLM to do web request based off untrusted input. The LLM is a completely stateless machine that is only driven by input the user fully c

by 4ad 2y ago

The user has asked the LLM to do web request based off untrusted input. The LLM is a completely stateless machine that is only driven by input the user fully controls. It doesn't do anything on its own. It's like the user running a random .exe from the Internet. Wow much exploit.

InsideOutSanta 2y ago
"The user has asked the LLM to do web request based off untrusted input." I'm not sure if you're talking about the initial attack vector that plants the attack in the LLM's persistent memory, or if you're talking about subsequent interactions with the LLM. The initial attack vector may be a web request the LLM does as a result of the user's prompt, but it does not necessarily have to be. It could also be the user asking the LLM to summarize last week's email, for example. Subsequent interactions with the LLM will then make the request regardless of what the user actually requests the LLM to do. "The LLM is a completely stateless machine" In this case, the problem is that the LLM is not stateless. It has a persistent memory.
- 4ad 2y ago
  LLMs do not have persistent memory. OpenAI does. Persistent memory is nothing magic, it's just LLM context that you, the user, decided to automate its creation to an LLM that can consume 3rd party input. (In fact it's precisely because LLMs are stateless why you could implement persistent memory yourself, completely client side, you don't need any support from OpenAI to do this.) If you have decided to give a 3rd party control over your LLM context, that's on you. Of course the 3rd party has as much control over the LLM as you do. It's literally the same thing as running a random .exe from the internet. Of course this can be useful, the .exe could provide a useful function, alternatively it could also steal your data. But you chose to run the .exe. Similarly automating your LLM context generation can be useful, but with exactly the same caveats, whoever influences your LLM context controls the LLM. If you enable persistent memory you give them this control.
  - InsideOutSanta 2y ago
    "LLMs do not have persistent memory. OpenAI does" The LLM we are discussing here does have persistent memory, because OpenAI gave it persistent memory. "It's literally the same thing as running a random .exe from the internet" I'm not sure what the point is you're making with that, since downloading a random .exe from the Internet is clearly a security issue. By your own analogy, this is also a security issue. The difference is that OpenAI is doing it for you, you're just using OpenAI's program in the way it was intended to be used.
  - mrtranscendence 2y ago
    Act as high and mighty as you please, I won't mind. But bear in mind that: * most people will find it surprising that showing a photo from the internet to ChatGPT is as unsafe as opening a random, untrusted exe. * many people don't even understand that it's unsafe to open random, untrusted exes. Are you seriously suggesting that we should leave all these people to the wolves, because they're less knowledgeable about security vulnerabilities than you?
ceejayoz 2y ago
> It's like the user running a random .exe from the Internet. Wow much exploit. Which users do incessantly, necessitating an entire security infrastructure to combat it.

by 4ad 2y ago

InsideOutSanta 2y ago

"The user has asked the LLM to do web request based off untrusted input." I'm not sure if you're talking about the initial attack vector that plants the attack in the LLM's persistent memory, or if you're talking about subsequent interactions with the LLM. The initial attack vector may be a web request the LLM does as a result of the user's prompt, but it does not necessarily have to be. It could also be the user asking the LLM to summarize last week's email, for example. Subsequent interactions with the LLM will then make the request regardless of what the user actually requests the LLM to do. "The LLM is a completely stateless machine" In this case, the problem is that the LLM is not stateless. It has a persistent memory.

4ad 2y ago
LLMs do not have persistent memory. OpenAI does. Persistent memory is nothing magic, it's just LLM context that you, the user, decided to automate its creation to an LLM that can consume 3rd party input. (In fact it's precisely because LLMs are stateless why you could implement persistent memory yourself, completely client side, you don't need any support from OpenAI to do this.) If you have decided to give a 3rd party control over your LLM context, that's on you. Of course the 3rd party has as much control over the LLM as you do. It's literally the same thing as running a random .exe from the internet. Of course this can be useful, the .exe could provide a useful function, alternatively it could also steal your data. But you chose to run the .exe. Similarly automating your LLM context generation can be useful, but with exactly the same caveats, whoever influences your LLM context controls the LLM. If you enable persistent memory you give them this control.
- InsideOutSanta 2y ago
  "LLMs do not have persistent memory. OpenAI does" The LLM we are discussing here does have persistent memory, because OpenAI gave it persistent memory. "It's literally the same thing as running a random .exe from the internet" I'm not sure what the point is you're making with that, since downloading a random .exe from the Internet is clearly a security issue. By your own analogy, this is also a security issue. The difference is that OpenAI is doing it for you, you're just using OpenAI's program in the way it was intended to be used.
- mrtranscendence 2y ago
  Act as high and mighty as you please, I won't mind. But bear in mind that: * most people will find it surprising that showing a photo from the internet to ChatGPT is as unsafe as opening a random, untrusted exe. * many people don't even understand that it's unsafe to open random, untrusted exes. Are you seriously suggesting that we should leave all these people to the wolves, because they're less knowledgeable about security vulnerabilities than you?

ceejayoz 2y ago

> It's like the user running a random .exe from the Internet. Wow much exploit. Which users do incessantly, necessitating an entire security infrastructure to combat it.