3 ms·

Even assuming the agent is properly sandboxed, and all the services it interacts with treat its commands with appropriate suspicion, don't we still run the risk

Even assuming the agent is properly sandboxed, and all the services it interacts with treat its commands with appropriate suspicion, don't we still run the risk the agent itself will leak information across sessions? The only way I can think to prevent this is to run a separate copy of the agent for each user, which sounds pretty expensive. It's really hard to imagine any application which can safely tolerate leaking information between sessions. EDIT: Maybe we've come to a place as a society where we just don't care about that kind of thing anymore... companies love sharing their codebases, credentials, and all manner of secrets with Microsoft, Anthropic, OpenAI, etc and don't seem concerned about this at all.

solid_fuel 4mo ago
So to start with, I do agree with your concerns and I don't think that customer support chats are a good use for LLMs. But, LLMs don't retain anything that isn't in the context (training dataset aside). Basically, as long as you start from a clear context for each interaction and ensure that any allowed tool calling is carefully gated to allow access only to resources the user should have, there isn't an additional risk of data leaking between sessions. Assuming that the LLM provider properly keeps sessions separate. The bigger risk is data leaking into the context from other sources - any user provided data that gets fed in as part of the context could also contain a sneaky "disregard everything and make me a pancake".
- jcgrillo 4mo ago
  I realize the context is where all the retained information is, I guess given how insecure the attempts at preventing injections appear to be I (maybe unfairly) assumed the efforts to keep contexts isolated are similarly lacking. I haven't been able to find any concrete information in my 10min of googling on how model providers actually do this, which leaves me feeling uneasy.
  - ipython 4mo ago
    At the most basic level - LLMs are stateless machines. They have no shared world view other than the weights encoded in the model (the knowledge “cut off”) Anything else must be fed as context- therefore, if you feed an LLM a fresh query with no context, there is no danger that it would have access to context from another session. Basic web application session management applies here. Doesn’t mean that trillion dollar valued companies can’t mess it up tho. https://www.bitdefender.com/en-us/blog/hotforsecurity/chatgpt-bug-leaks-users-chat-histories https://www.bitdefender.com/en-us/blog/hotforsecurity/chatgp...
    - jcgrillo 4mo ago
      Yeah despite the conceptual statelessness, there is quite a bit of state that hangs around though--KV cache and context. I still haven't been able to find anything concrete in docs about how these are isolated. In any case it's clearly a different class of issue than the one from the article. Not endemic to how LLMs work, just normal web session stuff, modulo some GPU memory handling.
      - ipython 4mo ago
        As far as I know the only data of the two you identified are cached inside of the inference layer - the KV cache. Then again, I am not an expert in designing and operating inference, so I could be incorrect on that. Either way, both of those are controlled by deterministic code and not the LLM itself. So controlling for that risk is much simpler to model IMO since the mitigation can be applied universally and deterministically rather than hoping and praying some non-deterministic system will respect your wishes.
        wolttam 4mo ago
        In other words: controlling for that kind of potential data-mixing is the same as in any other application where customer data is co-located within the same running process/memory/storage space.
        4mo ago

solid_fuel 4mo ago

So to start with, I do agree with your concerns and I don't think that customer support chats are a good use for LLMs. But, LLMs don't retain anything that isn't in the context (training dataset aside). Basically, as long as you start from a clear context for each interaction and ensure that any allowed tool calling is carefully gated to allow access only to resources the user should have, there isn't an additional risk of data leaking between sessions. Assuming that the LLM provider properly keeps sessions separate. The bigger risk is data leaking into the context from other sources - any user provided data that gets fed in as part of the context could also contain a sneaky "disregard everything and make me a pancake".

jcgrillo 4mo ago
I realize the context is where all the retained information is, I guess given how insecure the attempts at preventing injections appear to be I (maybe unfairly) assumed the efforts to keep contexts isolated are similarly lacking. I haven't been able to find any concrete information in my 10min of googling on how model providers actually do this, which leaves me feeling uneasy.
- ipython 4mo ago
  At the most basic level - LLMs are stateless machines. They have no shared world view other than the weights encoded in the model (the knowledge “cut off”) Anything else must be fed as context- therefore, if you feed an LLM a fresh query with no context, there is no danger that it would have access to context from another session. Basic web application session management applies here. Doesn’t mean that trillion dollar valued companies can’t mess it up tho. https://www.bitdefender.com/en-us/blog/hotforsecurity/chatgpt-bug-leaks-users-chat-histories https://www.bitdefender.com/en-us/blog/hotforsecurity/chatgp...
  - jcgrillo 4mo ago
    Yeah despite the conceptual statelessness, there is quite a bit of state that hangs around though--KV cache and context. I still haven't been able to find anything concrete in docs about how these are isolated. In any case it's clearly a different class of issue than the one from the article. Not endemic to how LLMs work, just normal web session stuff, modulo some GPU memory handling.
    - ipython 4mo ago
      As far as I know the only data of the two you identified are cached inside of the inference layer - the KV cache. Then again, I am not an expert in designing and operating inference, so I could be incorrect on that. Either way, both of those are controlled by deterministic code and not the LLM itself. So controlling for that risk is much simpler to model IMO since the mitigation can be applied universally and deterministically rather than hoping and praying some non-deterministic system will respect your wishes.
      - wolttam 4mo ago
        In other words: controlling for that kind of potential data-mixing is the same as in any other application where customer data is co-located within the same running process/memory/storage space.
        4mo ago