3 ms·

This may seem obvious to you and others, but giving an LLM agent write access to a database is a big no-no that is worthy of fears. There's actually a lot of re

This may seem obvious to you and others, but giving an LLM agent write access to a database is a big no-no that is worthy of fears. There's actually a lot of really good reasons to do that from the standpoint of product usefulness! But then you've got an end-user reprogrammable agent that could cause untold mayhem to your database: overwrite critical info, exfiltrate customer data, etc. Now the "obvious" answer here is to just not do that, but I would wager it's not terribly obvious to a lot of people, and moreoever, without making it clear what the risks are, the people who might object to doing this in an organization could "lose" to the people who argue for more product usefulness.

danShumway 3y ago
Agreed, and notably, the people with safety concerns are already regularly "losing" to product designers who want more capabilities. Wuzzie's blog (https://embracethered.com/blog/ https://embracethered.com/blog/) has a number of examples of data exfiltration that would be largely prevented by merely sanitizing Markdown output and refusing to auto-fetch external resources like images in that Markdown output. In some cases, companies have been convinced to fix that. But as far as I know, OpenAI still refuses to change that behavior for ChatGPT, even though they're aware it presents an exfiltration risk. And I think sanitizing Markdown output in the client, not allowing arbitrary image embeds from external domains -- it's the bottom of the barrel, it's something I would want handled in many applications even if they weren't being wired to an LLM. ---- It's tricky to link to older resources because the space moves fast and (hopefully) some of these examples have changed or the companies have introduced better safeguards, but https://kai-greshake.de/posts/in-escalating-order-of-stupidity/ https://kai-greshake.de/posts/in-escalating-order-of-stupidi... highlights some of the things that companies are currently trying to do with LLMs, including "wire them up to external data and then use them to help make military decisions." There are a subset of people who correctly point out that with very careful safeguards around access, usage, input, and permissions, these concerns can be mitigated either entirely or at least to a large degree -- the tradeoff being that this does significantly limit what we can do with LLMs. But the overall corporate space either does not understand the risks or is ignoring them.
dragonwriter 3y ago
> This may seem obvious to you and others, but giving an LLM agent write access to a database is a big no-no that is worthy of fears. That's...a risk area for prompt injection, but any interaction outside the user-LLM conduit, even if it is not "write access to a database" in an obvious way -- like web browsing -- is a risk. Why? Because (1) even if it is only GET requests, GET requests can be used to transfer information to remote servers, (2) because the content of GET requests must be processed through the LLM prompt to be used in formulating a response, it means that data from external sources (not just the user) can be used for prompt injection. That means, if an LLM has web browsing capability, there is a risk that (1) third party (not user) prompt injection may be carried out, and that (2) this will result in leakage of any information available to the LLM, including from the user request, being leaked to an external entity. Now, if you have web browsing plus more robust tool access where the LLM has authenticated access to user email and other accounts, (even if it is only read access, though the ability to write to or take other non-query actions adds more risk) expands the scope of risk, because it is more data that can be leaked with read access, as well as more user-adverse actions that can be taken with write access, all of which conceivably could be triggered by third party content (and if the personal sources to which it has access also contain third party sourced content -- e.g., email accounts have content from the mail sender -- they are also additional channels through which an injection can be initiated, as well as additional sources of data that can be exfiltrated by an injection.)

danShumway 3y ago

Agreed, and notably, the people with safety concerns are already regularly "losing" to product designers who want more capabilities. Wuzzie's blog (https://embracethered.com/blog/ https://embracethered.com/blog/) has a number of examples of data exfiltration that would be largely prevented by merely sanitizing Markdown output and refusing to auto-fetch external resources like images in that Markdown output. In some cases, companies have been convinced to fix that. But as far as I know, OpenAI still refuses to change that behavior for ChatGPT, even though they're aware it presents an exfiltration risk. And I think sanitizing Markdown output in the client, not allowing arbitrary image embeds from external domains -- it's the bottom of the barrel, it's something I would want handled in many applications even if they weren't being wired to an LLM. ---- It's tricky to link to older resources because the space moves fast and (hopefully) some of these examples have changed or the companies have introduced better safeguards, but https://kai-greshake.de/posts/in-escalating-order-of-stupidity/ https://kai-greshake.de/posts/in-escalating-order-of-stupidi... highlights some of the things that companies are currently trying to do with LLMs, including "wire them up to external data and then use them to help make military decisions." There are a subset of people who correctly point out that with very careful safeguards around access, usage, input, and permissions, these concerns can be mitigated either entirely or at least to a large degree -- the tradeoff being that this does significantly limit what we can do with LLMs. But the overall corporate space either does not understand the risks or is ignoring them.

dragonwriter 3y ago

> This may seem obvious to you and others, but giving an LLM agent write access to a database is a big no-no that is worthy of fears. That's...a risk area for prompt injection, but any interaction outside the user-LLM conduit, even if it is not "write access to a database" in an obvious way -- like web browsing -- is a risk. Why? Because (1) even if it is only GET requests, GET requests can be used to transfer information to remote servers, (2) because the content of GET requests must be processed through the LLM prompt to be used in formulating a response, it means that data from external sources (not just the user) can be used for prompt injection. That means, if an LLM has web browsing capability, there is a risk that (1) third party (not user) prompt injection may be carried out, and that (2) this will result in leakage of any information available to the LLM, including from the user request, being leaked to an external entity. Now, if you have web browsing plus more robust tool access where the LLM has authenticated access to user email and other accounts, (even if it is only read access, though the ability to write to or take other non-query actions adds more risk) expands the scope of risk, because it is more data that can be leaked with read access, as well as more user-adverse actions that can be taken with write access, all of which conceivably could be triggered by third party content (and if the personal sources to which it has access also contain third party sourced content -- e.g., email accounts have content from the mail sender -- they are also additional channels through which an injection can be initiated, as well as additional sources of data that can be exfiltrated by an injection.)