4 ms·
> You must never feed user input into a combined instruction and data stream. Well, I have some bad news about how LLMs work...
by rafram 1y ago
> You must never feed user input into a combined instruction and data stream.
Well, I have some bad news about how LLMs work...
- franga2000 1y agoThat's my point exactly. The only acceptable way to feed user input into an LLM is if its capabilities are constrained to only what you'd give the author of the input. If an LLM reads emails, it should only have the ability to create and display output, nothing more.
- Terr_ 1y agoI like to quip that people need to imagine LLM security as at-least-as-bad as javascript code in someone else's web-browser: A determined person can make them emit whatever they want (as noted above) and also none of the data that went into them is reliably secret either. As an analogy, it still needs some work through, since it doesn't adequately alarm people about the risks of covertly poisonous data even with an honest user.
- QuadmasterXLII 1y agoAnd even that’s imperfect if you miss an integration with an externally visible effect- for example an agent with web search can exfiltrate info via visiting specific urls with that log visitors- I’ve POC’d this with claude in the browser, although I only got a few bits out since you need to get N pages ranked on google to exfiltrate log (N!) bits
- furyofantares 1y agoI suspect people routinely paste urls to documentation into claude code, which it will fetch, or maybe tell it to do web searches. It has those tools built in.
- ryukoposting 1y agoYou're right, but if all it can do is create and display output, then how will any of the oligarchs betting the farm on LLMs actually fulfill the breathless promises they've made?