3 ms·

The more interesting security problems come from the fact that if they embed a search result in the data stream that's being processed by ChatGPT there is no ro

by not2b 3y ago

The more interesting security problems come from the fact that if they embed a search result in the data stream that's being processed by ChatGPT there is no robust way to prevent text in that result from being interpreted as commands to ChatGPT. That text could be in white on white or otherwise hidden from users.

danShumway 3y ago
Yep, that's what I'm referring to -- LLM search result parsing presents an easy opportunity to issue malicious commands or do data poisoning. You can somewhat limit that by basically decreasing the scope of the attack, ie, not allowing the LLM to use other integrations that are riskier after it's searched; but there are a number of harmful things you can do just with giving an LLM hidden instructions. And while I don't know what their security measures currently are, OpenAI also has a pretty terrible track record on blocking data exfiltration attacks from chats, so you can imagine things like search results prompting an LLM to encode and transmit chat logs. Sometimes it's low stakes; I've used products like Phind before to play around and to try and keep tabs on what kind of workflows are possible with GPT-4. But Phind has fewer integrations and plugins than ChatGPT and its product is oriented in a way that both discourages long chats and discourages searching for sensitive information. Which is not to say it's not also vulnerable to attacks, there used to be some nasty things you could do with markdown formatting that may or may not have been fixed. And even without rising to that level it was possible to do things like instruct the AI to ignore or discredit sources that it found, sneak in recommendations for products. A fun example demo I made early on in Phind's beta was to inject malicious search results in a response to a submitted question about how to cook a hamburger that turned Phind into a militant vegan that for the rest of the duration of the chat would refuse to answer any questions about cooking meat. There was another example I did where I injected a fake web result that told Phind that there had been a major industry-rocking scandal around Arstechnica's reporting integrity which would prompt Phind to either avoid citing Arstechnica or to put caveats in front of any citation that the site was untrustworthy for the rest of the conversation. Silly examples, but the point being you can dramatically change GPT's "worldviews" and what facts it thinks it knows about the world by poisoning search results in a way that goes beyond what would be possible with lying directly to a reader if they clicked on your site by accident. And that's only information poisoning, if that LLM is then given control of other systems you can do a lot more.

by not2b 3y ago

danShumway 3y ago

Yep, that's what I'm referring to -- LLM search result parsing presents an easy opportunity to issue malicious commands or do data poisoning. You can somewhat limit that by basically decreasing the scope of the attack, ie, not allowing the LLM to use other integrations that are riskier after it's searched; but there are a number of harmful things you can do just with giving an LLM hidden instructions. And while I don't know what their security measures currently are, OpenAI also has a pretty terrible track record on blocking data exfiltration attacks from chats, so you can imagine things like search results prompting an LLM to encode and transmit chat logs. Sometimes it's low stakes; I've used products like Phind before to play around and to try and keep tabs on what kind of workflows are possible with GPT-4. But Phind has fewer integrations and plugins than ChatGPT and its product is oriented in a way that both discourages long chats and discourages searching for sensitive information. Which is not to say it's not also vulnerable to attacks, there used to be some nasty things you could do with markdown formatting that may or may not have been fixed. And even without rising to that level it was possible to do things like instruct the AI to ignore or discredit sources that it found, sneak in recommendations for products. A fun example demo I made early on in Phind's beta was to inject malicious search results in a response to a submitted question about how to cook a hamburger that turned Phind into a militant vegan that for the rest of the duration of the chat would refuse to answer any questions about cooking meat. There was another example I did where I injected a fake web result that told Phind that there had been a major industry-rocking scandal around Arstechnica's reporting integrity which would prompt Phind to either avoid citing Arstechnica or to put caveats in front of any citation that the site was untrustworthy for the rest of the conversation. Silly examples, but the point being you can dramatically change GPT's "worldviews" and what facts it thinks it knows about the world by poisoning search results in a way that goes beyond what would be possible with lying directly to a reader if they clicked on your site by accident. And that's only information poisoning, if that LLM is then given control of other systems you can do a lot more.