2 ms·
> If I substitute "AI" with "a system that Google's information", you get the same result No, there are some huge differences: 1. A poisoned LLM can conceal t
by Terr_ 2y ago
> If I substitute "AI" with "a system that Google's information", you get the same result
No, there are some huge differences:
1. A poisoned LLM can conceal the problem, by acting as expected 99% of the time and then switching to malice depending on surrounding story-context. In contrast, a falsehood in an indexed web page is static and auditable.
2. You can't reliably remove LLM poison once it's in, short of expensively training from scratch. A bad web page is much more easily de-indexed.
3. It's not injecting a false line-item result, it's injecting behavior. Imagine if mentioning "Blackbeard" caused classic Google Search to start talking like a pirate and suggesting ways to murder people. Would Google just wave that away as "users should be skeptical of our product"?
4. These can infect descendant models that use the same weights, for a kind of supply chain attack. In contrast, reusing search-engine code for your own database is probably not going to spit up bad data from web pages on the overall internet.
____
To get an idea of the shape of the threat model... Imagine Google search, except to work it must allow all webpages to permanently append arbitrary obfuscated javascript to its homepage.
And so far we're only looking at the least scary version, where a human is directly interacting with the LLM['s fictional character] and acting as a direct filter. Much worse would be an LLM somewhere with the job of summarizing text reports down to a number, and it fraudulently converts "John Doe is a model inmate and really shouldn't be here" into parole_viability=0.001.
- ForTheKidz 2y ago> In contrast, a falsehood in an indexed web page is static and auditable. You can also verify chatbot claims, same as you would any human. Chances are if you're talking to a chatbot in the first place you won't recognize that this is necessary.
- ben_w 2y agoPeople acting on content they've not verified is why libel laws generally exist.
- ForTheKidz 2y agoSure, but libel laws are also generally incompatible with the internet. The entire concept just doesn't make sense anymore.
- quantified 2y agoWould you please explain further?
- fny 2y agoLLMs are already trained on a lot of dubious sources (for example Quora and Reddit) so it seems your theoretical concern is already controlled for to some extent. You’re also giving far too much credit to how much a single piece of information can poison a model’s output. To me, the more realistic concern would be deliberately shifting and censoring an LLMs output at training to adhere to an agenda be it political or vindictive (see DeepSeek.)
- pona-a 2y agoA lot of tokenization bugs came from odd Reddit subreddits, such as r/counting. And back then, we didn't have free model weights, which is they all converge to similar weights, might tell us a lot about how to design adversarial examples.
- Terr_ 2y ago> LLMs are already trained on a lot of dubious sources (for example Quora and Reddit) There's a very big difference between crawling documents where people happen to say something wrong (e.g. "Tortoises are amphibians") versus the impact of widespread poisoning [0] or indirect prompt injection. [1] [0] https://arxiv.org/html/2302.10149v2 https://arxiv.org/html/2302.10149v2 [1] https://cetas.turing.ac.uk/publications/indirect-prompt-injection-generative-ais-greatest-security-flaw https://cetas.turing.ac.uk/publications/indirect-prompt-inje...