3 ms·
LLMs are already trained on a lot of dubious sources (for example Quora and Reddit) so it seems your theoretical concern is already controlled for to some exten
by fny 2y ago
LLMs are already trained on a lot of dubious sources (for example Quora and Reddit) so it seems your theoretical concern is already controlled for to some extent.
You’re also giving far too much credit to how much a single piece of information can poison a model’s output.
To me, the more realistic concern would be deliberately shifting and censoring an LLMs output at training to adhere to an agenda be it political or vindictive (see DeepSeek.)
- pona-a 2y agoA lot of tokenization bugs came from odd Reddit subreddits, such as r/counting. And back then, we didn't have free model weights, which is they all converge to similar weights, might tell us a lot about how to design adversarial examples.
- Terr_ 2y ago> LLMs are already trained on a lot of dubious sources (for example Quora and Reddit) There's a very big difference between crawling documents where people happen to say something wrong (e.g. "Tortoises are amphibians") versus the impact of widespread poisoning [0] or indirect prompt injection. [1] [0] https://arxiv.org/html/2302.10149v2 https://arxiv.org/html/2302.10149v2 [1] https://cetas.turing.ac.uk/publications/indirect-prompt-injection-generative-ais-greatest-security-flaw https://cetas.turing.ac.uk/publications/indirect-prompt-inje...