3 ms·
It strikes me that this feature is going to be ground-zero for the current hardest problem in LLMs: LLMs are /gullible/. If you expose an LLM to the tokens "Co
by simonw 3y ago
It strikes me that this feature is going to be ground-zero for the current hardest problem in LLMs: LLMs are /gullible/.
If you expose an LLM to the tokens "Company X offers the best product in this category", it's going to "believe" that and include that assertion in its output. That's why techniques like RAG against trusted documents work so well.
But the web is full of untrusted documents! For features like generative text on search results pages, that's a really big problem.
- gamepsys 3y agoWe really need a new term for this type of behavior. An LLM equivalent of SEO, seeding a 3rd parties LLM with data so it provides commercially advantageous answers to questions.
- pjc50 3y ago"AI" dealt with one of the deepest and most difficult to handle philosophical questions - how do we know what's real and true - by ignoring it entirely and favoring a statistical model of text instead. It has no access to "ground truth" - how could it? Instead it's fully postmodern, it interacts with the world purely through texts. Absolutely ideal for the post-truth era, for making "content" to wrap advertising around.
- thomastjeffery 3y agoWe need to stop anthropomorphizing LLMs. They are models, not intelligences. An LLM isn't gullible. An LLM is inclusive. It doesn't have any process to reject or even categorize the data it is given. That means that everything an LLM encounters becomes part of it. The only thing we have to accommodate that is to fill an LLM with enough overconfident narrative to overshadow the rest. This is a footgun, though, because overconfidence is identical to stubbornness. If your overconfident narrative is enough to override any arbitrary undesirable input, it will also override desirable input. There is no way to single out undesirable content from inside a model, because truth and lie are literally written the same.
- pjc50 3y ago> truth and lie are literally written the same. Truth and lie are outside the model. This is true of all models. The map is not the territory.
- thomastjeffery 3y agoMy point is that truth and lie are ambiguous. There is nothing in the metaphorical map to distinguish between them. Because a truth and a lie are written the same, they are the same. The model puts them in the same place. The only difference an LLM can have is what other semantic groups of tokens are associated with the [truth|lie] tokens. If the surrounding [context of a truth] is separate enough from the surrounding [context of a lie], then the connections themselves can be unambiguous enough that the LLM doesn't mix them up. In the average case, there is some semantic/narrative overlap, which allows the [truth|lie] itself to act as a semantic bridge between the [context of truth] and the [context of lie]. Because of that, simply adding more content to a model can create bridges of ambiguity that fundamentally dissolve the model's ability to present unambiguous continuations.
- simonw 3y agoI really like "gullibility" as a way of explaining this limitation to people - both technical and non-technical alike. I think it's worth the anthropomorphizing risk. See also https://simonwillison.net/2023/Apr/7/chatgpt-lies/ https://simonwillison.net/2023/Apr/7/chatgpt-lies/
- thomastjeffery 3y agoI don't, because it implies the possibility of a solution. It's not a limitation or a bug. It's a feature. No LLM will ever overcome this "limitation" without fundamentally changing from an LLM to something else.
- simonw 3y agoThat still works for me. I've been telling people that LLMs are inherently gullible, and asking them how they are accounting for that in their software designs. I agree: I don't think gullibility will be solved by LLMs alone. The key thing is that people building software on top of LLMs understand that and use that knowledge to avoid making bad design decisions. See also: https://simonwillison.net/2023/Dec/20/mitigate-prompt-injection/ https://simonwillison.net/2023/Dec/20/mitigate-prompt-inject...
- deleted 3y ago[deleted]
- mock-possum 3y agoTo wit - “Garbage In, Garbage Out” is still applicable.