3 ms·
We need to stop anthropomorphizing LLMs. They are models, not intelligences. An LLM isn't gullible. An LLM is inclusive. It doesn't have any process to reject
by thomastjeffery 3y ago
We need to stop anthropomorphizing LLMs. They are models, not intelligences.
An LLM isn't gullible. An LLM is inclusive. It doesn't have any process to reject or even categorize the data it is given. That means that everything an LLM encounters becomes part of it.
The only thing we have to accommodate that is to fill an LLM with enough overconfident narrative to overshadow the rest. This is a footgun, though, because overconfidence is identical to stubbornness. If your overconfident narrative is enough to override any arbitrary undesirable input, it will also override desirable input.
There is no way to single out undesirable content from inside a model, because truth and lie are literally written the same.
- pjc50 3y ago> truth and lie are literally written the same. Truth and lie are outside the model. This is true of all models. The map is not the territory.
- thomastjeffery 3y agoMy point is that truth and lie are ambiguous. There is nothing in the metaphorical map to distinguish between them. Because a truth and a lie are written the same, they are the same. The model puts them in the same place. The only difference an LLM can have is what other semantic groups of tokens are associated with the [truth|lie] tokens. If the surrounding [context of a truth] is separate enough from the surrounding [context of a lie], then the connections themselves can be unambiguous enough that the LLM doesn't mix them up. In the average case, there is some semantic/narrative overlap, which allows the [truth|lie] itself to act as a semantic bridge between the [context of truth] and the [context of lie]. Because of that, simply adding more content to a model can create bridges of ambiguity that fundamentally dissolve the model's ability to present unambiguous continuations.
- simonw 3y agoI really like "gullibility" as a way of explaining this limitation to people - both technical and non-technical alike. I think it's worth the anthropomorphizing risk. See also https://simonwillison.net/2023/Apr/7/chatgpt-lies/ https://simonwillison.net/2023/Apr/7/chatgpt-lies/
- thomastjeffery 3y agoI don't, because it implies the possibility of a solution. It's not a limitation or a bug. It's a feature. No LLM will ever overcome this "limitation" without fundamentally changing from an LLM to something else.
- simonw 3y agoThat still works for me. I've been telling people that LLMs are inherently gullible, and asking them how they are accounting for that in their software designs. I agree: I don't think gullibility will be solved by LLMs alone. The key thing is that people building software on top of LLMs understand that and use that knowledge to avoid making bad design decisions. See also: https://simonwillison.net/2023/Dec/20/mitigate-prompt-injection/ https://simonwillison.net/2023/Dec/20/mitigate-prompt-inject...
- thomastjeffery 3y agoI generally agree, but the reason I want to nitpick is that the narrative we have implies that things will get better; and that in turn implies that they may have already, right under our noses.