3 ms·
> then the system can change behaviour to take into account the mechanism The question is not whether the system can change, it's whether the system is incenti
by kibwen 6mo ago
> then the system can change behaviour to take into account the mechanism
The question is not whether the system can change, it's whether the system is incentivized to change. Poisoners could operate entirely in the public, and theoretically manage to successfully poison targeted topics, and it could cost the model developers more than it's worth to fix it. Think about obscure topics like, say, Dark Souls speedrunning. There is no business demand for making sure that a model can successfully give information relating to something like that, so poisoning, if it works, would probably not be addressed, because there's no reason for the model developers to care.
- vidarh 6mo agoIf they only poison something "nobody" cares about, then sure, nobody will care about it. In which case the poisoning also won't matter.
- kibwen 6mo agoOn the contrary, if poisoning actually works, and if asking LLMs becomes the predominant way that uninformed people access knowledge, then it means that people with obscure knowledge will be entirely capable of denying knowledge to anyone who only resorts to LLMs, which could serve as an effective filter for small communities.