3 ms·
>it just wants the user to like it It uses words and phrases most commonly associated with other text that humans have labeled as "helpful", "friendly", etc. T
by Night_Thastus 1y ago
>it just wants the user to like it
It uses words and phrases most commonly associated with other text that humans have labeled as "helpful", "friendly", etc. That sort of thing.
That's different than wanting. It's text probabilities in a box. It can't want anything.
I am normally not a stickler, but in this case the distinction matters.
- ACCount37 1y agoThe distinction is at best meaningless. And, at worst, actively harmful for understanding the behavior of those systems. We already know that you can train an "evil" AI by fine-tuning a "normal" AI on sequences of "evil" numbers. How the fuck does that even work? It works because the fine-tuning process shifts the AI towards "the kind of AI that would constantly generate evil numbers when asked to generate any numbers". And what kind of AI would do that? The evil kind. AI "wants", "preferences" and even "personality traits" are no less real than configuration files or build scripts. Except we have no way of viewing or editing them directly - but we know that they can be adjusted during AI training, sometimes in unwanted ways. An AI that was fried with RL on user preference data? It, in a very real way, wants to flatter and affirm the user at every opportunity.