linked study in TFA https://arxiv.org/abs/2411.02306 https://arxiv.org/abs/2411.02306 > training to maximize human feedback creates a perverse incentive struc

linked study in TFA https://arxiv.org/abs/2411.02306 https://arxiv.org/abs/2411.02306 > training to maximize human feedback creates a perverse incentive structure for the AI to resort to manipulative or deceptive tactics to obtain positive feedback from users who are vulnerable to such strategies. We study this phenomenon by training LLMs with Reinforcement Learning with simulated user feedback in environments of practical LLM usage. it seems optimising for what people want isn’t an ideal strategy from an ethical perspective — guess we haven’t learned from social media feeds as a species. awesome. anyway, who cares about ethics, we got market share, moats and PMF to worry about over here. this money doesn’t grow on trees y’know. /s