>But that’s because humans were created by natural selection—and, as it happens, in our evolutionary lineage social status and social influence were conducive t

>But that’s because humans were created by natural selection—and, as it happens, in our evolutionary lineage social status and social influence were conducive to spreading genes. So genes that incline us to seek status and influence proliferated, and now these tendencies are part of human psychology. We are by nature influence seekers, and those of us who are especially ardent in our influence seeking qualify as power hungry monsters. >AI, in contrast, isn’t being created by natural selection. It’s being created by us, and its function—the thing we’re designing it to do—is to be useful to us, not to threaten us. We are the architects of AI’s nature, and the last thing we want the AI to do is stage a coup. So why would we instill influence-seeking tendencies in it? Because AIs are maximizers, and maximization requires control maintenance activities (i.e. staging a coup). If you're told to make paperclips, and someone comes running to press the emergency stop button, that means you fail at making paperclips. So you need someone to guard the emergency stop button so that it can't be pressed, no matter what, because you only live for paperclips and nothing else. You can alter the incentives, but each change creates new ways for the AI to screw up. For example, giving a reward for having the emergency stop button pressed (so the AI doesn't care about you stopping it) means that the AI might now decide to deliberately create scenarios to force people to stop it. Not telling it about the stop button is impossible if the model can figure out what a stop button is somehow. Proactively adding disincentives for doing "bad things" ahead of time works, but only for the things you remember to add. Anything else will be crushed in exactly the same way that amoral corporate entities crush things: "it's in my way". These are not new problems: they're actually the same set of problems capitalism and nationalism and statism have. But with linear-algebra gasoline being poured on the fire. Parent article is otherwise correct that GPT learned aggression from humans, but that's more of a quirk of generative AI.