If we want to implement a human-like intelligence, I think it’s very likely that we’ll have to emulate a lot of the environment we inhabit. To me, human culture

If we want to implement a human-like intelligence, I think it’s very likely that we’ll have to emulate a lot of the environment we inhabit. To me, human culture and parts of our brain is like software which is instantiated in the individual. To run it on a machine instead means that the machine will have to emulate the “machine” that the software runs on, which is the whole brain, body and its environment. We could of course try to make a non-human type of intelligence. But it’s not at all clear what that would be, or if it’s at all possible. The only intelligence we know for sure can exist is animal/human style intelligence. And I don’t think there’s anyone who is actually trying to construct a non-human intelligence. To do that it’s very likely that you’d have to set up a computer environment where programs evolve naturally and fight for computing resources over a long time. It could take anywhere from decades to millions of years. Where most of our efforts are going, is to create augmented intelligence. We’re creating programs that expand on human intelligence. Interface with it. Cater to it. If you imagine AI programs as individuals in an evolutionary environment. What is it that they’re competing for? They’re competing for which program can most satisfy humans. Those that satisfy us live. Those that don’t die. Just as our evolutionary environment creates drives for gathering resources, cooperating with others when possible, hurting/killing others when necessary... programs are almost exclusively driven to satisfy humans. That’s why I think the “paperclip maximizer” example is so ridiculous. First it assumes that general intelligence is just some magical algorithm we just haven’t discovered yet. Then it assumes that such an AI can make catastrophic decisions without complex motivations. Whether to kill humans or not is more efficient for making paper clips is an undecidable problem. A human might kill someone to achieve its goal because that’s something our evolutionary environment has trained us for. We have motivations like pride, ego, anger, envy, etc. that overrides the problem of figuring out “is killing this human optimal for my goals” It’s far more likely that an AI catastrophe will be far harder to predict and much stranger. It could be that specialized (and relatively dumb in the genera sense) AIs become so good at satisfying our desires that we become completely incapacitated. There’s already signs of this. Just to be clear, I’m not saying we shouldn’t be worried about AI. But the alarmists seems to be too focused on various imagined future scenarios, all of which are likely to be wrong. We should keep a very keen eye on the consequences of AI right now in the present moment, and talk about problems that actually arise. Perhaps with a little bit of extrapolation.. thinking about how present day small problems could develop into bigger problems.