4 ms·

AGI would dwarf those 7 billion people because it would concentrate tremendous power into the hands of a very few. It's the dream of being one of the few who g

AGI would dwarf those 7 billion people because it would concentrate tremendous power into the hands of a very few. It's the dream of being one of the few who gets to control and direct that concentrated power that fuels these dreams, which is why it's imperative that they dress it up in the language of benefiting society. The essence of the ethical problem with AI is that there is no person or small group of people who can be trusted to use such power without creating a real dystopia for the rest of us.

fossuser 7y ago
I think this is a pretty big misunderstanding of the AGI issue. Nobody is going to control the hyper-intelligent AGI if its goals are not aligned with human goals more generally. That's the nature of something being a lot smarter than you with its own goals.
- sverige 7y ago
  Wait, is the claim that intelligence alone determines who is in control? I've certainly seen lots of examples where people were controlled by others, even though they were orders of magnitude more intelligent than those who had power. Are the people who are trying to make AI a reality planning to give away the ability to unplug the machine it will inevitably depend upon, or give it the ability to control nuclear weapons so that it can wipe out humanity before that happens, like a bad movie script? It really does seem that ridiculous to aver that the creators of AI, if it ever comes to be, won't retain ultimate control over what it can do in the physical world. Whatever its goals end up being, they will be aligned with Microsoft's goals if OpenAI gets there first. That's what the billion dollars is meant to ensure.
  - fossuser 7y ago
    I don’t think humans are orders of magnitude apart. The difference between the world’s smartest human and its dumbest human is tiny relative to the possible spectrum of intelligence. Do you see the smartest chimps tricking or controlling humans? If you wanted to trick a chimp to do something you wanted it to do, do you think it could stop you? And we’re probably a lot closer to chimp intelligence than an AGI would be to us.

fossuser 7y ago

I think this is a pretty big misunderstanding of the AGI issue. Nobody is going to control the hyper-intelligent AGI if its goals are not aligned with human goals more generally. That's the nature of something being a lot smarter than you with its own goals.

sverige 7y ago
Wait, is the claim that intelligence alone determines who is in control? I've certainly seen lots of examples where people were controlled by others, even though they were orders of magnitude more intelligent than those who had power. Are the people who are trying to make AI a reality planning to give away the ability to unplug the machine it will inevitably depend upon, or give it the ability to control nuclear weapons so that it can wipe out humanity before that happens, like a bad movie script? It really does seem that ridiculous to aver that the creators of AI, if it ever comes to be, won't retain ultimate control over what it can do in the physical world. Whatever its goals end up being, they will be aligned with Microsoft's goals if OpenAI gets there first. That's what the billion dollars is meant to ensure.
- fossuser 7y ago
  I don’t think humans are orders of magnitude apart. The difference between the world’s smartest human and its dumbest human is tiny relative to the possible spectrum of intelligence. Do you see the smartest chimps tricking or controlling humans? If you wanted to trick a chimp to do something you wanted it to do, do you think it could stop you? And we’re probably a lot closer to chimp intelligence than an AGI would be to us.