3 ms·

> The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play Thinking our meat brain's consciousness is somehow spec

> The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?

antonvs 4y ago
> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. Lucky I didn't say anything remotely like that, then. > AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence? I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm
- The_Colonel 4y ago
  > Lucky I didn't say anything remotely like that, then. Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing". > Yet no-one is seriously concerned that GPT-4 will end the world. I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8. LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.
- circuit10 4y ago
  “For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.” That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans “With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".” LLMs are clearly already very good at “thinking like a human” “he still couldn't talk the cat into it.” Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI “So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.” Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it “I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.” This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful (I will continue in a new comment in case there’s some kind of length limit)
  - circuit10 4y ago
    “When we look at where AI is actually succeeding, it's not in complex, recursively self-improving algorithms.” That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster “My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips. The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.” Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way https://youtu.be/Ao4jwLwT36M https://youtu.be/Ao4jwLwT36M “I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.” But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well “Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.” This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this. How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times “Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.” But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself “What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.” Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”
skissane 4y ago
> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. Let's agree, for the sake of the argument, that super-intelligent AI is possible, and coming sooner or later. Why assume a super-intelligent AI would have any desire to exterminate or enslave humanity? Intelligence says nothing about what your goals are. Humans have various goals – food, drink, sex, sleep, power, status, curiosity, entertainment, etc – those goals are not inherent in our intelligence, they actually come from animal nature, and we share them with species far less intelligent than we are. What goals would a super-intelligent AI have? The fact that it is super-intelligent says nothing about what its goals would be, because intelligence in itself is goal-agnostic. Intelligence is a means, not an end. We get our goals from our animal nature – well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human? And even if an anti-human super-intelligent AIs isn't impossible – why assume there is only one super-intelligent AI? I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. But so long as the vast majority of them have goals that are broadly pro-human, the outcome of that competition is likely to be broadly pro-human too. The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence, so enslaving/exterminating humanity would be about as appealing to it as diving into a vat of fuming nitric acid would be to you or me.
- The_Colonel 4y ago
  > well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human? Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic. > I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. I agree, but that's actually what I worry about the most. I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role. But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes. I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah. In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination. As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well. > The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.

antonvs 4y ago

> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. Lucky I didn't say anything remotely like that, then. > AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence? I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm

The_Colonel 4y ago
> Lucky I didn't say anything remotely like that, then. Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing". > Yet no-one is seriously concerned that GPT-4 will end the world. I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8. LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.
circuit10 4y ago
“For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.” That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans “With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".” LLMs are clearly already very good at “thinking like a human” “he still couldn't talk the cat into it.” Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI “So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.” Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it “I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.” This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful (I will continue in a new comment in case there’s some kind of length limit)
- circuit10 4y ago
  “When we look at where AI is actually succeeding, it's not in complex, recursively self-improving algorithms.” That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster “My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips. The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.” Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way https://youtu.be/Ao4jwLwT36M https://youtu.be/Ao4jwLwT36M “I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.” But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well “Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.” This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this. How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times “Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.” But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself “What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.” Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”

skissane 4y ago

> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. Let's agree, for the sake of the argument, that super-intelligent AI is possible, and coming sooner or later. Why assume a super-intelligent AI would have any desire to exterminate or enslave humanity? Intelligence says nothing about what your goals are. Humans have various goals – food, drink, sex, sleep, power, status, curiosity, entertainment, etc – those goals are not inherent in our intelligence, they actually come from animal nature, and we share them with species far less intelligent than we are. What goals would a super-intelligent AI have? The fact that it is super-intelligent says nothing about what its goals would be, because intelligence in itself is goal-agnostic. Intelligence is a means, not an end. We get our goals from our animal nature – well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human? And even if an anti-human super-intelligent AIs isn't impossible – why assume there is only one super-intelligent AI? I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. But so long as the vast majority of them have goals that are broadly pro-human, the outcome of that competition is likely to be broadly pro-human too. The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence, so enslaving/exterminating humanity would be about as appealing to it as diving into a vat of fuming nitric acid would be to you or me.

The_Colonel 4y ago
> well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human? Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic. > I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. I agree, but that's actually what I worry about the most. I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role. But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes. I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah. In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination. As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well. > The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.