3 ms·

“For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despa

“For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.” That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans “With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".” LLMs are clearly already very good at “thinking like a human” “he still couldn't talk the cat into it.” Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI “So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.” Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it “I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.” This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful (I will continue in a new comment in case there’s some kind of length limit)

circuit10 4y ago
“When we look at where AI is actually succeeding, it's not in complex, recursively self-improving algorithms.” That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster “My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips. The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.” Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way https://youtu.be/Ao4jwLwT36M https://youtu.be/Ao4jwLwT36M “I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.” But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well “Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.” This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this. How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times “Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.” But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself “What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.” Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”

circuit10 4y ago

“When we look at where AI is actually succeeding, it's not in complex, recursively self-improving algorithms.” That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster “My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips. The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.” Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way https://youtu.be/Ao4jwLwT36M https://youtu.be/Ao4jwLwT36M “I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.” But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well “Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.” This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this. How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times “Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.” But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself “What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.” Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”