5 ms·

I think another common fallacy is to assume that humans have more general intelligence than we actually do. This is manifest in three ways: 1. Assuming that we

by audunw 6y ago

I think another common fallacy is to assume that humans have more general intelligence than we actually do. This is manifest in three ways: 1. Assuming that we can build an AI than can do what a human does, without being embedded in the physical world and in constant communication with other humans in the same way we are. I think this is covered by "Intelligence is all in the brain" in the article 2. Not considering that intelligent breakthroughs by humans isn't partly a product of chance.. of millions, and now billions of humans trying semi-random things. Obviously there's SOME intelligence behind it, otherwise we wouldn't have achieved more than other animals. But maybe we're overestimating how much of our results are product of intelligence alone. 3. Not considering that we use pretty dumb heuristics to come to decisions. I think the paperclip maximizer is a silly example, because the decision of whether an AI should kill or cooperate with humanity to maximize the production of paperclips is probably undecidable. Coming to a clear decision probably requires more computing power than you could have on a single planet. We humans don't need to be certain about the outcome to make a decision. We have emotions like fear, anger, pride and jealousy to nudge us towards decisions like going to war with other people to grab their resources. We need to remember that we're a product of evolving in an environment where we had to compete with other humans for resources, so that's why we're prone to those kinds of decisions. AIs will be product of an environment of competing to please humans to gain computational resources. So the heuristics they develop will probably be strongly tied to achieving that goal. Not that AIs couldn't be dangerous, put probably more because humans make active decisions to instruct AIs to harm other humans.

someguyorother 6y ago
I like this refutation of the paperclip maximizer scenario, which ties in to your observation: Let's say there's a runaway superintelligence equipped with an optimization function that says it should produce a maximal amount of paperclips. Then either its objective, "produce a maximal amount of paperclips" is defined literally or not. If it's defined literally, e.g. "make your sensors return this data consistent with lots of paperclips having been produced", then it's far easier for the AI to corrupt its sensors with paperclip porn than actually destroy the Earth, and there's no problem. On the other hand, if the objective is not defined literally, then the AI must be able to understand fuzzy instructions in the way the humans intended it to. In that case, it would be no problem to tell the AI to not be an ass either. So the problem happens when the AI isn't intelligent enough, and uses heuristics instead of properly optimizing. The extreme case would be grey goo, which doesn't think at all, but just blindly consumes everything. Giving extreme power to something with limited intelligence is generally a bad move.
- newsbinator 6y ago
  The fear is the AI misunderstanding fuzzy instructions in an unpredictable way.
- Isinlor 6y ago
  Thinking about it in terms of intelligence is actually misleading. It's not about intelligence, it's about power to change the world. Take SARS-CoV-2. It is a small RNA program ~32kb that duplicates and optimizes itself by random search. It managed to hijack more computing power than our best supercomputers have [0] and kill a lot of us in the process. By my estimates just the process of copying all SARS-CoV-2 vrions in the world with reproduction rate of 1 takes at least 1 petaFLOPS and up to 100 exaFLOPS. That's in the range of TOP500 supercomputers (1.3 petaFLOPS - 442 petaFLOPS) [1]. SARS-CoV-2 has certainly a lot of computing power and is still able to outsmart the whole human civilization. Is that a super-intelligence? Imagine if we manage to create equally stupid program that will figure out how to hijack heat or electrical power of our civilization and feed it into its own growth. Quite likely it would result in nuclear meltdowns all around the world. Exponentially growing computational processes are dangerous no matter if they are intelligent or whether they even have any objectives besides just being. [0] https://news.ycombinator.com/item?id=26646029 https://news.ycombinator.com/item?id=26646029 [1] https://www.top500.org/statistics/perfdevel/ https://www.top500.org/statistics/perfdevel/
  - someguyorother 6y ago
    Which the refutation makes clear: it's not about intelligence, because a sufficiently powerful intelligence will just outwit its objective function in the most efficient way possible (which is closer to "porn" than "destroy the Earth"). Grey goo is powerful not because it can outwit, but because it has so much brute force. On the other hand, intelligence is closer to efficiency of action: not requiring exponential time to solve something that's NP-complete, for instance. I don't think you can translate computing power like you do (a crypto miner ASIC handles a lot of bits per second, but zero FLOPS as it's all integer math; a nuclear bomb is a self-propagating reaction that alters the mass equivalent of lots of bits, but zero FLOPS). But apart from that, I agree. And I just think the "beware AI, it will become so smart that it turns into the sorcerer's apprentice" fear is misguided, because it focuses on the wrong thing. It sure makes a good narrative though! That's why it's so popular.
nil-sec 6y ago
This is a good point which I think stems from wrongly equating human level intelligence to AGI in the popular literature. It’s not at all clear what a general intelligence should be and it’s much less clear that humans have general intelligence. In my view we have a set of very good innate priors (e.g. space/time continuity, intuitive physics) that have been optimized by evolution for thousands of years. These priors in turn allow us to learn fast from unlabeled data, but would anyone call such a system general intelligence? I’m not sure.
- newsbinator 6y ago
  I call General Intelligence any system that wonders for no particular reason if others systems have General Intelligence.

by audunw 6y ago

someguyorother 6y ago

I like this refutation of the paperclip maximizer scenario, which ties in to your observation: Let's say there's a runaway superintelligence equipped with an optimization function that says it should produce a maximal amount of paperclips. Then either its objective, "produce a maximal amount of paperclips" is defined literally or not. If it's defined literally, e.g. "make your sensors return this data consistent with lots of paperclips having been produced", then it's far easier for the AI to corrupt its sensors with paperclip porn than actually destroy the Earth, and there's no problem. On the other hand, if the objective is not defined literally, then the AI must be able to understand fuzzy instructions in the way the humans intended it to. In that case, it would be no problem to tell the AI to not be an ass either. So the problem happens when the AI isn't intelligent enough, and uses heuristics instead of properly optimizing. The extreme case would be grey goo, which doesn't think at all, but just blindly consumes everything. Giving extreme power to something with limited intelligence is generally a bad move.

newsbinator 6y ago
The fear is the AI misunderstanding fuzzy instructions in an unpredictable way.
Isinlor 6y ago
Thinking about it in terms of intelligence is actually misleading. It's not about intelligence, it's about power to change the world. Take SARS-CoV-2. It is a small RNA program ~32kb that duplicates and optimizes itself by random search. It managed to hijack more computing power than our best supercomputers have [0] and kill a lot of us in the process. By my estimates just the process of copying all SARS-CoV-2 vrions in the world with reproduction rate of 1 takes at least 1 petaFLOPS and up to 100 exaFLOPS. That's in the range of TOP500 supercomputers (1.3 petaFLOPS - 442 petaFLOPS) [1]. SARS-CoV-2 has certainly a lot of computing power and is still able to outsmart the whole human civilization. Is that a super-intelligence? Imagine if we manage to create equally stupid program that will figure out how to hijack heat or electrical power of our civilization and feed it into its own growth. Quite likely it would result in nuclear meltdowns all around the world. Exponentially growing computational processes are dangerous no matter if they are intelligent or whether they even have any objectives besides just being. [0] https://news.ycombinator.com/item?id=26646029 https://news.ycombinator.com/item?id=26646029 [1] https://www.top500.org/statistics/perfdevel/ https://www.top500.org/statistics/perfdevel/
- someguyorother 6y ago
  Which the refutation makes clear: it's not about intelligence, because a sufficiently powerful intelligence will just outwit its objective function in the most efficient way possible (which is closer to "porn" than "destroy the Earth"). Grey goo is powerful not because it can outwit, but because it has so much brute force. On the other hand, intelligence is closer to efficiency of action: not requiring exponential time to solve something that's NP-complete, for instance. I don't think you can translate computing power like you do (a crypto miner ASIC handles a lot of bits per second, but zero FLOPS as it's all integer math; a nuclear bomb is a self-propagating reaction that alters the mass equivalent of lots of bits, but zero FLOPS). But apart from that, I agree. And I just think the "beware AI, it will become so smart that it turns into the sorcerer's apprentice" fear is misguided, because it focuses on the wrong thing. It sure makes a good narrative though! That's why it's so popular.

nil-sec 6y ago

This is a good point which I think stems from wrongly equating human level intelligence to AGI in the popular literature. It’s not at all clear what a general intelligence should be and it’s much less clear that humans have general intelligence. In my view we have a set of very good innate priors (e.g. space/time continuity, intuitive physics) that have been optimized by evolution for thousands of years. These priors in turn allow us to learn fast from unlabeled data, but would anyone call such a system general intelligence? I’m not sure.

newsbinator 6y ago
I call General Intelligence any system that wonders for no particular reason if others systems have General Intelligence.