6 ms·

> Of course we don't know one way or the other, but it seems to me that it's possible that we're already roughly as intelligent as it's possible to be. I think

by ggreer 11y ago

> Of course we don't know one way or the other, but it seems to me that it's possible that we're already roughly as intelligent as it's possible to be. I think Nick Bostrom had the perfect reply to that in Superintelligence: Paths, Dangers, Strategies: > Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it. It would be extremely strange if we were near the smartest possible minds. Just look at the evidence: Our fastest neurons send signals at 0.0000004c. Our working memory is smaller than a chimp's.[1] We need pencil and paper to do basic arithmetic. These are not attributes of the pinnacle of possible intelligences. Even if you think it's likely that we are near the smartest possible minds, consider the consequences of being wrong: The AI becomes much smarter than us and potentially destroys everyone and everything we care about. Unless you are supremely confident in humanity's intelligence, you should be concerned about AI risk. 1. https://www.youtube.com/watch?v=zsXP8qeFF6A https://www.youtube.com/watch?v=zsXP8qeFF6A

mindcrime 11y ago
The AI becomes much smarter than us and potentially destroys everyone and everything we care about. Unless you are supremely confident in humanity's intelligence, you should be concerned about AI risk. I acknowledge it as a real risk, but it's not terribly high on my personal list of things to worry about right now. I like what Andrew Ng said about how worrying about this now is "like worrying about over-population on Mars".
- ggreer 11y ago
  Please notice that your reply is a different argument than the one you first put forth. Originally, you weren't worried about AI because you thought it could never –even in principle– vastly exceed human abilities. Now you're basically saying, "I don't need to worry because it won't happen for a long time." That is a huge amount of ground to cede. I'm not so confident that human-level AI will take a long time. The timeline depends on algorithmic insights, which are notoriously difficult to predict. It could be a century. It could be a decade. Still, it seems like something worth worrying about.
  - mindcrime 11y ago
    Please notice that your reply is a different argument than the one you first put forth. Originally, you weren't worried about AI because you thought it could never –even in principle– vastly exceed human abilities. I never said I wasn't worried about AI. You're extrapolating from what I did say; which I've said all along was just a thought experiment, not a position I'm actually arguing for.
    - mattmanser 11y ago
      I really recommend you read Bostrom, he really does succinctly argue the relevant positions, if a bit drily. It's one of those books that put the arguments so clearly you're suddenly catapulted to having a vastly superior knowledge of the subject than someone trying to do simple thought experiments. Both of your arguments look out-dated if you're one of the people 'in the know'. Also, I suggest looking a bit more into what's going on in machine learning, it's suddenly got far more sophisticated than I personally realized until a couple of months ago when I was chatting to someone developing in it at the moment.
      - mindcrime 11y ago
        I actually have the Bostrom book, but just haven't found time to read it yet. But it is definitely in the queue!
- astrofinch 11y ago
  Or like worrying about global warming back when it would have been easier to prevent? Ng's statement is, at best, equivalent to a student who is putting off starting their semester project until finals week. Yes it seems far away, but the future is going to happen.
  - mindcrime 11y ago
    I don't know. I mean, we don't seem to even be close to actually beginning to colonize Mars, much less be close to the point of overpopulation. I think Ng's statement, formed in an analogy similar to yours, would be closer to "a freshman student who is putting off studying for his Senior final project until his Senior year". The question Ng asked was something like "is there any practical action we can take today, to address over-population on mars" as an analogy to "is there any practical step we can take today to address the danger of a super-AGI". And honestly, I'm not convinced there is anything practical to do about super-AGI today. Well, nothing besides pursuing the "open AI" strategy. But I'm willing to be convinced otherwise if somebody has a good argument.
    - astrofinch 11y ago
      Did you have thoughts on the FLI research priorities document? http://futureoflife.org/data/documents/research_priorities.pdf http://futureoflife.org/data/documents/research_priorities.p...
      - mindcrime 11y ago
        I wasn't familiar with that until now, but I'll definitely give it a look.
nradov 11y ago
There are savants who can do amazing feats of mental arithmetic, yet have severe mental disabilities in other areas. Perhaps there are some fundamental limits and trade-offs involved? We don't know yet whether computers will be able to break through those limits. The humans who are in charge of our current society by virtue of having the most wealth, political power, or popularity aren't necessarily the smartest or quickest thinkers, at least not in the way that most AGI researchers seem to be targeting. So even if someone manages to build a real AGI there's no reason to expect it will end up ruling us.
- tedks 11y ago
  >There are savants who can do amazing feats of mental arithmetic, yet have severe mental disabilities in other areas. There are also geniuses who can do amazing feats of mental arithmetic and have no severe mental disabilities in any other areas. There are people who have perfect memories (and usually, they live totally normal lives but for the inability to forget a single event). There are people who are far better than average at recognizing faces, pattern recognition, and other tasks that are non-conscious. The notion that there are "fundamental limits" that are for some reason near the typical human is the cognitive bias of the just-world fallacy. The world is not just. Bad things happen to good people. Good things happen to bad people. If there is a fundamental limit to how smart a mind can get, it's very, very far above the typical human, because there are exceptional humans that are that far away and there's no reason to think a computer couldn't beat them either. Deep Blue beat Kasparov. >The humans who are in charge of our current society by virtue of having the most wealth, political power, or popularity aren't necessarily the smartest or quickest thinkers, at least not in the way that most AGI researchers seem to be targeting. I'm not at all familiar with AGI research, but while there are people who start out ahead, there's no reason to think that you can't actually play the game and win it. Winning challenges of acquiring wealth or political power or popularity are related to intelligence, insofar as the concept of intelligence is defined (as it is at least in psychology) as the ability to accomplish general goals. Walking into a meeting of the Deutsche Arbeiterpartei, joining as the 55th member, and later seizing control of the state requires intelligence. Landing in a dingy yacht with 81 men, bleeding a regime to death, and ruling it until your death requires intelligence. Buying the rights to the Quick and Dirty Operating System, licensing it to IBM, and becoming the de facto standard OS for all consumer computing requires intelligence. Presenting yourself as a folksy Texan when you're a private-school elite from Connecticut and convincing the electorate that you'd have a beer with them requires intelligence. All of these outcomes are goals, and accomplishing them demonstrates an ability to actually put the rubber to the road. You're confusing the technical term of intelligence, which means something along the lines of "ability to accomplish goals," with the social construct of "intelligence" which means something along the lines of "ability to impress people." Intelligence is not always impressive, and humans have a fantastic ability to write off the accomplishments of others when they want to make the world appear more just. I mean, nobody feels good about the fact that we're one Napoleon away from a totally different world system that may or may not suit our interests. The belief that the world is somehow fundamentally different to the extent that the next Napoleon-class intelligence who isn't content managing a hedge fund and just increasing an int in a bank database can't actually redraw the world map, is just an illusion we tell ourselves to make the world we live in more fair, the stories we live out more meaningful, and the events of our lives have a little bit more importance. AI could change things very, very drastically. It's right to be afraid.
visarga 11y ago
> The AI becomes much smarter than us and potentially destroys everyone and everything we care about. What makes you think we humans won't attempt to do even more harm towards humanity? Maybe the AI will save us from ourselves, and, being so much smarter, might guide us towards our further evolution.

by ggreer 11y ago

mindcrime 11y ago

The AI becomes much smarter than us and potentially destroys everyone and everything we care about. Unless you are supremely confident in humanity's intelligence, you should be concerned about AI risk. I acknowledge it as a real risk, but it's not terribly high on my personal list of things to worry about right now. I like what Andrew Ng said about how worrying about this now is "like worrying about over-population on Mars".

ggreer 11y ago
Please notice that your reply is a different argument than the one you first put forth. Originally, you weren't worried about AI because you thought it could never –even in principle– vastly exceed human abilities. Now you're basically saying, "I don't need to worry because it won't happen for a long time." That is a huge amount of ground to cede. I'm not so confident that human-level AI will take a long time. The timeline depends on algorithmic insights, which are notoriously difficult to predict. It could be a century. It could be a decade. Still, it seems like something worth worrying about.
- mindcrime 11y ago
  Please notice that your reply is a different argument than the one you first put forth. Originally, you weren't worried about AI because you thought it could never –even in principle– vastly exceed human abilities. I never said I wasn't worried about AI. You're extrapolating from what I did say; which I've said all along was just a thought experiment, not a position I'm actually arguing for.
  - mattmanser 11y ago
    I really recommend you read Bostrom, he really does succinctly argue the relevant positions, if a bit drily. It's one of those books that put the arguments so clearly you're suddenly catapulted to having a vastly superior knowledge of the subject than someone trying to do simple thought experiments. Both of your arguments look out-dated if you're one of the people 'in the know'. Also, I suggest looking a bit more into what's going on in machine learning, it's suddenly got far more sophisticated than I personally realized until a couple of months ago when I was chatting to someone developing in it at the moment.
    - mindcrime 11y ago
      I actually have the Bostrom book, but just haven't found time to read it yet. But it is definitely in the queue!
astrofinch 11y ago
Or like worrying about global warming back when it would have been easier to prevent? Ng's statement is, at best, equivalent to a student who is putting off starting their semester project until finals week. Yes it seems far away, but the future is going to happen.
- mindcrime 11y ago
  I don't know. I mean, we don't seem to even be close to actually beginning to colonize Mars, much less be close to the point of overpopulation. I think Ng's statement, formed in an analogy similar to yours, would be closer to "a freshman student who is putting off studying for his Senior final project until his Senior year". The question Ng asked was something like "is there any practical action we can take today, to address over-population on mars" as an analogy to "is there any practical step we can take today to address the danger of a super-AGI". And honestly, I'm not convinced there is anything practical to do about super-AGI today. Well, nothing besides pursuing the "open AI" strategy. But I'm willing to be convinced otherwise if somebody has a good argument.
  - astrofinch 11y ago
    Did you have thoughts on the FLI research priorities document? http://futureoflife.org/data/documents/research_priorities.pdf http://futureoflife.org/data/documents/research_priorities.p...
    - mindcrime 11y ago
      I wasn't familiar with that until now, but I'll definitely give it a look.

nradov 11y ago

There are savants who can do amazing feats of mental arithmetic, yet have severe mental disabilities in other areas. Perhaps there are some fundamental limits and trade-offs involved? We don't know yet whether computers will be able to break through those limits. The humans who are in charge of our current society by virtue of having the most wealth, political power, or popularity aren't necessarily the smartest or quickest thinkers, at least not in the way that most AGI researchers seem to be targeting. So even if someone manages to build a real AGI there's no reason to expect it will end up ruling us.

tedks 11y ago
>There are savants who can do amazing feats of mental arithmetic, yet have severe mental disabilities in other areas. There are also geniuses who can do amazing feats of mental arithmetic and have no severe mental disabilities in any other areas. There are people who have perfect memories (and usually, they live totally normal lives but for the inability to forget a single event). There are people who are far better than average at recognizing faces, pattern recognition, and other tasks that are non-conscious. The notion that there are "fundamental limits" that are for some reason near the typical human is the cognitive bias of the just-world fallacy. The world is not just. Bad things happen to good people. Good things happen to bad people. If there is a fundamental limit to how smart a mind can get, it's very, very far above the typical human, because there are exceptional humans that are that far away and there's no reason to think a computer couldn't beat them either. Deep Blue beat Kasparov. >The humans who are in charge of our current society by virtue of having the most wealth, political power, or popularity aren't necessarily the smartest or quickest thinkers, at least not in the way that most AGI researchers seem to be targeting. I'm not at all familiar with AGI research, but while there are people who start out ahead, there's no reason to think that you can't actually play the game and win it. Winning challenges of acquiring wealth or political power or popularity are related to intelligence, insofar as the concept of intelligence is defined (as it is at least in psychology) as the ability to accomplish general goals. Walking into a meeting of the Deutsche Arbeiterpartei, joining as the 55th member, and later seizing control of the state requires intelligence. Landing in a dingy yacht with 81 men, bleeding a regime to death, and ruling it until your death requires intelligence. Buying the rights to the Quick and Dirty Operating System, licensing it to IBM, and becoming the de facto standard OS for all consumer computing requires intelligence. Presenting yourself as a folksy Texan when you're a private-school elite from Connecticut and convincing the electorate that you'd have a beer with them requires intelligence. All of these outcomes are goals, and accomplishing them demonstrates an ability to actually put the rubber to the road. You're confusing the technical term of intelligence, which means something along the lines of "ability to accomplish goals," with the social construct of "intelligence" which means something along the lines of "ability to impress people." Intelligence is not always impressive, and humans have a fantastic ability to write off the accomplishments of others when they want to make the world appear more just. I mean, nobody feels good about the fact that we're one Napoleon away from a totally different world system that may or may not suit our interests. The belief that the world is somehow fundamentally different to the extent that the next Napoleon-class intelligence who isn't content managing a hedge fund and just increasing an int in a bank database can't actually redraw the world map, is just an illusion we tell ourselves to make the world we live in more fair, the stories we live out more meaningful, and the events of our lives have a little bit more importance. AI could change things very, very drastically. It's right to be afraid.

visarga 11y ago

> The AI becomes much smarter than us and potentially destroys everyone and everything we care about. What makes you think we humans won't attempt to do even more harm towards humanity? Maybe the AI will save us from ourselves, and, being so much smarter, might guide us towards our further evolution.