4 ms·

A little bit in the same vein (arguing against the singularity): "The impossibility of intelligence explosion" https://medium.com/@francois.chollet/the-impossi

A little bit in the same vein (arguing against the singularity): "The impossibility of intelligence explosion" https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec https://medium.com/@francois.chollet/the-impossibility-of-in...

pas 8y ago
Thanks for the link! > If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, [...] > There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news. We measure IQ with pretty general pattern matching. An IQ of 150 is interesting, but those people are very very close to us. If we are talking about human specific tasks we should look at how IQ correlates with let's say playing poker, performing well on X factor, etc. IQ predicts how well someone will earn money in a very vague sense, but maybe there are better tests to know if someone will be able to persuade humans. (Do we have data about the IQ of fraudsters? Politicians? Bible/tupperware sales agents?) > Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment. It might be true, but that's largely because humans can't self-modify their brains. And the fundamental problem is we can't even modify our minds at will. We can sure train ourselves, but we can't fundamentally alter our minds, because it's too tied to our brains. > If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world Or simply 150 IQ is meaninglessly low. It's still in the human envelope. (Since we know there are 7 million people with IQ of 150.) What happens around 300? Or around 500? That's so far out of the human range (depending on how the IQ test is calibrated, 200 IQ is one in ~4.8 billion for the 16SD one: https://www.iqcomparisonsite.com/iqtable.aspx https://www.iqcomparisonsite.com/iqtable.aspx ) These essays arguing against the possibility somehow forget that there are things we would benefit from, a lot. Simple faculties of the mind we are missing. Like a great alarm clock, a simple to-do list in your mind. (Sure you can train yourself to not forget things and wake up at specific times, but it's not like downloading a perfect alarm/todo app.) Working memory. We're at around 6 or 7. And it's very hard to push it up even by one in general. ( But it's possible to kind of try and train yourself to do this, which seems to help with working memory: https://www.gwern.net/DNB-FAQ#does-it-really-work https://www.gwern.net/DNB-FAQ#does-it-really-work ) Also, it's important to note, that that people who wanted to take over the world were usually very disturbed individuals. The human psyche is not really well suited to senseless killing while also perfectly simulating empathy and other things. (Of course psychopaths are a real thing.) > Most of our intelligence is not in our brain, it is externalized as our civilization And that's the point that should really scare people. Because we can't cram all that into our heads, but a human level AI, being software connected to a lot of storage, can. That said, it's understandable that we can't understand the how of the intelligence explosion. That's basically the number one premise of the whole thing. But we can't simply disprove it. Any kind of proof that is so sloppy that simply throws around a formula like "because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1`" can be ignored. Because we are not interested in a general complex system, we are interested in an AI that approximates that explosion. It might be linear with a very-very steep slope, or it can be just a nice parabola. We're still fucked. > Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks. This is a very strong point. That said, an AI can work around the bottlenecks. With self-modification nothing prevents it from reinventing itself from scratch every time it encounters a bottleneck. Diminishing returns are of course a problem for it. > scientific progress, because it is conceptually very close to intelligence itself — science, as a problem-solving system, is very close to being a runaway superhuman AI. Yes, and no. Because the components are still very much "just" humans. And that's the bottleneck. And we can't reinvent science from the ground up to replace that bottleneck. (Yet.) > What bottlenecks [...] are slowing down recursive self-improvement [..]? So many, I can’t even count them. Here are a few [, that] would also apply to recursively self-improving AIs. Let's see them! > It gets increasingly harder to keep up with the firehose of new publications. Yes, but no. This means if the AI can represent and access knowledge efficiently, then it'll be able to handle it, just like Google is perfectly capable of handling a growing World Wide Web. > As scientific knowledge expands, the time and effort that have to be invested in education and training grows, and the field of inquiry of individual researchers gets increasingly narrow. Because we can't do direct mind to mind knowledge transfer. Which is again something AIs don't really have as a constraint. > conclusion remember, if everything goes according to [my] plan, then we don't have to worry. so don't worry. I mean, okay, sure, but WTF.
xamuel 8y ago
In that article, the author claims that a human brain cannot create something more intelligent than itself, and argues by saying that over several billion years of evolution, no human brain has ever done so, thus it must not be possible. That argument is totally nonsense. A hundred years ago, you could argue that humans cannot invent an internet, because over billions of years (minus one hundred), it's never happened. There's a more rigorous way to argue for the claim in question, which debuted in my doctoral dissertation. Or, not quite the exact claim in question, but rather the following claim: that an idealized mechanical knowing agent cannot create a more intelligent idealized mechanical knowing agent than itself (at least, not one which it can actually trust). The idea behind the proof is quite simple, but requires familiarity with computable ordinal numbers. If knowing agent X creates knowing agent Y, then presumably X knows Y's sourcecode. Thus, X can infer a sourcecode for the list of all naturals n such that Y knows "n is the code of a computable ordinal". If X actually trusts Y, then X knows that whenever Y knows "n is the code of a computable ordinal", then n really is the code of a computable ordinal. Combining these, X can infer the ordinality of the ordering obtained by concatenating together all the ordinals which Y knows are ordinals. Thus, X knows a larger computable ordinal than all the computable ordinals Y knows. Which, depending on your definition of intelligence, arguably suggests X is more intelligent than Y.

pas 8y ago

Thanks for the link! > If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, [...] > There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news. We measure IQ with pretty general pattern matching. An IQ of 150 is interesting, but those people are very very close to us. If we are talking about human specific tasks we should look at how IQ correlates with let's say playing poker, performing well on X factor, etc. IQ predicts how well someone will earn money in a very vague sense, but maybe there are better tests to know if someone will be able to persuade humans. (Do we have data about the IQ of fraudsters? Politicians? Bible/tupperware sales agents?) > Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment. It might be true, but that's largely because humans can't self-modify their brains. And the fundamental problem is we can't even modify our minds at will. We can sure train ourselves, but we can't fundamentally alter our minds, because it's too tied to our brains. > If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world Or simply 150 IQ is meaninglessly low. It's still in the human envelope. (Since we know there are 7 million people with IQ of 150.) What happens around 300? Or around 500? That's so far out of the human range (depending on how the IQ test is calibrated, 200 IQ is one in ~4.8 billion for the 16SD one: https://www.iqcomparisonsite.com/iqtable.aspx https://www.iqcomparisonsite.com/iqtable.aspx ) These essays arguing against the possibility somehow forget that there are things we would benefit from, a lot. Simple faculties of the mind we are missing. Like a great alarm clock, a simple to-do list in your mind. (Sure you can train yourself to not forget things and wake up at specific times, but it's not like downloading a perfect alarm/todo app.) Working memory. We're at around 6 or 7. And it's very hard to push it up even by one in general. ( But it's possible to kind of try and train yourself to do this, which seems to help with working memory: https://www.gwern.net/DNB-FAQ#does-it-really-work https://www.gwern.net/DNB-FAQ#does-it-really-work ) Also, it's important to note, that that people who wanted to take over the world were usually very disturbed individuals. The human psyche is not really well suited to senseless killing while also perfectly simulating empathy and other things. (Of course psychopaths are a real thing.) > Most of our intelligence is not in our brain, it is externalized as our civilization And that's the point that should really scare people. Because we can't cram all that into our heads, but a human level AI, being software connected to a lot of storage, can. That said, it's understandable that we can't understand the how of the intelligence explosion. That's basically the number one premise of the whole thing. But we can't simply disprove it. Any kind of proof that is so sloppy that simply throws around a formula like "because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1`" can be ignored. Because we are not interested in a general complex system, we are interested in an AI that approximates that explosion. It might be linear with a very-very steep slope, or it can be just a nice parabola. We're still fucked. > Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks. This is a very strong point. That said, an AI can work around the bottlenecks. With self-modification nothing prevents it from reinventing itself from scratch every time it encounters a bottleneck. Diminishing returns are of course a problem for it. > scientific progress, because it is conceptually very close to intelligence itself — science, as a problem-solving system, is very close to being a runaway superhuman AI. Yes, and no. Because the components are still very much "just" humans. And that's the bottleneck. And we can't reinvent science from the ground up to replace that bottleneck. (Yet.) > What bottlenecks [...] are slowing down recursive self-improvement [..]? So many, I can’t even count them. Here are a few [, that] would also apply to recursively self-improving AIs. Let's see them! > It gets increasingly harder to keep up with the firehose of new publications. Yes, but no. This means if the AI can represent and access knowledge efficiently, then it'll be able to handle it, just like Google is perfectly capable of handling a growing World Wide Web. > As scientific knowledge expands, the time and effort that have to be invested in education and training grows, and the field of inquiry of individual researchers gets increasingly narrow. Because we can't do direct mind to mind knowledge transfer. Which is again something AIs don't really have as a constraint. > conclusion remember, if everything goes according to [my] plan, then we don't have to worry. so don't worry. I mean, okay, sure, but WTF.

xamuel 8y ago

In that article, the author claims that a human brain cannot create something more intelligent than itself, and argues by saying that over several billion years of evolution, no human brain has ever done so, thus it must not be possible. That argument is totally nonsense. A hundred years ago, you could argue that humans cannot invent an internet, because over billions of years (minus one hundred), it's never happened. There's a more rigorous way to argue for the claim in question, which debuted in my doctoral dissertation. Or, not quite the exact claim in question, but rather the following claim: that an idealized mechanical knowing agent cannot create a more intelligent idealized mechanical knowing agent than itself (at least, not one which it can actually trust). The idea behind the proof is quite simple, but requires familiarity with computable ordinal numbers. If knowing agent X creates knowing agent Y, then presumably X knows Y's sourcecode. Thus, X can infer a sourcecode for the list of all naturals n such that Y knows "n is the code of a computable ordinal". If X actually trusts Y, then X knows that whenever Y knows "n is the code of a computable ordinal", then n really is the code of a computable ordinal. Combining these, X can infer the ordinality of the ordering obtained by concatenating together all the ordinals which Y knows are ordinals. Thus, X knows a larger computable ordinal than all the computable ordinals Y knows. Which, depending on your definition of intelligence, arguably suggests X is more intelligent than Y.