4 ms·

“Intelligence is situational — there is no such thing as general intelligence. Your brain is one piece in a broader system which includes your body, your enviro

by jp555 6y ago

“Intelligence is situational — there is no such thing as general intelligence. Your brain is one piece in a broader system which includes your body, your environment, other humans, and culture as a whole. […] Currently, our environment, not our brain, is acting as the bottleneck to our intelligence.” “Human intelligence is largely externalized, contained not in our brain but in our civilization. We are our tools — our brains are modules in a cognitive system much larger than ourselves. A system that is already self-improving, and has been for a long time.” “Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions […], cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.” “Recursive intelligence expansion is already happening — at the level of our civilization. It will keep happening in the age of AI, and it progresses at a roughly linear pace.“ François Chollet https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec https://medium.com/@francois.chollet/the-impossibility-of-in...

deleted 6y ago
[deleted]
Eliezer 6y ago
Here's an extensive reply to Chollet's essay, also by the author of "There's No Fire Alarm": https://intelligence.org/2017/12/06/chollet/ https://intelligence.org/2017/12/06/chollet/ > ...some systems function very well in a broad variety of structured low-entropy environments. E.g. the human brain functions much better than other primate brains in an extremely broad set of environments, including many that natural selection did not explicitly optimize for. We remain functional on the Moon, because the Moon has enough in common with the Earth on a sufficiently deep meta-level that, for example, induction on past experience goes on functioning there. >> The intelligence of an octopus is specialized in the problem of being an octopus. The intelligence of a human is specialized in the problem of being human. > The problem that a human solves is much more general than the problem an octopus solves, which is why we can walk on the Moon and the octopus can’t. >> Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions arising from the broader context in which they exist, cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement. > Falsified by a graph of world GDP on almost any timescale.
- pavlov 6y ago
  Any graph of world GDP is meaninglessly short on the timescale of the Earth’s ecosystem. What looks like unstoppable success to us could be a blip in the broader context.
  - square_usual 6y ago
    That is not a "practical" time scale though. Sure, if we wait long enough, GDP might flatten out, but on human time scales so far it has grown exponentially. A similar analogy can be made for AGI.
- jkhdigital 6y ago
  Still plenty of time for GDP to turn into a sigmoid
  - pjscott 6y ago
    Sure, and if we're very lucky it'll eventually go quadratic as our species expands out into the universe in an ever-widening sphere. :-) Few things can grow exponentially forever, but a lot of things can grow exponentially for long enough to have huge consequences -- and that's usually what we care about.
- loopz 6y ago
  GDP is not an absolute measure.
jkhdigital 6y ago
Surprised he didn't mention the Flynn effect: https://en.wikipedia.org/wiki/Flynn_effect https://en.wikipedia.org/wiki/Flynn_effect A book I recently read (Range, by David Epstein) seems to draw a connection through some other research that points to modernization and globalization as the drivers of this trend, in the sense that IQ tests measure pure abstract reasoning ability and we now live in a world dominated by abstraction so our brains get used to thinking about all of our experiences abstractly from a younger age. Basically, each successive generation has more practice in thinking about the world abstractly since it is required use to modern technology effectively. Anyways, this seems to buttress what Chollet is saying about externalized human intelligence. It is difficult to imagine anything resembling a "general" intelligence that isn't attached to a life-form that is already a self-replicating engine of negentropy, bootstrapping solutions to hard problems through some form of non-volatile memory. I would almost go so far as to say that "intelligence" and evolution are inseparable.
- jbay808 6y ago
  > It is difficult to imagine anything resembling a "general" intelligence that isn't attached to a life-form[...] How is that argument any stronger than these ones that (non-animal) heavier than air flight is impossible? https://www.xaprb.com/blog/flight-is-impossible/ https://www.xaprb.com/blog/flight-is-impossible/ I think these are actually stronger arguments, because I can think of at least three very least significant material-science advantages that birds have even over modern technology that would, from a physics point of view, let me entertain the idea that a 19th century physicist could believe you'd at least need to be made of flesh to fly. I can't think of any comparable case for intelligence being restricted to biology.
  - jkhdigital 6y ago
    Yes, I realize that the phrase "difficult to imagine" is fun to pounce on. There's a big difference between "flight" and "general intelligence"; namely that the former is like Justice Potter Stewart's definition of pornography and the latter is... not. Anyways, maybe what I'm suggesting is that if you can point to something and say "that is a (artificial?) general intelligence", then I suspect you will also feel compelled to say "that is (artificial?) life". Definitely wandering off into the weeds here, but why not: I see evolution as a computational problem. In its most primitive, there is no computer, but any specific problem can still be solved with luck and memory. If you have a structure that recognizes a solution, then you just wait until you randomly bump into it. The challenges of evolution in a given environment are specific to that environment, and thus memory without a computer can go a really, really long way. If human intelligence is general intelligence, it is still the logical progression of this chain of memory and (later on) compute operations. I can reason about the abstraction of general intelligence, but it feels like any concrete general intelligence will proceed through essentially the same steps. The general intelligence must reside in a concrete existence, and whatever form this takes I feel very strongly that it will be recognizable as what we call "life".
dntbnmpls 6y ago
> Intelligence is situational — there is no such thing as general intelligence. Not sure about that. But certainly human intelligence is far more general than goldfish intelligence similar to how a turing machine is more general than a finite state automata. > Your brain is one piece in a broader system which includes your body, your environment, other humans, and culture as a whole. I could lose an arm and still be as intelligent. Stephen Hawking lost his ability to walk and he was still as intelligent. Turing could move to Princeton from england and still maintain his intelligence. Sure your body, environment, culture, etc helps you develop and informs who are, but certainly it is the brain that ultimately matters. > “Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions […], cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.” But evolution and societies doesn't work in a "recursively self-improving" manner. I'm rather partial to the idea of punctuated equilibrium. Momentum builds and we have a sudden exponential progress until we arrive at a new normal. Evolution and societal progress isn't that of continuous linear or exponential growth. Rather it's a series of peaks and valleys. > “Recursive intelligence expansion is already happening — at the level of our civilization. It will keep happening in the age of AI, and it progresses at a roughly linear pace.“ I suppose but historically this isn't true. No human society or civilization progressed at a roughly linear pace. There are periods of stagnation or even decay followed by burst of creativity and progress. Otherwise, all societies/civilizations would remain relatively equal to each other. How does linear progress explain the sudden rise of europe in the last 500 years to dominate the globe? Or the sudden rise of french as lingua franca and the equally sudden rise of english? The sudden economic growth with the discovery of oil. Or the sudden burst of human population? https://ourworldindata.org/grapher/world-population-since-10000-bce-ourworldindata-series https://ourworldindata.org/grapher/world-population-since-10...
Zaak 6y ago
> “Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions […], cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.” Moore's Law. Use computers to make better computers. Exponential growth over more than 7 orders of magnitude and, although the growth rate is slowing, it hasn't run out of steam yet. If AI eventually exhibits anywhere near that level of recursive self-improvement, godlike superintelligences lie in our future.
- jp555 6y ago
  Moore’s Law is an S-curve too. Its already approaching its asymptote - accelerating diminishing returns.

by jp555 6y ago

deleted 6y ago

[deleted]

Eliezer 6y ago

Here's an extensive reply to Chollet's essay, also by the author of "There's No Fire Alarm": https://intelligence.org/2017/12/06/chollet/ https://intelligence.org/2017/12/06/chollet/ > ...some systems function very well in a broad variety of structured low-entropy environments. E.g. the human brain functions much better than other primate brains in an extremely broad set of environments, including many that natural selection did not explicitly optimize for. We remain functional on the Moon, because the Moon has enough in common with the Earth on a sufficiently deep meta-level that, for example, induction on past experience goes on functioning there. >> The intelligence of an octopus is specialized in the problem of being an octopus. The intelligence of a human is specialized in the problem of being human. > The problem that a human solves is much more general than the problem an octopus solves, which is why we can walk on the Moon and the octopus can’t. >> Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions arising from the broader context in which they exist, cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement. > Falsified by a graph of world GDP on almost any timescale.

pavlov 6y ago
Any graph of world GDP is meaninglessly short on the timescale of the Earth’s ecosystem. What looks like unstoppable success to us could be a blip in the broader context.
- square_usual 6y ago
  That is not a "practical" time scale though. Sure, if we wait long enough, GDP might flatten out, but on human time scales so far it has grown exponentially. A similar analogy can be made for AGI.
jkhdigital 6y ago
Still plenty of time for GDP to turn into a sigmoid
- pjscott 6y ago
  Sure, and if we're very lucky it'll eventually go quadratic as our species expands out into the universe in an ever-widening sphere. :-) Few things can grow exponentially forever, but a lot of things can grow exponentially for long enough to have huge consequences -- and that's usually what we care about.
loopz 6y ago
GDP is not an absolute measure.

jkhdigital 6y ago

Surprised he didn't mention the Flynn effect: https://en.wikipedia.org/wiki/Flynn_effect https://en.wikipedia.org/wiki/Flynn_effect A book I recently read (Range, by David Epstein) seems to draw a connection through some other research that points to modernization and globalization as the drivers of this trend, in the sense that IQ tests measure pure abstract reasoning ability and we now live in a world dominated by abstraction so our brains get used to thinking about all of our experiences abstractly from a younger age. Basically, each successive generation has more practice in thinking about the world abstractly since it is required use to modern technology effectively. Anyways, this seems to buttress what Chollet is saying about externalized human intelligence. It is difficult to imagine anything resembling a "general" intelligence that isn't attached to a life-form that is already a self-replicating engine of negentropy, bootstrapping solutions to hard problems through some form of non-volatile memory. I would almost go so far as to say that "intelligence" and evolution are inseparable.

jbay808 6y ago
> It is difficult to imagine anything resembling a "general" intelligence that isn't attached to a life-form[...] How is that argument any stronger than these ones that (non-animal) heavier than air flight is impossible? https://www.xaprb.com/blog/flight-is-impossible/ https://www.xaprb.com/blog/flight-is-impossible/ I think these are actually stronger arguments, because I can think of at least three very least significant material-science advantages that birds have even over modern technology that would, from a physics point of view, let me entertain the idea that a 19th century physicist could believe you'd at least need to be made of flesh to fly. I can't think of any comparable case for intelligence being restricted to biology.
- jkhdigital 6y ago
  Yes, I realize that the phrase "difficult to imagine" is fun to pounce on. There's a big difference between "flight" and "general intelligence"; namely that the former is like Justice Potter Stewart's definition of pornography and the latter is... not. Anyways, maybe what I'm suggesting is that if you can point to something and say "that is a (artificial?) general intelligence", then I suspect you will also feel compelled to say "that is (artificial?) life". Definitely wandering off into the weeds here, but why not: I see evolution as a computational problem. In its most primitive, there is no computer, but any specific problem can still be solved with luck and memory. If you have a structure that recognizes a solution, then you just wait until you randomly bump into it. The challenges of evolution in a given environment are specific to that environment, and thus memory without a computer can go a really, really long way. If human intelligence is general intelligence, it is still the logical progression of this chain of memory and (later on) compute operations. I can reason about the abstraction of general intelligence, but it feels like any concrete general intelligence will proceed through essentially the same steps. The general intelligence must reside in a concrete existence, and whatever form this takes I feel very strongly that it will be recognizable as what we call "life".

dntbnmpls 6y ago

> Intelligence is situational — there is no such thing as general intelligence. Not sure about that. But certainly human intelligence is far more general than goldfish intelligence similar to how a turing machine is more general than a finite state automata. > Your brain is one piece in a broader system which includes your body, your environment, other humans, and culture as a whole. I could lose an arm and still be as intelligent. Stephen Hawking lost his ability to walk and he was still as intelligent. Turing could move to Princeton from england and still maintain his intelligence. Sure your body, environment, culture, etc helps you develop and informs who are, but certainly it is the brain that ultimately matters. > “Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions […], cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.” But evolution and societies doesn't work in a "recursively self-improving" manner. I'm rather partial to the idea of punctuated equilibrium. Momentum builds and we have a sudden exponential progress until we arrive at a new normal. Evolution and societal progress isn't that of continuous linear or exponential growth. Rather it's a series of peaks and valleys. > “Recursive intelligence expansion is already happening — at the level of our civilization. It will keep happening in the age of AI, and it progresses at a roughly linear pace.“ I suppose but historically this isn't true. No human society or civilization progressed at a roughly linear pace. There are periods of stagnation or even decay followed by burst of creativity and progress. Otherwise, all societies/civilizations would remain relatively equal to each other. How does linear progress explain the sudden rise of europe in the last 500 years to dominate the globe? Or the sudden rise of french as lingua franca and the equally sudden rise of english? The sudden economic growth with the discovery of oil. Or the sudden burst of human population? https://ourworldindata.org/grapher/world-population-since-10000-bce-ourworldindata-series https://ourworldindata.org/grapher/world-population-since-10...

Zaak 6y ago

> “Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions […], cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.” Moore's Law. Use computers to make better computers. Exponential growth over more than 7 orders of magnitude and, although the growth rate is slowing, it hasn't run out of steam yet. If AI eventually exhibits anywhere near that level of recursive self-improvement, godlike superintelligences lie in our future.

jp555 6y ago
Moore’s Law is an S-curve too. Its already approaching its asymptote - accelerating diminishing returns.