10 ms·

>Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows expo

>Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. I've always wondered: why should recursive intelligence improvement be possible? That is, if you're an agent, you're nominally searching for improved versions of your source-code to self-improve. That's obviously going to be a discrete, combinatorially large search space. Why should each search space have constant or falling entropy, conditional on the agent's existing source code and knowledge? I don't really think "intelligence" can drive the entropy (or the compute-time) of any given search-problem to zero just by existing, so it seems like it should need to draw some resource other than memory space and energy from its environment. This resource is probably information: you would need some knowledge of the world to improve yourself. Successive improvements would then require successively better, finer-grained understandings of the world. That makes sense, but implies that self-improvement becomes more difficult with each round you attempt, since you've already conditioned on most of the environmental information you can get. Entropy can't go to zero, precision can't go to infinity.

naasking 10y ago
> I've always wondered: why should recursive intelligence improvement be possible? That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons. [1] https://en.wikipedia.org/wiki/G%C3%B6del_machine https://en.wikipedia.org/wiki/G%C3%B6del_machine
- eli_gottlieb 10y ago
  >Though theoretically possible, no full implementation has existed before. And it's called a Goedel Machine because it has to prove, within an initial fixed logical language, that its improved program will be globally optimal. Rice's Theorem says this task is impossible in general (provided we stay within the deterministic, logical setting), and Chaitin's Incompleteness Theorem implies it should be increasingly impossible even in specific: the longer a successor program we search for (to incorporate knowledge that makes it smarter), the greater the chance of hitting a sufficiently complex successor candidate that we can't prove its optimality with our existing Goedel Machine program. Again, that's within the paradigm of deterministic logics in which proof-checking is semi-decidable. Real brains don't stick within that paradigm (they're natively probabilistic), and thus AGI probably won't stay within that paradigm either.
  - yarou 10y ago
    Really fascinating stuff. I guess an AGI has to skirt the boundary between the purely rational (i.e. evaluating statements of propositional logic), and irrational (i.e. intuition).
    - kobeya 10y ago
      What do you think intuition is, mechanically?
      - yarou 10y ago
        Probably a hidden Markov process[1]? [1] https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model
        kobeya 10y ago
        The point is more that intuition isn't something opposed to rationality, rather it is a decidedly deterministic logical process that you merely lack conscious insight into.
        yarou 10y ago
        I hadn't ever thought of it that way. I guess it's a false dichotomy. I've been reading up on deep belief networks and Restricted Boltzmann machines. Using these tools definitely reflects a deterministic logical process, there just happens to be hidden state involved.
  - naasking 10y ago
    Exactly, you have to tack on quite a few "provisos" in order to argue it's impossible. There are in fact many ways to circumvent incompleteness upon which every theorem you've quoted depends. Goedel machines have actually been built [1], so their existence isn't just theoretical. [1] http://people.idsia.ch/~juergen/agi2011bas.pdf http://people.idsia.ch/~juergen/agi2011bas.pdf [2] http://people.idsia.ch/~juergen/selfreflection.pdf http://people.idsia.ch/~juergen/selfreflection.pdf Edit: to see another way forward, consider alternate/finitistic arithmetics [3] which are neither stronger nor weaker than Peano arithmetic, but which do not exhibit incompleteness. Arguably, Goedel's theorems pass due to the inherent infinities of the successor axiom, but with an arithmetic like Andrew Boucher's system F (not to be confused with system F in programming language theory), no successor axiom exists, and so we cannot assume that the set of numbers is infinite, and we can enjoy full induction without difficulty. There is still a lot of surprising territory to explore in the foundations of mathematics. [3] https://golem.ph.utexas.edu/category/2011/10/weak_systems_of_arithmetic.html https://golem.ph.utexas.edu/category/2011/10/weak_systems_of...
joe_the_user 10y ago
I've always wondered: why should recursive intelligence improvement be possible? Why does intelligence improvement have to be recursive improvement to the intelligence "source code"? Why can't it simply involve adding more hardware? Maybe the resulting intelligence of putting a million pieces of human level hardware next to each other would only be at the level of million human beings working in close cooperation with perfect communication ability. But that seems pretty powerful. That might not be perfectly omnipotent but it seems like it would be pretty effect, in the fashion the humans are effective and constantly improving themselves, slowly on average, and quickly in the best instances.
- goatlover 10y ago
  Aren't humans already doing this with supercomputers and the internet overall? We can already have a million pieces of hardware communicating. What makes this any more powerful for an AI or uploaded mind? We're already doing it. So either the recursive improvements is already underway, with no need for AGI or uploaded minds, or the future AGI and uploaded minds won't do any better than we are.
- eli_gottlieb 10y ago
  >Why can't it simply involve adding more hardware? Because adding linear amounts of hardware won't give you linear increases in performance. Almost no machine learning algorithms are O(n) in asymptotic complexity. Actually, I can't think of one that is.

naasking 10y ago

> I've always wondered: why should recursive intelligence improvement be possible? That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons. [1] https://en.wikipedia.org/wiki/G%C3%B6del_machine https://en.wikipedia.org/wiki/G%C3%B6del_machine

eli_gottlieb 10y ago
>Though theoretically possible, no full implementation has existed before. And it's called a Goedel Machine because it has to prove, within an initial fixed logical language, that its improved program will be globally optimal. Rice's Theorem says this task is impossible in general (provided we stay within the deterministic, logical setting), and Chaitin's Incompleteness Theorem implies it should be increasingly impossible even in specific: the longer a successor program we search for (to incorporate knowledge that makes it smarter), the greater the chance of hitting a sufficiently complex successor candidate that we can't prove its optimality with our existing Goedel Machine program. Again, that's within the paradigm of deterministic logics in which proof-checking is semi-decidable. Real brains don't stick within that paradigm (they're natively probabilistic), and thus AGI probably won't stay within that paradigm either.
- yarou 10y ago
  Really fascinating stuff. I guess an AGI has to skirt the boundary between the purely rational (i.e. evaluating statements of propositional logic), and irrational (i.e. intuition).
  - kobeya 10y ago
    What do you think intuition is, mechanically?
    - yarou 10y ago
      Probably a hidden Markov process[1]? [1] https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model
      - kobeya 10y ago
        The point is more that intuition isn't something opposed to rationality, rather it is a decidedly deterministic logical process that you merely lack conscious insight into.
        yarou 10y ago
        I hadn't ever thought of it that way. I guess it's a false dichotomy. I've been reading up on deep belief networks and Restricted Boltzmann machines. Using these tools definitely reflects a deterministic logical process, there just happens to be hidden state involved.
- naasking 10y ago
  Exactly, you have to tack on quite a few "provisos" in order to argue it's impossible. There are in fact many ways to circumvent incompleteness upon which every theorem you've quoted depends. Goedel machines have actually been built [1], so their existence isn't just theoretical. [1] http://people.idsia.ch/~juergen/agi2011bas.pdf http://people.idsia.ch/~juergen/agi2011bas.pdf [2] http://people.idsia.ch/~juergen/selfreflection.pdf http://people.idsia.ch/~juergen/selfreflection.pdf Edit: to see another way forward, consider alternate/finitistic arithmetics [3] which are neither stronger nor weaker than Peano arithmetic, but which do not exhibit incompleteness. Arguably, Goedel's theorems pass due to the inherent infinities of the successor axiom, but with an arithmetic like Andrew Boucher's system F (not to be confused with system F in programming language theory), no successor axiom exists, and so we cannot assume that the set of numbers is infinite, and we can enjoy full induction without difficulty. There is still a lot of surprising territory to explore in the foundations of mathematics. [3] https://golem.ph.utexas.edu/category/2011/10/weak_systems_of_arithmetic.html https://golem.ph.utexas.edu/category/2011/10/weak_systems_of...

joe_the_user 10y ago

I've always wondered: why should recursive intelligence improvement be possible? Why does intelligence improvement have to be recursive improvement to the intelligence "source code"? Why can't it simply involve adding more hardware? Maybe the resulting intelligence of putting a million pieces of human level hardware next to each other would only be at the level of million human beings working in close cooperation with perfect communication ability. But that seems pretty powerful. That might not be perfectly omnipotent but it seems like it would be pretty effect, in the fashion the humans are effective and constantly improving themselves, slowly on average, and quickly in the best instances.

goatlover 10y ago
Aren't humans already doing this with supercomputers and the internet overall? We can already have a million pieces of hardware communicating. What makes this any more powerful for an AI or uploaded mind? We're already doing it. So either the recursive improvements is already underway, with no need for AGI or uploaded minds, or the future AGI and uploaded minds won't do any better than we are.
eli_gottlieb 10y ago
>Why can't it simply involve adding more hardware? Because adding linear amounts of hardware won't give you linear increases in performance. Almost no machine learning algorithms are O(n) in asymptotic complexity. Actually, I can't think of one that is.