4 ms·

> It’s wrong because most real-world problems don’t scale linearly. In the real world, the interesting problems are much much harder than that. And this is wha

> It’s wrong because most real-world problems don’t scale linearly. In the real world, the interesting problems are much much harder than that. And this is what has been nagging me about the Singularity and its associated predictions. Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. Futurism, thinking about the future of AI and technology is still important. Long-term thinking, planning, and prediction are always good. But progress will probably be much slower than anticipated.

eli_gottlieb 10y ago
>Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. I've always wondered: why should recursive intelligence improvement be possible? That is, if you're an agent, you're nominally searching for improved versions of your source-code to self-improve. That's obviously going to be a discrete, combinatorially large search space. Why should each search space have constant or falling entropy, conditional on the agent's existing source code and knowledge? I don't really think "intelligence" can drive the entropy (or the compute-time) of any given search-problem to zero just by existing, so it seems like it should need to draw some resource other than memory space and energy from its environment. This resource is probably information: you would need some knowledge of the world to improve yourself. Successive improvements would then require successively better, finer-grained understandings of the world. That makes sense, but implies that self-improvement becomes more difficult with each round you attempt, since you've already conditioned on most of the environmental information you can get. Entropy can't go to zero, precision can't go to infinity.
- naasking 10y ago
  > I've always wondered: why should recursive intelligence improvement be possible? That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons. [1] https://en.wikipedia.org/wiki/G%C3%B6del_machine https://en.wikipedia.org/wiki/G%C3%B6del_machine
  - eli_gottlieb 10y ago
    >Though theoretically possible, no full implementation has existed before. And it's called a Goedel Machine because it has to prove, within an initial fixed logical language, that its improved program will be globally optimal. Rice's Theorem says this task is impossible in general (provided we stay within the deterministic, logical setting), and Chaitin's Incompleteness Theorem implies it should be increasingly impossible even in specific: the longer a successor program we search for (to incorporate knowledge that makes it smarter), the greater the chance of hitting a sufficiently complex successor candidate that we can't prove its optimality with our existing Goedel Machine program. Again, that's within the paradigm of deterministic logics in which proof-checking is semi-decidable. Real brains don't stick within that paradigm (they're natively probabilistic), and thus AGI probably won't stay within that paradigm either.
    - yarou 10y ago
      Really fascinating stuff. I guess an AGI has to skirt the boundary between the purely rational (i.e. evaluating statements of propositional logic), and irrational (i.e. intuition).
      - kobeya 10y ago
        What do you think intuition is, mechanically?
        yarou 10y ago
        Probably a hidden Markov process[1]? [1] https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model
        kobeya 10y ago
        The point is more that intuition isn't something opposed to rationality, rather it is a decidedly deterministic logical process that you merely lack conscious insight into.
- joe_the_user 10y ago
  I've always wondered: why should recursive intelligence improvement be possible? Why does intelligence improvement have to be recursive improvement to the intelligence "source code"? Why can't it simply involve adding more hardware? Maybe the resulting intelligence of putting a million pieces of human level hardware next to each other would only be at the level of million human beings working in close cooperation with perfect communication ability. But that seems pretty powerful. That might not be perfectly omnipotent but it seems like it would be pretty effect, in the fashion the humans are effective and constantly improving themselves, slowly on average, and quickly in the best instances.
  - goatlover 10y ago
    Aren't humans already doing this with supercomputers and the internet overall? We can already have a million pieces of hardware communicating. What makes this any more powerful for an AI or uploaded mind? We're already doing it. So either the recursive improvements is already underway, with no need for AGI or uploaded minds, or the future AGI and uploaded minds won't do any better than we are.
  - eli_gottlieb 10y ago
    >Why can't it simply involve adding more hardware? Because adding linear amounts of hardware won't give you linear increases in performance. Almost no machine learning algorithms are O(n) in asymptotic complexity. Actually, I can't think of one that is.
ThomPete 10y ago
Think about it differently. The hard problems are going to be increasingly easier and easier to solve because they themselves consists of simpler problems. What technology is really good at is to share it's findings across the entire network. What one learn every other entity now have access to. The mistake in dismissing the thinking is in believing that complex problems are irreducible complex, they are not. They are simply the layer of ever more simple problems. Thats how we came to create the complexity and the problems that arose from it.
- daveguy 10y ago
  > The mistake in dismissing the thinking is in believing that complex problems are irreducible complex, they are not. This is almost certainly false. Current state of the art in complexity theory is that some complex problems are irreducible. The expectation is that P != NP and that means there are irreducible problems. Assuming that P=NP is almost, but not quite, like assuming perpetual motion.
  - ThomPete 10y ago
    Sure but thats not what we are discussing here. We are discussing things in everyday live not mathematical complexity. The claim was that the interesting problems are much more complex and thus can't be solved easily by machine learning.My point is that most of these complex problems aren't really anything but simpler problems gaining complexity by connection of simpler problems not by being irreducible. Any engineer will tell you thats exactly how you solve complex problem, by breaking them into smaller ones.
    - dkarapetyan 10y ago
      The way we model the world is with mathematical models and the complexity bounds on those mathematical models are actually statements about irreducibility of those problems to simpler ones. In fact, the model is lower bound approximation to the real thing and if the approximation has certain complexity then the real thing will be much worse. That is the whole point of having a model. You approximate reality with something simpler and hope the solution to the simpler model problem is close enough to the solution of the real problem. So what real world problems are you talking about? I'd say much of engineering and science is coming up with these solvable models for real world phenomena and so far no one has cracked the nut on coming up with a simple solution to problems that have complicated models. You might argue AGI is not one of those problems but that's a different argument.

eli_gottlieb 10y ago

>Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress. I've always wondered: why should recursive intelligence improvement be possible? That is, if you're an agent, you're nominally searching for improved versions of your source-code to self-improve. That's obviously going to be a discrete, combinatorially large search space. Why should each search space have constant or falling entropy, conditional on the agent's existing source code and knowledge? I don't really think "intelligence" can drive the entropy (or the compute-time) of any given search-problem to zero just by existing, so it seems like it should need to draw some resource other than memory space and energy from its environment. This resource is probably information: you would need some knowledge of the world to improve yourself. Successive improvements would then require successively better, finer-grained understandings of the world. That makes sense, but implies that self-improvement becomes more difficult with each round you attempt, since you've already conditioned on most of the environmental information you can get. Entropy can't go to zero, precision can't go to infinity.

naasking 10y ago
> I've always wondered: why should recursive intelligence improvement be possible? That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons. [1] https://en.wikipedia.org/wiki/G%C3%B6del_machine https://en.wikipedia.org/wiki/G%C3%B6del_machine
- eli_gottlieb 10y ago
  >Though theoretically possible, no full implementation has existed before. And it's called a Goedel Machine because it has to prove, within an initial fixed logical language, that its improved program will be globally optimal. Rice's Theorem says this task is impossible in general (provided we stay within the deterministic, logical setting), and Chaitin's Incompleteness Theorem implies it should be increasingly impossible even in specific: the longer a successor program we search for (to incorporate knowledge that makes it smarter), the greater the chance of hitting a sufficiently complex successor candidate that we can't prove its optimality with our existing Goedel Machine program. Again, that's within the paradigm of deterministic logics in which proof-checking is semi-decidable. Real brains don't stick within that paradigm (they're natively probabilistic), and thus AGI probably won't stay within that paradigm either.
  - yarou 10y ago
    Really fascinating stuff. I guess an AGI has to skirt the boundary between the purely rational (i.e. evaluating statements of propositional logic), and irrational (i.e. intuition).
    - kobeya 10y ago
      What do you think intuition is, mechanically?
      - yarou 10y ago
        Probably a hidden Markov process[1]? [1] https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model
        kobeya 10y ago
        The point is more that intuition isn't something opposed to rationality, rather it is a decidedly deterministic logical process that you merely lack conscious insight into.
joe_the_user 10y ago
I've always wondered: why should recursive intelligence improvement be possible? Why does intelligence improvement have to be recursive improvement to the intelligence "source code"? Why can't it simply involve adding more hardware? Maybe the resulting intelligence of putting a million pieces of human level hardware next to each other would only be at the level of million human beings working in close cooperation with perfect communication ability. But that seems pretty powerful. That might not be perfectly omnipotent but it seems like it would be pretty effect, in the fashion the humans are effective and constantly improving themselves, slowly on average, and quickly in the best instances.
- goatlover 10y ago
  Aren't humans already doing this with supercomputers and the internet overall? We can already have a million pieces of hardware communicating. What makes this any more powerful for an AI or uploaded mind? We're already doing it. So either the recursive improvements is already underway, with no need for AGI or uploaded minds, or the future AGI and uploaded minds won't do any better than we are.
- eli_gottlieb 10y ago
  >Why can't it simply involve adding more hardware? Because adding linear amounts of hardware won't give you linear increases in performance. Almost no machine learning algorithms are O(n) in asymptotic complexity. Actually, I can't think of one that is.

ThomPete 10y ago

Think about it differently. The hard problems are going to be increasingly easier and easier to solve because they themselves consists of simpler problems. What technology is really good at is to share it's findings across the entire network. What one learn every other entity now have access to. The mistake in dismissing the thinking is in believing that complex problems are irreducible complex, they are not. They are simply the layer of ever more simple problems. Thats how we came to create the complexity and the problems that arose from it.

daveguy 10y ago
> The mistake in dismissing the thinking is in believing that complex problems are irreducible complex, they are not. This is almost certainly false. Current state of the art in complexity theory is that some complex problems are irreducible. The expectation is that P != NP and that means there are irreducible problems. Assuming that P=NP is almost, but not quite, like assuming perpetual motion.
- ThomPete 10y ago
  Sure but thats not what we are discussing here. We are discussing things in everyday live not mathematical complexity. The claim was that the interesting problems are much more complex and thus can't be solved easily by machine learning.My point is that most of these complex problems aren't really anything but simpler problems gaining complexity by connection of simpler problems not by being irreducible. Any engineer will tell you thats exactly how you solve complex problem, by breaking them into smaller ones.
  - dkarapetyan 10y ago
    The way we model the world is with mathematical models and the complexity bounds on those mathematical models are actually statements about irreducibility of those problems to simpler ones. In fact, the model is lower bound approximation to the real thing and if the approximation has certain complexity then the real thing will be much worse. That is the whole point of having a model. You approximate reality with something simpler and hope the solution to the simpler model problem is close enough to the solution of the real problem. So what real world problems are you talking about? I'd say much of engineering and science is coming up with these solvable models for real world phenomena and so far no one has cracked the nut on coming up with a simple solution to problems that have complicated models. You might argue AGI is not one of those problems but that's a different argument.