2 ms·

The field isn't going to get trapped in a local minimum with neural nets because the field is too big now. When AI was 20 people each at Stanford, MIT, and CMU,

by maldeh 10y ago

The field isn't going to get trapped in a local minimum with neural nets because the field is too big now. When AI was 20 people each at Stanford, MIT, and CMU, that could happen. With 50,000 people taking machine learning courses, there are enough people for some to focus on optimizing existing technologies without taking away from new ideas. Riffing off the OP's argument of AI as gradient descent, one could say the 50,000 new people aren't necessarily doing purely random-restart hillclimbing so much as sequential monte carlo, sampling around a distribution laid out by the main pillars of the ML community. That is to say, it should only be robust against weak local minima. Hopefully we should be self-aware enough to know when we're trapped in a strong one, and await a true random-restart.

ewjordan 10y ago
No, we don't even need to be very clever: evolution is no better able to evade weak local minima than any but the dumbest search strategies humans use, and it quite easily discovered a neural algorithm that lead to intelligence when scaled sufficiently.
- legutierr 10y ago
  Easily? It took 4 billion years and multiple global extinction events, and it has only happened once. Flight, the eye, air breathing all evolved multiple times independently, within distantly related evolutionary lines. Intelligence is rare in nature, in spite of its advantages, probably because it is a very distant jump from local maxima.
  - vintermann 10y ago
    Do you define human intelligence as the only real intelligence, then? Or are you talking about some super-early primitive predecessor of a brain? (I'm sorry, I have no idea whether the latter has only evolved once.)
    - legutierr 10y ago
      That's the real question though, isn't it? I think what we are talking about is the kind of intelligence that could qualify as a "general AI" if implemented in a computer. Although I'm sure that there many different types of intelligence that could match up, in nature there really is only one that we know about.
      - vintermann 10y ago
        I don't think such a thing as general AI, or even general intelligence, is sensible to talk about. There are always tradeoffs: an algorithm good at finding some sort of patterns, necessarily has to be worse at finding other sorts of patterns. There are actually ways to formalize and prove that, but it's intuitively obvious that an algorithm that can find the answer to a set of questions, can't be faster than an algorithm to find the answer to a subset of those questions.
  - adrianN 10y ago
    Cephalopods are reasonably smart and their intelligence evolved mostly idependently. That's reassuring. https://en.wikipedia.org/wiki/Cephalopod_intelligence https://en.wikipedia.org/wiki/Cephalopod_intelligence
- chriswarbo 10y ago
  > when scaled sufficiently Whilst technically correct, this obscures the truly collosal scale at which evolution computes. If we consider each organism as a processor executing a single fitness observation, we can fit hundreds of thousands of such processors (e.g. bacteria) on the few square millimetres of a pin head. These processors are spread (less densely) across the 510 million square kilometre surface of the Earth. They are suspended in oceans, which have an average depth of 3.7 kilometres; and in the atmosphere, at least up to 10 kilometres (based on our observations so far). Whilst the generation time (AKA execution time) of such organisms varies wildly, it is very commonly less than an hour. Yet even with these truly staggering resources at its disposal, evolution still took 3.3 billion years of wall clock time to come up with multicellular organisms. Relying on evolution to come up with meta-level optimisation mechanisms like intelligence is also rather flaky; we only have one example of biological evolution to study, acting on one representation (DNA + RNA + proteins). This just-so-happened to stumble upon aerobic respiration, multicellularity, neurons and brains, but it's certainly not a "goal" of the system, and sampling bias prevents us knowing how likely an outcome that is (i.e. whether it's an attractor). Still, organisms with nervous systems are a vanishingly small proportion of all lifeforms; and in fact, every "advance" is dwarfed by the success of single celled microbes. We could always leave an evolving system running for longer, but setting up conditions which lead to open-ended evolution is still an unknown area; simulations always tend to level off/saturate at some point. As a "last resort" we could provide the system with a mechanism to sample organisms completely at random, and use that as proof that anything is possible given enough time; but in that case why use evolution at all, when we can just perform that sampling directly, or alternatively just enumerate organisms?
  - ewjordan 10y ago
    I nodded along with pretty much everything you said, so I think we're arguing different points but not necessarily disagreeing. I agree, evolution's scale is colossal, but restrict your view to the tiny bit of the evolutionary machine that is implicated in figuring out how to arrange neurons (which evolution "discovered" previously, to be fair, with incredible effort) in a fashion that renders their behavior intelligent, and it becomes a vastly smaller effort. These beings are all rather large, at least worm-sized or greater. Their lifecycles are measured in days to years, not minutes or hours. And they only had 250 million years or so to make the leap from "bunch of neurons hard-coded to do specific jobs" to "generally intelligent arrangement of neurons capable of higher level thought". All of this cuts the scale by at least 5-10 orders of magnitude compared to what you correctly point out as the overall colossal scale of evolutionary computation. And to me, it says that compared to all the other amazingly difficult stuff that evolution has discovered, intelligence was a damn easy find, especially since there's almost no fitness benefit - hell, worms probably do better than we do as far as evolutionary fitness goes, they're tiny, numerous, and reproduce like crazy. My money is on the fact that human intelligence guiding a search cuts another couple orders of magnitude off of how much of a long-shot discovering an intelligent algorithm would be via a random search. I could definitely be wrong, it's very hard to be precise when you're vaguely arguing about orders of magnitude, but to me it doesn't seem like the algorithmic "magic" in the brain can possibly be very complex if evolution was able to get there, there just aren't many other circumstances where evolution stumbles on something that damn clever.

by maldeh 10y ago

ewjordan 10y ago

No, we don't even need to be very clever: evolution is no better able to evade weak local minima than any but the dumbest search strategies humans use, and it quite easily discovered a neural algorithm that lead to intelligence when scaled sufficiently.

legutierr 10y ago
Easily? It took 4 billion years and multiple global extinction events, and it has only happened once. Flight, the eye, air breathing all evolved multiple times independently, within distantly related evolutionary lines. Intelligence is rare in nature, in spite of its advantages, probably because it is a very distant jump from local maxima.
- vintermann 10y ago
  Do you define human intelligence as the only real intelligence, then? Or are you talking about some super-early primitive predecessor of a brain? (I'm sorry, I have no idea whether the latter has only evolved once.)
  - legutierr 10y ago
    That's the real question though, isn't it? I think what we are talking about is the kind of intelligence that could qualify as a "general AI" if implemented in a computer. Although I'm sure that there many different types of intelligence that could match up, in nature there really is only one that we know about.
    - vintermann 10y ago
      I don't think such a thing as general AI, or even general intelligence, is sensible to talk about. There are always tradeoffs: an algorithm good at finding some sort of patterns, necessarily has to be worse at finding other sorts of patterns. There are actually ways to formalize and prove that, but it's intuitively obvious that an algorithm that can find the answer to a set of questions, can't be faster than an algorithm to find the answer to a subset of those questions.
- adrianN 10y ago
  Cephalopods are reasonably smart and their intelligence evolved mostly idependently. That's reassuring. https://en.wikipedia.org/wiki/Cephalopod_intelligence https://en.wikipedia.org/wiki/Cephalopod_intelligence
chriswarbo 10y ago
> when scaled sufficiently Whilst technically correct, this obscures the truly collosal scale at which evolution computes. If we consider each organism as a processor executing a single fitness observation, we can fit hundreds of thousands of such processors (e.g. bacteria) on the few square millimetres of a pin head. These processors are spread (less densely) across the 510 million square kilometre surface of the Earth. They are suspended in oceans, which have an average depth of 3.7 kilometres; and in the atmosphere, at least up to 10 kilometres (based on our observations so far). Whilst the generation time (AKA execution time) of such organisms varies wildly, it is very commonly less than an hour. Yet even with these truly staggering resources at its disposal, evolution still took 3.3 billion years of wall clock time to come up with multicellular organisms. Relying on evolution to come up with meta-level optimisation mechanisms like intelligence is also rather flaky; we only have one example of biological evolution to study, acting on one representation (DNA + RNA + proteins). This just-so-happened to stumble upon aerobic respiration, multicellularity, neurons and brains, but it's certainly not a "goal" of the system, and sampling bias prevents us knowing how likely an outcome that is (i.e. whether it's an attractor). Still, organisms with nervous systems are a vanishingly small proportion of all lifeforms; and in fact, every "advance" is dwarfed by the success of single celled microbes. We could always leave an evolving system running for longer, but setting up conditions which lead to open-ended evolution is still an unknown area; simulations always tend to level off/saturate at some point. As a "last resort" we could provide the system with a mechanism to sample organisms completely at random, and use that as proof that anything is possible given enough time; but in that case why use evolution at all, when we can just perform that sampling directly, or alternatively just enumerate organisms?
- ewjordan 10y ago
  I nodded along with pretty much everything you said, so I think we're arguing different points but not necessarily disagreeing. I agree, evolution's scale is colossal, but restrict your view to the tiny bit of the evolutionary machine that is implicated in figuring out how to arrange neurons (which evolution "discovered" previously, to be fair, with incredible effort) in a fashion that renders their behavior intelligent, and it becomes a vastly smaller effort. These beings are all rather large, at least worm-sized or greater. Their lifecycles are measured in days to years, not minutes or hours. And they only had 250 million years or so to make the leap from "bunch of neurons hard-coded to do specific jobs" to "generally intelligent arrangement of neurons capable of higher level thought". All of this cuts the scale by at least 5-10 orders of magnitude compared to what you correctly point out as the overall colossal scale of evolutionary computation. And to me, it says that compared to all the other amazingly difficult stuff that evolution has discovered, intelligence was a damn easy find, especially since there's almost no fitness benefit - hell, worms probably do better than we do as far as evolutionary fitness goes, they're tiny, numerous, and reproduce like crazy. My money is on the fact that human intelligence guiding a search cuts another couple orders of magnitude off of how much of a long-shot discovering an intelligent algorithm would be via a random search. I could definitely be wrong, it's very hard to be precise when you're vaguely arguing about orders of magnitude, but to me it doesn't seem like the algorithmic "magic" in the brain can possibly be very complex if evolution was able to get there, there just aren't many other circumstances where evolution stumbles on something that damn clever.