6 ms·

Strong AI Requires Autonomous Building of Composable Models

58 points by andreyk 5y ago

swayvil 5y ago
For the AI it's models all the way down. For us there is something prior to the model. That's a world of difference.
- jkhdigital 5y ago
  Any system capable of homeostasis--maintaining its own existence in the face of a range of environmental conditions--is using models. The models can be hard-coded in genetic material (which was "computed" over aeons of selection pressure) or dynamically learned and encoded in the span of the organism's life. Still models. It's like bootstrapping certain operating systems from source--some of the dependencies exist only in machine code; the original source code has been lost to the sands of time. A human being is modeled by 23 chromosomes' worth of DNA, and when that model is planted in a properly-functioning adult female reproductive system it turns itself into a new human being. I don't see why the world of CPUs and nonvolatile memory is any less capable of being a substrate for intelligent, self-replicating entities as the world of particles and atoms.
  - swayvil 5y ago
    Aesthetics can be used too. No need for a model.
    - redisman 5y ago
      I’m not sure what exactly you mean by aesthetics but photons going into your eyeball and ending up as “images” constructed by your brain is much closer to a model fed by some data than it being some objective reality ingestor
      - swayvil 5y ago
        Aesthetics is a sense of right/wrong and beauty/ugly. It is used as a guide for action, among other things. It operates independently from the play of models. It is used to get axioms, among other things.
castratikron 5y ago
Sphex wasp story sounds like JIT supply chains. Maybe AI doesn't have far to go after all?
- jkhdigital 5y ago
  The irony is that there’s an iceberg of robustly composed and functional models that the “brittle” Sphex wasp behavior is resting on top of, like everything that allows the wasp to even know where it’s nest is and how to stun prey and bring it back. Disrupting the behavioral program causes the wasp to run a loop, not to curl up and die. AI has a very, very long way to go.
holomorphically 5y ago
I wonder when AI researchers are going to learn some category theory: https://bartoszmilewski.com/2014/11/04/category-the-essence-of-composition/ https://bartoszmilewski.com/2014/11/04/category-the-essence-....
juliangamble 5y ago
“The developmental psychologist Jean Mandlar says that image schemas make up the foundation of human thought.” Interesting that the author refutes that the brain thinks in symbols.
joe_the_user 5y ago
It seems like AI system needs to be able to create subsystem that act like they're composable, that have properties like fully composable items but can't be fully composable. Full composability, as the author eludes, requires a messy reality be fully condensed to a compact model - and by now, we know the world is going to be more messy than exactly representable by a reasonably compact model. An alternative to an AI just using compact models is an AI searching for areas approximatable by a compact model and then having an approximation of the boundary/limit of this models (and constantly updating that boundary/limit).
- m0zg 5y ago
  At this point I think just having the composability without the compactness would be a major breakthrough. Heck, even just figuring out how to efficiently harness multimodal inputs or better handle inputs that vary over time (e.g. video) would be a breakthrough as well. Case in point: nearly all academic models for object detection are borderline useless in practice because they work on individual frames, rather than on video, and as a result they aren't very robust and they lose objects frame to frame for no good reason. There are tracking models, but they are far less researched than single-frame ones, and much more complex. In essence, it's like that drunk who was looking for his car keys under a street lamp rather than where he lost them because it's easier to search for them there. People don't realize how little of this surface we've scratched so far, and how awesome this stuff will get once we properly scratch it all over.
  - joe_the_user 5y ago
    At this point I think just having the composability without the compactness would be a major breakthrough. I'd claim that practical composibility without compactness impossible. Being huge, like a deep neural net, makes composability intractable. If your "elements" are akin to deep nets, composition will be very, very hard to calculate and so impossible to train.
    - m0zg 5y ago
      By "composability" I mean you'd be able to train models separately and then perhaps slightly refine the composed model as a whole on the final task(s). And models would be using each other's outputs, sort of like today you can take an ImageNet backbone and put an object detection head on top, except at a higher level. I.e. an optical (+IR) multi-camera object tracking model, fused with lidar and radar, feeding into a path planning model. As of today, I have just described a billion dollars worth of modeling at least. That needs to be simpler and cheaper. Currently stuff like this has to be trained end to end, which, as you've pointed out, is very difficult to do, to the point where there are only a handful of companies that can convincingly pull it off (Tesla, Google, OpenAI, _maybe_ Facebook), and even they are arguably at a local minimum this approach allows. And this local minimum won't get us to "real" AI, which I think was the point of this rather rambling article.
golemotron 5y ago
Here come the category theorists.
- gsam 5y ago
  Category theory isn't the only way to solve this. Arguably a purely continuous dynamical system like differential equations with certain boundary conditions would similarly work. Discrete dynamical systems however, are much better at representing finite, discrete relationships, particularly with recursion. I'm only just learning about these topics, but simple rules lead to modelling indeterminately complex behaviour (Rule 30, logistic map). These can be viewed quite clearly through the lens of category theory as functors and fixed points. However, the gaps between automata theory, discrete (and non-linear) dynamical systems and finally category theory are still very wide at the moment.
  - malux85 5y ago
    This is one of the most interesting comments I have read in months, can I contact you?
    - gsam 5y ago
      Do you have somewhere to send you some details? Otherwise, you can probably find a mail on my Github with the same name.
- jkhdigital 5y ago
  And intensional type theorists
xiphias2 5y ago
Transformers look like a great building block for these models, but if course there are lots of challenges on the technical details.
scionaura 5y ago
If I had a dollar for every time some guy banged on about what is required for strong AI... Nobody should believe this kind of article even if someone really clever wrote it
- otabdeveloper4 5y ago
  It's simple. Instead of worrying about how to make machines smarter, just make people stupider instead. It worked for the Turing test, so why not for AGI too?

58 points by andreyk 5y ago

swayvil 5y ago

For the AI it's models all the way down. For us there is something prior to the model. That's a world of difference.

jkhdigital 5y ago
Any system capable of homeostasis--maintaining its own existence in the face of a range of environmental conditions--is using models. The models can be hard-coded in genetic material (which was "computed" over aeons of selection pressure) or dynamically learned and encoded in the span of the organism's life. Still models. It's like bootstrapping certain operating systems from source--some of the dependencies exist only in machine code; the original source code has been lost to the sands of time. A human being is modeled by 23 chromosomes' worth of DNA, and when that model is planted in a properly-functioning adult female reproductive system it turns itself into a new human being. I don't see why the world of CPUs and nonvolatile memory is any less capable of being a substrate for intelligent, self-replicating entities as the world of particles and atoms.
- swayvil 5y ago
  Aesthetics can be used too. No need for a model.
  - redisman 5y ago
    I’m not sure what exactly you mean by aesthetics but photons going into your eyeball and ending up as “images” constructed by your brain is much closer to a model fed by some data than it being some objective reality ingestor
    - swayvil 5y ago
      Aesthetics is a sense of right/wrong and beauty/ugly. It is used as a guide for action, among other things. It operates independently from the play of models. It is used to get axioms, among other things.

castratikron 5y ago

Sphex wasp story sounds like JIT supply chains. Maybe AI doesn't have far to go after all?

jkhdigital 5y ago
The irony is that there’s an iceberg of robustly composed and functional models that the “brittle” Sphex wasp behavior is resting on top of, like everything that allows the wasp to even know where it’s nest is and how to stun prey and bring it back. Disrupting the behavioral program causes the wasp to run a loop, not to curl up and die. AI has a very, very long way to go.

holomorphically 5y ago

I wonder when AI researchers are going to learn some category theory: https://bartoszmilewski.com/2014/11/04/category-the-essence-of-composition/ https://bartoszmilewski.com/2014/11/04/category-the-essence-....

juliangamble 5y ago

“The developmental psychologist Jean Mandlar says that image schemas make up the foundation of human thought.” Interesting that the author refutes that the brain thinks in symbols.

joe_the_user 5y ago

It seems like AI system needs to be able to create subsystem that act like they're composable, that have properties like fully composable items but can't be fully composable. Full composability, as the author eludes, requires a messy reality be fully condensed to a compact model - and by now, we know the world is going to be more messy than exactly representable by a reasonably compact model. An alternative to an AI just using compact models is an AI searching for areas approximatable by a compact model and then having an approximation of the boundary/limit of this models (and constantly updating that boundary/limit).

m0zg 5y ago
At this point I think just having the composability without the compactness would be a major breakthrough. Heck, even just figuring out how to efficiently harness multimodal inputs or better handle inputs that vary over time (e.g. video) would be a breakthrough as well. Case in point: nearly all academic models for object detection are borderline useless in practice because they work on individual frames, rather than on video, and as a result they aren't very robust and they lose objects frame to frame for no good reason. There are tracking models, but they are far less researched than single-frame ones, and much more complex. In essence, it's like that drunk who was looking for his car keys under a street lamp rather than where he lost them because it's easier to search for them there. People don't realize how little of this surface we've scratched so far, and how awesome this stuff will get once we properly scratch it all over.
- joe_the_user 5y ago
  At this point I think just having the composability without the compactness would be a major breakthrough. I'd claim that practical composibility without compactness impossible. Being huge, like a deep neural net, makes composability intractable. If your "elements" are akin to deep nets, composition will be very, very hard to calculate and so impossible to train.
  - m0zg 5y ago
    By "composability" I mean you'd be able to train models separately and then perhaps slightly refine the composed model as a whole on the final task(s). And models would be using each other's outputs, sort of like today you can take an ImageNet backbone and put an object detection head on top, except at a higher level. I.e. an optical (+IR) multi-camera object tracking model, fused with lidar and radar, feeding into a path planning model. As of today, I have just described a billion dollars worth of modeling at least. That needs to be simpler and cheaper. Currently stuff like this has to be trained end to end, which, as you've pointed out, is very difficult to do, to the point where there are only a handful of companies that can convincingly pull it off (Tesla, Google, OpenAI, _maybe_ Facebook), and even they are arguably at a local minimum this approach allows. And this local minimum won't get us to "real" AI, which I think was the point of this rather rambling article.

golemotron 5y ago

Here come the category theorists.

gsam 5y ago
Category theory isn't the only way to solve this. Arguably a purely continuous dynamical system like differential equations with certain boundary conditions would similarly work. Discrete dynamical systems however, are much better at representing finite, discrete relationships, particularly with recursion. I'm only just learning about these topics, but simple rules lead to modelling indeterminately complex behaviour (Rule 30, logistic map). These can be viewed quite clearly through the lens of category theory as functors and fixed points. However, the gaps between automata theory, discrete (and non-linear) dynamical systems and finally category theory are still very wide at the moment.
- malux85 5y ago
  This is one of the most interesting comments I have read in months, can I contact you?
  - gsam 5y ago
    Do you have somewhere to send you some details? Otherwise, you can probably find a mail on my Github with the same name.
jkhdigital 5y ago
And intensional type theorists

xiphias2 5y ago

Transformers look like a great building block for these models, but if course there are lots of challenges on the technical details.

scionaura 5y ago

If I had a dollar for every time some guy banged on about what is required for strong AI... Nobody should believe this kind of article even if someone really clever wrote it

otabdeveloper4 5y ago
It's simple. Instead of worrying about how to make machines smarter, just make people stupider instead. It worked for the Turing test, so why not for AGI too?