4 ms·

Self-Assembling Artificial Neural Networks Through Neural Developmental Programs

65 points by hardmaru 3y ago

great_psy 3y ago
Seems promising. I guess the one catch is that all this extra computation required for self assembly might not be received back in the form of a better system compared to using that computation to better train a human designed neural network.
- modeless 3y ago
  Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. I'd be surprised if some kind of self-optimizing architecture wasn't standard in a decade or two.
  - dragonwriter 3y ago
    > Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. Is it? The “easy” tasks for neural nets to beat humans at are things that are harder for humans to do than for humans to provide volumes of high-quality training data by way of which neural nets can be trained. It’s not clear to ne that designing neural nets really fits that.
    - dsign 3y ago
      The current crop of LLM are better aware of all the state of the art in NN architecture than yours-truly. They know things such as the number of layers used for X, Y and Z, the class of cost function which is popular with a specific architecture, dropout-vs-no-dropout, etc. That kind of knowledge still needs a human engineer to be leveraged, but even now it seems it's more because nobody has built a harness for the LLM to be able to put together neural network architectures. And, there are already automatic constraint checkers for NN architectures, so even if an LLM hallucinates something wrong, constraint checks could be used to automatically call its attention to some design flaws.
  - signa11 3y ago
    > ... Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. I'd be surprised if some kind of self-optimizing architecture wasn't standard in a decade or two. wouldn't evolutionary algorithms be better suited for this ? evolution is how we got here after all ? perhaps different constraints on the 'state space' can 'guide' the whole process...
    - buffalobuffalo 3y ago
      There are several evolutionary algorithms that are used to design network topography as well as weights. NEAT is a popular one: https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t... There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on. Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it. These problems are often enough to prohibit GA use.
      - mnky9800n 3y ago
        > There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on. > Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it. This is true for the living as well.
      - signa11 3y ago
        i went over these resources, and what strikes me as kind of odd, is the fact that the folks involved in this research / exploration tried to evolve both the architecture and the weights. i was naively thinking that architecture should be evolved, while weights should be computed just as it is being done now i.e. via back-propogation etc.
      - falcor84 3y ago
        I suppose that eventually, we'll have the metric being actual "money" that the neural nets try to earn (by mining crpyto, or solving captchas, etc.), such they'd have to pay for their own infrastructure costs. It would then be up to each of them to decide if they'd like to spawn "children", and then decide on an appropriate revenue-share model with them (maybe even pyramid schemes).
  - great_psy 3y ago
    Maybe, but I think humans are better at building a NN architecture. A NN could design a better wing for a bird, but a human can design a jet engine. While the NN could eventually get to the jet engine, all the computation might be better used to train a human design architecture on more data/more epochs/ more complex optimization
djoldman 3y ago
Abstract: > Biological nervous systems are created in a fundamentally different way than current artificial neural networks. Despite its impressive results in a variety of different domains, deep learning often requires considerable engineering effort to design high-performing neural architectures. By contrast, biological nervous systems are grown through a dynamic self-organizing process. In this paper, we take initial steps toward neural networks that grow through a developmental process that mirrors key properties of embryonic development in biological organisms. The growth process is guided by another neural network, which we call a Neural Developmental Program (N DP) and which operates through local communication alone. We investigate the role of neural growth on different machine learning benchmarks and different optimization methods (evolutionary training, online RL, offline RL, and supervised learning). Additionally, we highlight future research directions and opportunities enabled by having self-organization driving the growth of neural networks.
Nevermark 3y ago
Except our brain wasn’t built by meta-neural networks. Its development is governed by a DNA algorithm. Our DNA program is changing slowly by evolution, but at any given time, the design we have is one design intended to work well across all survival problems. Each of us doesn’t have a brain architecture that our body custom designed for some different set of problems or data.
Nevermark 3y ago
Except our brain wasn’t built by meta-neural networks. Its development is governed by a DNA algorithm. Our DNA program is changing slowly by evolution, but at any given time, the design we have is one design intended to work well across all survival problems. Each of us doesn’t have a brain architecture that our body custom designed for some different set of problems or data. Not critiquing the tech. Critiquing some of its rationalization.
- vacuity 3y ago
  Our brains indeed are not quite like neural networks, but evolution isn't uniform for everyone, including in terms of brain development. There's also sexual dimorphism and epigenetics to consider. A lot more haphazard than custom designing, that's for sure.
- kbelder 3y ago
  But the expression of the brain as it grows is very dynamic, influenced by many factors outside the genetic, including itself, self-referentially. You could pretty easily raise one twin as an imbecile and the other as competent, purely by the means of changing their sensory inputs during the phase of brain development. If you had no scruples, of course.

65 points by hardmaru 3y ago

Seems promising. I guess the one catch is that all this extra computation required for self assembly might not be received back in the form of a better system compared to using that computation to better train a human designed neural network.

modeless 3y ago
Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. I'd be surprised if some kind of self-optimizing architecture wasn't standard in a decade or two.
- dragonwriter 3y ago
  > Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. Is it? The “easy” tasks for neural nets to beat humans at are things that are harder for humans to do than for humans to provide volumes of high-quality training data by way of which neural nets can be trained. It’s not clear to ne that designing neural nets really fits that.
  - dsign 3y ago
    The current crop of LLM are better aware of all the state of the art in NN architecture than yours-truly. They know things such as the number of layers used for X, Y and Z, the class of cost function which is popular with a specific architecture, dropout-vs-no-dropout, etc. That kind of knowledge still needs a human engineer to be leveraged, but even now it seems it's more because nobody has built a harness for the LLM to be able to put together neural network architectures. And, there are already automatic constraint checkers for NN architectures, so even if an LLM hallucinates something wrong, constraint checks could be used to automatically call its attention to some design flaws.
- signa11 3y ago
  > ... Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. I'd be surprised if some kind of self-optimizing architecture wasn't standard in a decade or two. wouldn't evolutionary algorithms be better suited for this ? evolution is how we got here after all ? perhaps different constraints on the 'state space' can 'guide' the whole process...
  - buffalobuffalo 3y ago
    There are several evolutionary algorithms that are used to design network topography as well as weights. NEAT is a popular one: https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t... There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on. Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it. These problems are often enough to prohibit GA use.
    - mnky9800n 3y ago
      > There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on. > Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it. This is true for the living as well.
    - signa11 3y ago
      i went over these resources, and what strikes me as kind of odd, is the fact that the folks involved in this research / exploration tried to evolve both the architecture and the weights. i was naively thinking that architecture should be evolved, while weights should be computed just as it is being done now i.e. via back-propogation etc.
    - falcor84 3y ago
      I suppose that eventually, we'll have the metric being actual "money" that the neural nets try to earn (by mining crpyto, or solving captchas, etc.), such they'd have to pay for their own infrastructure costs. It would then be up to each of them to decide if they'd like to spawn "children", and then decide on an appropriate revenue-share model with them (maybe even pyramid schemes).
- great_psy 3y ago
  Maybe, but I think humans are better at building a NN architecture. A NN could design a better wing for a bird, but a human can design a jet engine. While the NN could eventually get to the jet engine, all the computation might be better used to train a human design architecture on more data/more epochs/ more complex optimization

djoldman 3y ago

Abstract: > Biological nervous systems are created in a fundamentally different way than current artificial neural networks. Despite its impressive results in a variety of different domains, deep learning often requires considerable engineering effort to design high-performing neural architectures. By contrast, biological nervous systems are grown through a dynamic self-organizing process. In this paper, we take initial steps toward neural networks that grow through a developmental process that mirrors key properties of embryonic development in biological organisms. The growth process is guided by another neural network, which we call a Neural Developmental Program (N DP) and which operates through local communication alone. We investigate the role of neural growth on different machine learning benchmarks and different optimization methods (evolutionary training, online RL, offline RL, and supervised learning). Additionally, we highlight future research directions and opportunities enabled by having self-organization driving the growth of neural networks.

Nevermark 3y ago

Except our brain wasn’t built by meta-neural networks. Its development is governed by a DNA algorithm. Our DNA program is changing slowly by evolution, but at any given time, the design we have is one design intended to work well across all survival problems. Each of us doesn’t have a brain architecture that our body custom designed for some different set of problems or data.

Nevermark 3y ago

vacuity 3y ago
Our brains indeed are not quite like neural networks, but evolution isn't uniform for everyone, including in terms of brain development. There's also sexual dimorphism and epigenetics to consider. A lot more haphazard than custom designing, that's for sure.
kbelder 3y ago
But the expression of the brain as it grows is very dynamic, influenced by many factors outside the genetic, including itself, self-referentially. You could pretty easily raise one twin as an imbecile and the other as competent, purely by the means of changing their sensory inputs during the phase of brain development. If you had no scruples, of course.