10 ms·

New neural network architecture inspired by neural system of a worm

218 points by burrito_brain 4y ago

nynx 4y ago
As far as I can tell, they analytically solved the style of ODE used in biologically-motivated neural networks (usually spiking, but not in this case) and then trained a network built from those to do stuff.
sillysaurusx 4y ago
It makes a good headline, but reading over the paper (https://www.nature.com/articles/s42256-022-00556-7.pdf https://www.nature.com/articles/s42256-022-00556-7.pdf) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud feels like daring history to make a fool of me. My view is that biology just happened to evolve how it did, so there’s no point in copying it; it worked because it worked. If we have to train networks from scratch, then we have to find our own solutions, which will necessarily be different than nature’s. I find analogies useful; dividing a model into short term memory vs long term memory, for example. But it’s best not to take it too seriously, like we’re somehow cloning a brain. Not to mention that ML nets still don’t control their own loss functions, so we’re a poor shadow of nature. ML circa 2023 is still in the intelligent design phase, since we have to very intelligently design our networks. I await the day that ML networks can say “Ok, add more parameters here” or “Use this activation instead” (or learn an activation altogether — why isn’t that a thing?).
- adamzen 4y ago
  Learned activation functions do seem to be a thing(https://arxiv.org/abs/1906.09529 https://arxiv.org/abs/1906.09529)
- comfypotato 4y ago
  The open worm project is the product of microscopically mapping the neural network (literally the biological network of neurons) in a nematode. How isn’t this biologically inspired? If I’m reading it correctly, the equations that you’re misinterpreting are the neuron models that make each node in the map. I would guess that part of the inspiration for using the word “liquid” comes from the origins of the project in which they were modeling the ion channels in the synapses. They’ve been training these artificial nematodes to swim for years. The original project was fascinating (in a useless way): you could put the model of the worm in a physics engine and it would behave like the real-life nematode. Without any programming! It was just an emergent behavior of the mapped-out neuron models (connected to muscle models). It makes sense that they’ve isolated the useful part of the network to train it for other behaviors. I used to follow this project, and I thought it had lost steam. Glad to see Ramin is still hard at work.
  - sillysaurusx 4y ago
    Interesting. Is there a way to run it? One of the challenges with work like this is that you have to figure out how to get output from it. What would the output be? As far as my objection, it seems like an optimization, not an architecture inspired by the worm. I.e. “inspired by” makes it sound like this particular optimization was derived from studying the worm’s neural networks and translating it into code, when it was the other way around. But it would be fascinating if that wasn’t the case.
    - comfypotato 4y ago
      See for yourself! There’s a simulator (have only tried on desktop) to run the worm model in your browser. As the name implies, the project is completely open source (if you’re feeling ambitious). This is the website for the project that produced the research in the article: https://openworm.org/ https://openworm.org/ Nematodes make up much of this particular segment of the history of neuroscience. This project builds on lots of data produced by prior researchers. Years of dissecting the worms and mapping out the connections between the neurons (and muscles, organs, etc.). It is by far the most completely-mapped organism. The neuronal models, similarly, are based on our understanding of biological neurons. For example: the code has values in each ion channel that store voltages across the membranes. An action potential is modeled by these voltages running along the axons to fire other neurons. I’m personally more familiar with heart models (biomedical engineering background here) but I’m sure it’s similar. In the heart models: calcium, potassium, and sodium concentrations are updated every unit of time, and the differences in concentrations produce voltages.
- f_devd 4y ago
  Learnable activation functions are a thing famously Swish[0] is is a trainable SiLU which was found through symbolic search/optimization [1], but as it turns out that doesn't magically make make neural networks orders better. [0]: https://en.m.wikipedia.org/wiki/Swish_function https://en.m.wikipedia.org/wiki/Swish_function [1]: https://arxiv.org/abs/1710.05941 https://arxiv.org/abs/1710.05941
- uoaei 4y ago
  > I’m skeptical that biological systems will ever serve as a basis for ML nets in practice There is no fundamental difference between information processing systems implemented in silico vs in vivo, except architecture. Architecture is what constrains the manifold of internal representations: this is called "inductive bias" in the field of machine learning. The math (technically, the non-equilibrium statistical physics crossed with information theory) is fundamentally the same. Everything at the functionalist level follows from architecture; what enables these functions is the universal principles of information processing per se. "It worked because it worked" because there is no other way for it to work given the initial conditions of our neighborhood in the universe. I'm not saying "Everything ends up looking like a brain". Rather, I am saying "The brain, attendant nervous and sensory systems, etc. vs neural networks implemented as nonlinear functions are running the same instructions on different hardware, thus resulting in different algorithms." The way I like to put it is: trust Nature's engineers, they've been at it much longer than any of us have.
  - skibidibipiti 4y ago
    > There is no fundamental difference between information processing in silicon and in vivo A neuron has dozens of neurotransmitters, while artificial neurons produce 1 output. I don't know much about neurology, but how is the information processing similar? What do you mean are running the same instructions? > there is no other way for it to work Plants exhibit learned behaviors
    - uoaei 4y ago
      > how is the information processing similar? The representational capacities are of course not the same -- the same "thoughts" cannot be expressed in both systems. But the concept of "processing over abstract representations enacted in physical dynamics within cognitive systems" is shared between all systems of this kind. I am referring to "information processing" at the physical level, i.e., "'useful' work per energy quantum as communicated through noisy channels". > What do you mean are running the same instructions? The underlying physical principles of such information processing are equivalent regardless of physical implementation. > plants exhibit learned behaviors A good example of what I mean. The architecture is different, but the underlying dynamics is the same. There is a convincing (to me) theory of the origins of life[1][2] that states that thermodynamics -- and, by extension, information theory -- is the appropriate level of abstraction for understanding what distinguishes living processes from inanimate ones. The theory posits that a system, well-defined by some (possibly arbitrary) boundaries, "learns" (develops channels through which "patterns" can be "recognized" and possibly interacted with) as an inevitable result of physics. Put another way, a learning system is one that represents its experiences through the cumulative wearing-in over time of channels of energy flows. What concepts the system can possibly represent depends on in what ways the system can wear while maintaining its essential functions. What specifically the system learns is the set of concepts which collectively best communicate (physically, i.e., from the "inputs" through the "processing" functions and to the "outputs") the historical set of its experiences of its environment and of itself. I want to note that this discussion has nothing to say on perception, only sensation and reaction: in other words, it is an exclusively materialist analysis. Optimization theory describes its notion of learning roughly as such (considering "loss" as energy potentials), but with the same language we could also describe a human brain, or a black hole's accretion disk, or an ant colony dug deep into clay. References: [1] https://www.englandlab.com/uploads/7/8/0/3/7803054/2013jcpsrep.pdf https://www.englandlab.com/uploads/7/8/0/3/7803054/2013jcpsr... [2] https://www.quantamagazine.org/a-new-thermodynamics-theory-of-the-origin-of-life-20140122/ https://www.quantamagazine.org/a-new-thermodynamics-theory-o... Parallel directions of research: https://en.wikipedia.org/wiki/Entropy_and_life https://en.wikipedia.org/wiki/Entropy_and_life https://en.wikipedia.org/wiki/Free_energy_principle https://en.wikipedia.org/wiki/Free_energy_principle
- ly3xqhl8g9 4y ago
  "I’m skeptical that biological systems will ever serve as a basis for ML nets in practice" First of all, ML engineers need to stop being so brainphiliacs, caring only about the 'neural networks' of the brain or brain-like systems. Lacrymaria olor has more intelligence, in terms of adapting to exploring/exploiting a given environment, than all our artificial neural networks combined and it has no neurons because it is merely a single-cell organism [1]. Once you stop caring about the brain and neurons and you find out that almost every cell in the body has gap junctions and voltage-gated ion channels which for all intents and purposes implement boolean logic and act as transistors for cell-to-cell communication, biology appears less as something which has been overcome and more something towards which we must strive with our primitive technologies: for instance, we can only dream of designing rotary engines as small, powerful, and resilient as the ATP synthase protein [2]. [1] Michael Levin: Intelligence Beyond the Brain, https://youtu.be/RwEKg5cjkKQ?t=202 https://youtu.be/RwEKg5cjkKQ?t=202 [2] Masasuke Yoshida, ATP Synthase. A Marvellous Rotary Engine of the Cell, https://pubmed.ncbi.nlm.nih.gov/11533724 https://pubmed.ncbi.nlm.nih.gov/11533724
  - phaedrus 4y ago
    I wonder if there's a step-change where single-celled animals with complex behavior are actually smarter than the simplest multiple-celled animals with a nervous system.
    - whatshisface 4y ago
      The cells of multi-celled animals still have complex behaviors.
  - outworlder 4y ago
    Indeed. All cells must do complex computations, by their own nature. Just the process of producing proteins and each of its steps – from 'unrolling' a given DNA section, copying it, reading instructions... even a lowly ribosome is a computer (one that even kinda looks like a Turing machine from a distance)
  - mk_stjames 4y ago
- danielheath 4y ago
  It definitely won’t happen without a massive overhaul of chip design; a design that optimises for very broad connectivity with storage for the connection would be a step in that direction (neural connectivity is on the order of 10k connections each, and the connection stores temporal information about how recently it last fired / how often it’s fired recently)
- jononor 4y ago
  I think that learning to acquire new/additional training data would be a better first step towards learning agents, than trying to mutate its structure/hyper-parameters.
- smrtinsert 4y ago
  Is it still a milestone for all NNs?
- satvikpendem 4y ago
  > so there’s no point in copying it Not sure about that, a lot of solutions in nature are honed by billions of years of evolution, sometimes creating feats even more impressive than we can do currently. There is an entire field about copying biology to solve our problems: https://en.wikipedia.org/wiki/Biomimetics https://en.wikipedia.org/wiki/Biomimetics
- mr_toad 4y ago
  > It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. You say that like it isn’t a big deal. Finding an analytical solution to optimising the parameters of a non-linear equation is remarkable.
- xwolfi 4y ago
  Well I tend to agree but you seem to think biology evolved in a vacuum but it evolved inside an information source and we're all the information processors: ML having to process the same information but at scale, in the grand scheme, it will probably have to ressemble a brain in some ways. Just the sources we care about (colors in a picture, faces, prices, wind direction, whatever) and the output we can understand (text, images, sound) will have to skew it towards us in the way it has to model things.
- fatneckbeard 4y ago
  well i will agree on one thing.... corporations are constantly looking for a machine to do labor for free. life itself did not evolve just to do labor for a corporation so by trying to copy biological intelligent life, the result won't necessarily want to do what you tell it to do or be interested in your profit motives.
- mensetmanusman 4y ago
  “ which will necessarily be different than nature’s” We are nature’s…
- canadianfella 4y ago
  [dead]
- dd36 4y ago
  We are nature. For all we know the solutions that cells came up with were derived in similar ways. Kevin Kelly’s “What Technology Wants” documents how evolution repeats itself in our technology.
- dilawar 4y ago
  A hell lot of computation inside a neuron (in fact inside any cell) is chemical in nature. Proteins interacting, channels opening and closing, membrane doing membrainy stuff.. In fact their is a AND gate which is entirely chemical in nature. Simulating chemical reactions are slow in silicon therefore chemical side is ignored. If you glance over the graphs in chemistry papers, most of them are sigmoids. sigmoids are the sinusoid of chemical world. Its nice and heartening to see sinusoid appearing often in AI/ML as a fundamental computation.
- noobermin 4y ago
  So I wasn't skeptical in the way you found it, but it did sound a heck lot to me like tradition numerical solution of PDEs...but with NNs in there somehow.
- water-your-self 4y ago
  Looking at biology is what lead to CNNs the current AI boom. Thats the whole reason we call multi layered perceptrons as "neural nets" cheesey and flashy as it is, using a sliding filter was inspired by what we know about vision im biology.
f_devd 4y ago
Although the article is recent the paper from the article has been available on preprint/arxiv since June 2021[1], implementations for pytorch & tensorflow are also available[2] for those interested. [1]: https://arxiv.org/abs/2106.13898 https://arxiv.org/abs/2106.13898 [2]: https://github.com/raminmh/CfC https://github.com/raminmh/CfC
lairv 4y ago
Is there any reason to believe that biologically inspired architectures should yield better performance ? Brain are biological systems which have been trained through evolutionary processes. Neural Networks are algorithmic/linear algebra models trained through statistical methods One might argue that CNN are biologically inspired, but it's more likely that the reason they work is because they respects input symmetries
- danans 4y ago
  > Is there any reason to believe that biologically inspired architectures should yield better performance ? At the very least, they could yield far better efficiency. A 12W brain can achieve more an entire data center of GPUs, depending on what you are trying to achieve. Whether that would make something actually demonstrate sentience level performance is another question.
- OkayPhysicist 4y ago
  We know, for a fact, that biological brains work. Not only do they work, they work enormously well learning and adapting based on dramatically less available data, utilizing vastly less resources than anything we've conceived of in artificial computing. Biological architectures may not be the best possible, but empirical evidence demonstrates that they can result in intelligences ranging all the way up to sentience.
  - DiscourseFan 4y ago
    The question is if its appropriate to compare logical machines, which are built on things secondary, a-posteriori to the primary aspects of human cognition--that is to say logic--with the primary, biological, a-priori aspects of cognition which are in some sense inscrutable. I myself do not believe that we will never be able to comprehensively understand the way in which our minds work, only religion leaves mysteries up to God. But I think that using scientific empirical logic to understand how we are able to perform judgements such as those made with scientific empirical logic will never yield the proper result; judgement itself must be investigated. Something I don't think many researchers in the field of neural-networks are capable of doing.
dvh 4y ago
The old neuroscience saying goes like this: "Human brain have billions of neurons and so it is too complex to understand, that's why neuroscience study simpler organisms. Flatworm's brain have 52 neurons. We have no idea how it works". Did finally something changed in this regard?
- rmorey 4y ago
  Yes. The C. Elegans brain (~300 neurons) was the first organism to be completely mapped to a connectome (the map of all connections). The first complete connectome of any centralized brain, the fruit fly, is about to be completed by the Flywire project (https://home.flywire.ai/ https://home.flywire.ai/) ~100,000 neurons and ~70,000,000 synapses. We have just a little idea how it works ;)
  - mr_toad 4y ago
    A bit of extrapolation might suggested we could map out the connectome of a human brain in 40-50 years. Not that I’d suggest a linear extrapolation from two data points…
    - heavenlyblue 4y ago
      Are you willing to kill as many humans as there were flies killed when doing that?
      - mr_toad 4y ago
        You’d only need a fraction of the 100 million+ people who die every year. There are probably bigger ethical questions when it comes to simulating a human mind.
- babblingfish 4y ago
  There have been projects to systematically catalog all the synapses in the flatworm. The problem is that neural plasticity means these connections change dynamically over time based on the needs of the organism. Since the only way we can study the flatworm at the synapse level is by killing the worm and mounting it on slides and staining it and viewing it through a high power microscope, we can only analyze its structure at points frozen in time (and formaldehyde). The reason we will never be able to truly model and understand neural networks (irl) is because their plasticity is very difficult to study with our current methods. Not only do the quantity and location of the synapses change, but the concentration and type of neurotransmitters at the synapses change. And on top of that the concentration of the neurotransmitter receptors are constantly being up regulated and down regulated by the receiving neuron. Each of these factors is really important to what the neuron is actually doing. This is why even a simple organism can have basically an unlimited amount of complexity. To understand a dynamic system like this would require very precise measurements of very small particles in vivo which is currently impossible with our tools.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
Kalanos 4y ago
so liquid nets don't need inputs of a fixed length?
akomtu 4y ago
Imo, the next step in ML is unleashing the electron a bit. Right now we keep probabilistic electrons on leash in transistors, so they behave deterministically. This despotic method has taken us far, but without giving electrons some freedom back, we won't advance further.
- bGl2YW5j 4y ago
  I'm interested; could you expand on this please? How can we give electrons more 'freedom' and what would that result in?
  - akomtu 4y ago
    Transistors could have microscopic chambers where electrons could go for a walk and exercise limited freedoms. Their behavior would be modulated by the external magnetic field to prevent riots.
deleted 4y ago
[deleted]
steve76 4y ago
[dead]
d4rkp4ttern 4y ago
Maybe I have missed it but I don’t know why there is not much talk about simulating evolution at high speed: brains evolved over millions of years to adapt to the environment and ensure survival. So instead of trying to understand and reproduce brain structures, we instead simulate evolution of embodied agents ultra high speed and see if some paths lead to brains comparable to today’s organisms.
- jamesk_au 4y ago
  For a fun exploration of this, see the short story “Crystal Nights”, by Greg Egan: https://www.gregegan.net/MISC/CRYSTAL/Crystal.html https://www.gregegan.net/MISC/CRYSTAL/Crystal.html

218 points by burrito_brain 4y ago

nynx 4y ago

As far as I can tell, they analytically solved the style of ODE used in biologically-motivated neural networks (usually spiking, but not in this case) and then trained a network built from those to do stuff.

sillysaurusx 4y ago

It makes a good headline, but reading over the paper (https://www.nature.com/articles/s42256-022-00556-7.pdf https://www.nature.com/articles/s42256-022-00556-7.pdf) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud feels like daring history to make a fool of me. My view is that biology just happened to evolve how it did, so there’s no point in copying it; it worked because it worked. If we have to train networks from scratch, then we have to find our own solutions, which will necessarily be different than nature’s. I find analogies useful; dividing a model into short term memory vs long term memory, for example. But it’s best not to take it too seriously, like we’re somehow cloning a brain. Not to mention that ML nets still don’t control their own loss functions, so we’re a poor shadow of nature. ML circa 2023 is still in the intelligent design phase, since we have to very intelligently design our networks. I await the day that ML networks can say “Ok, add more parameters here” or “Use this activation instead” (or learn an activation altogether — why isn’t that a thing?).

adamzen 4y ago
Learned activation functions do seem to be a thing(https://arxiv.org/abs/1906.09529 https://arxiv.org/abs/1906.09529)
comfypotato 4y ago
The open worm project is the product of microscopically mapping the neural network (literally the biological network of neurons) in a nematode. How isn’t this biologically inspired? If I’m reading it correctly, the equations that you’re misinterpreting are the neuron models that make each node in the map. I would guess that part of the inspiration for using the word “liquid” comes from the origins of the project in which they were modeling the ion channels in the synapses. They’ve been training these artificial nematodes to swim for years. The original project was fascinating (in a useless way): you could put the model of the worm in a physics engine and it would behave like the real-life nematode. Without any programming! It was just an emergent behavior of the mapped-out neuron models (connected to muscle models). It makes sense that they’ve isolated the useful part of the network to train it for other behaviors. I used to follow this project, and I thought it had lost steam. Glad to see Ramin is still hard at work.
- sillysaurusx 4y ago
  Interesting. Is there a way to run it? One of the challenges with work like this is that you have to figure out how to get output from it. What would the output be? As far as my objection, it seems like an optimization, not an architecture inspired by the worm. I.e. “inspired by” makes it sound like this particular optimization was derived from studying the worm’s neural networks and translating it into code, when it was the other way around. But it would be fascinating if that wasn’t the case.
  - comfypotato 4y ago
    See for yourself! There’s a simulator (have only tried on desktop) to run the worm model in your browser. As the name implies, the project is completely open source (if you’re feeling ambitious). This is the website for the project that produced the research in the article: https://openworm.org/ https://openworm.org/ Nematodes make up much of this particular segment of the history of neuroscience. This project builds on lots of data produced by prior researchers. Years of dissecting the worms and mapping out the connections between the neurons (and muscles, organs, etc.). It is by far the most completely-mapped organism. The neuronal models, similarly, are based on our understanding of biological neurons. For example: the code has values in each ion channel that store voltages across the membranes. An action potential is modeled by these voltages running along the axons to fire other neurons. I’m personally more familiar with heart models (biomedical engineering background here) but I’m sure it’s similar. In the heart models: calcium, potassium, and sodium concentrations are updated every unit of time, and the differences in concentrations produce voltages.
f_devd 4y ago
Learnable activation functions are a thing famously Swish[0] is is a trainable SiLU which was found through symbolic search/optimization [1], but as it turns out that doesn't magically make make neural networks orders better. [0]: https://en.m.wikipedia.org/wiki/Swish_function https://en.m.wikipedia.org/wiki/Swish_function [1]: https://arxiv.org/abs/1710.05941 https://arxiv.org/abs/1710.05941
uoaei 4y ago
> I’m skeptical that biological systems will ever serve as a basis for ML nets in practice There is no fundamental difference between information processing systems implemented in silico vs in vivo, except architecture. Architecture is what constrains the manifold of internal representations: this is called "inductive bias" in the field of machine learning. The math (technically, the non-equilibrium statistical physics crossed with information theory) is fundamentally the same. Everything at the functionalist level follows from architecture; what enables these functions is the universal principles of information processing per se. "It worked because it worked" because there is no other way for it to work given the initial conditions of our neighborhood in the universe. I'm not saying "Everything ends up looking like a brain". Rather, I am saying "The brain, attendant nervous and sensory systems, etc. vs neural networks implemented as nonlinear functions are running the same instructions on different hardware, thus resulting in different algorithms." The way I like to put it is: trust Nature's engineers, they've been at it much longer than any of us have.
- skibidibipiti 4y ago
  > There is no fundamental difference between information processing in silicon and in vivo A neuron has dozens of neurotransmitters, while artificial neurons produce 1 output. I don't know much about neurology, but how is the information processing similar? What do you mean are running the same instructions? > there is no other way for it to work Plants exhibit learned behaviors
  - uoaei 4y ago
    > how is the information processing similar? The representational capacities are of course not the same -- the same "thoughts" cannot be expressed in both systems. But the concept of "processing over abstract representations enacted in physical dynamics within cognitive systems" is shared between all systems of this kind. I am referring to "information processing" at the physical level, i.e., "'useful' work per energy quantum as communicated through noisy channels". > What do you mean are running the same instructions? The underlying physical principles of such information processing are equivalent regardless of physical implementation. > plants exhibit learned behaviors A good example of what I mean. The architecture is different, but the underlying dynamics is the same. There is a convincing (to me) theory of the origins of life[1][2] that states that thermodynamics -- and, by extension, information theory -- is the appropriate level of abstraction for understanding what distinguishes living processes from inanimate ones. The theory posits that a system, well-defined by some (possibly arbitrary) boundaries, "learns" (develops channels through which "patterns" can be "recognized" and possibly interacted with) as an inevitable result of physics. Put another way, a learning system is one that represents its experiences through the cumulative wearing-in over time of channels of energy flows. What concepts the system can possibly represent depends on in what ways the system can wear while maintaining its essential functions. What specifically the system learns is the set of concepts which collectively best communicate (physically, i.e., from the "inputs" through the "processing" functions and to the "outputs") the historical set of its experiences of its environment and of itself. I want to note that this discussion has nothing to say on perception, only sensation and reaction: in other words, it is an exclusively materialist analysis. Optimization theory describes its notion of learning roughly as such (considering "loss" as energy potentials), but with the same language we could also describe a human brain, or a black hole's accretion disk, or an ant colony dug deep into clay. References: [1] https://www.englandlab.com/uploads/7/8/0/3/7803054/2013jcpsrep.pdf https://www.englandlab.com/uploads/7/8/0/3/7803054/2013jcpsr... [2] https://www.quantamagazine.org/a-new-thermodynamics-theory-of-the-origin-of-life-20140122/ https://www.quantamagazine.org/a-new-thermodynamics-theory-o... Parallel directions of research: https://en.wikipedia.org/wiki/Entropy_and_life https://en.wikipedia.org/wiki/Entropy_and_life https://en.wikipedia.org/wiki/Free_energy_principle https://en.wikipedia.org/wiki/Free_energy_principle
ly3xqhl8g9 4y ago
"I’m skeptical that biological systems will ever serve as a basis for ML nets in practice" First of all, ML engineers need to stop being so brainphiliacs, caring only about the 'neural networks' of the brain or brain-like systems. Lacrymaria olor has more intelligence, in terms of adapting to exploring/exploiting a given environment, than all our artificial neural networks combined and it has no neurons because it is merely a single-cell organism [1]. Once you stop caring about the brain and neurons and you find out that almost every cell in the body has gap junctions and voltage-gated ion channels which for all intents and purposes implement boolean logic and act as transistors for cell-to-cell communication, biology appears less as something which has been overcome and more something towards which we must strive with our primitive technologies: for instance, we can only dream of designing rotary engines as small, powerful, and resilient as the ATP synthase protein [2]. [1] Michael Levin: Intelligence Beyond the Brain, https://youtu.be/RwEKg5cjkKQ?t=202 https://youtu.be/RwEKg5cjkKQ?t=202 [2] Masasuke Yoshida, ATP Synthase. A Marvellous Rotary Engine of the Cell, https://pubmed.ncbi.nlm.nih.gov/11533724 https://pubmed.ncbi.nlm.nih.gov/11533724
- phaedrus 4y ago
  I wonder if there's a step-change where single-celled animals with complex behavior are actually smarter than the simplest multiple-celled animals with a nervous system.
  - whatshisface 4y ago
    The cells of multi-celled animals still have complex behaviors.
- outworlder 4y ago
  Indeed. All cells must do complex computations, by their own nature. Just the process of producing proteins and each of its steps – from 'unrolling' a given DNA section, copying it, reading instructions... even a lowly ribosome is a computer (one that even kinda looks like a Turing machine from a distance)
- mk_stjames 4y ago
danielheath 4y ago
It definitely won’t happen without a massive overhaul of chip design; a design that optimises for very broad connectivity with storage for the connection would be a step in that direction (neural connectivity is on the order of 10k connections each, and the connection stores temporal information about how recently it last fired / how often it’s fired recently)
jononor 4y ago
I think that learning to acquire new/additional training data would be a better first step towards learning agents, than trying to mutate its structure/hyper-parameters.
smrtinsert 4y ago
Is it still a milestone for all NNs?
satvikpendem 4y ago
> so there’s no point in copying it Not sure about that, a lot of solutions in nature are honed by billions of years of evolution, sometimes creating feats even more impressive than we can do currently. There is an entire field about copying biology to solve our problems: https://en.wikipedia.org/wiki/Biomimetics https://en.wikipedia.org/wiki/Biomimetics
mr_toad 4y ago
> It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. You say that like it isn’t a big deal. Finding an analytical solution to optimising the parameters of a non-linear equation is remarkable.
xwolfi 4y ago
Well I tend to agree but you seem to think biology evolved in a vacuum but it evolved inside an information source and we're all the information processors: ML having to process the same information but at scale, in the grand scheme, it will probably have to ressemble a brain in some ways. Just the sources we care about (colors in a picture, faces, prices, wind direction, whatever) and the output we can understand (text, images, sound) will have to skew it towards us in the way it has to model things.
fatneckbeard 4y ago
well i will agree on one thing.... corporations are constantly looking for a machine to do labor for free. life itself did not evolve just to do labor for a corporation so by trying to copy biological intelligent life, the result won't necessarily want to do what you tell it to do or be interested in your profit motives.
mensetmanusman 4y ago
“ which will necessarily be different than nature’s” We are nature’s…
canadianfella 4y ago
[dead]
dd36 4y ago
We are nature. For all we know the solutions that cells came up with were derived in similar ways. Kevin Kelly’s “What Technology Wants” documents how evolution repeats itself in our technology.
dilawar 4y ago
A hell lot of computation inside a neuron (in fact inside any cell) is chemical in nature. Proteins interacting, channels opening and closing, membrane doing membrainy stuff.. In fact their is a AND gate which is entirely chemical in nature. Simulating chemical reactions are slow in silicon therefore chemical side is ignored. If you glance over the graphs in chemistry papers, most of them are sigmoids. sigmoids are the sinusoid of chemical world. Its nice and heartening to see sinusoid appearing often in AI/ML as a fundamental computation.
noobermin 4y ago
So I wasn't skeptical in the way you found it, but it did sound a heck lot to me like tradition numerical solution of PDEs...but with NNs in there somehow.
water-your-self 4y ago
Looking at biology is what lead to CNNs the current AI boom. Thats the whole reason we call multi layered perceptrons as "neural nets" cheesey and flashy as it is, using a sliding filter was inspired by what we know about vision im biology.

f_devd 4y ago

Although the article is recent the paper from the article has been available on preprint/arxiv since June 2021[1], implementations for pytorch & tensorflow are also available[2] for those interested. [1]: https://arxiv.org/abs/2106.13898 https://arxiv.org/abs/2106.13898 [2]: https://github.com/raminmh/CfC https://github.com/raminmh/CfC

lairv 4y ago

Is there any reason to believe that biologically inspired architectures should yield better performance ? Brain are biological systems which have been trained through evolutionary processes. Neural Networks are algorithmic/linear algebra models trained through statistical methods One might argue that CNN are biologically inspired, but it's more likely that the reason they work is because they respects input symmetries

danans 4y ago
> Is there any reason to believe that biologically inspired architectures should yield better performance ? At the very least, they could yield far better efficiency. A 12W brain can achieve more an entire data center of GPUs, depending on what you are trying to achieve. Whether that would make something actually demonstrate sentience level performance is another question.
OkayPhysicist 4y ago
We know, for a fact, that biological brains work. Not only do they work, they work enormously well learning and adapting based on dramatically less available data, utilizing vastly less resources than anything we've conceived of in artificial computing. Biological architectures may not be the best possible, but empirical evidence demonstrates that they can result in intelligences ranging all the way up to sentience.
- DiscourseFan 4y ago
  The question is if its appropriate to compare logical machines, which are built on things secondary, a-posteriori to the primary aspects of human cognition--that is to say logic--with the primary, biological, a-priori aspects of cognition which are in some sense inscrutable. I myself do not believe that we will never be able to comprehensively understand the way in which our minds work, only religion leaves mysteries up to God. But I think that using scientific empirical logic to understand how we are able to perform judgements such as those made with scientific empirical logic will never yield the proper result; judgement itself must be investigated. Something I don't think many researchers in the field of neural-networks are capable of doing.

dvh 4y ago

The old neuroscience saying goes like this: "Human brain have billions of neurons and so it is too complex to understand, that's why neuroscience study simpler organisms. Flatworm's brain have 52 neurons. We have no idea how it works". Did finally something changed in this regard?

rmorey 4y ago
Yes. The C. Elegans brain (~300 neurons) was the first organism to be completely mapped to a connectome (the map of all connections). The first complete connectome of any centralized brain, the fruit fly, is about to be completed by the Flywire project (https://home.flywire.ai/ https://home.flywire.ai/) ~100,000 neurons and ~70,000,000 synapses. We have just a little idea how it works ;)
- mr_toad 4y ago
  A bit of extrapolation might suggested we could map out the connectome of a human brain in 40-50 years. Not that I’d suggest a linear extrapolation from two data points…
  - heavenlyblue 4y ago
    Are you willing to kill as many humans as there were flies killed when doing that?
    - mr_toad 4y ago
      You’d only need a fraction of the 100 million+ people who die every year. There are probably bigger ethical questions when it comes to simulating a human mind.
babblingfish 4y ago
There have been projects to systematically catalog all the synapses in the flatworm. The problem is that neural plasticity means these connections change dynamically over time based on the needs of the organism. Since the only way we can study the flatworm at the synapse level is by killing the worm and mounting it on slides and staining it and viewing it through a high power microscope, we can only analyze its structure at points frozen in time (and formaldehyde). The reason we will never be able to truly model and understand neural networks (irl) is because their plasticity is very difficult to study with our current methods. Not only do the quantity and location of the synapses change, but the concentration and type of neurotransmitters at the synapses change. And on top of that the concentration of the neurotransmitter receptors are constantly being up regulated and down regulated by the receiving neuron. Each of these factors is really important to what the neuron is actually doing. This is why even a simple organism can have basically an unlimited amount of complexity. To understand a dynamic system like this would require very precise measurements of very small particles in vivo which is currently impossible with our tools.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

Kalanos 4y ago

so liquid nets don't need inputs of a fixed length?

akomtu 4y ago

Imo, the next step in ML is unleashing the electron a bit. Right now we keep probabilistic electrons on leash in transistors, so they behave deterministically. This despotic method has taken us far, but without giving electrons some freedom back, we won't advance further.

bGl2YW5j 4y ago
I'm interested; could you expand on this please? How can we give electrons more 'freedom' and what would that result in?
- akomtu 4y ago
  Transistors could have microscopic chambers where electrons could go for a walk and exercise limited freedoms. Their behavior would be modulated by the external magnetic field to prevent riots.

deleted 4y ago

[deleted]

steve76 4y ago

[dead]

d4rkp4ttern 4y ago

Maybe I have missed it but I don’t know why there is not much talk about simulating evolution at high speed: brains evolved over millions of years to adapt to the environment and ensure survival. So instead of trying to understand and reproduce brain structures, we instead simulate evolution of embodied agents ultra high speed and see if some paths lead to brains comparable to today’s organisms.

jamesk_au 4y ago
For a fun exploration of this, see the short story “Crystal Nights”, by Greg Egan: https://www.gregegan.net/MISC/CRYSTAL/Crystal.html https://www.gregegan.net/MISC/CRYSTAL/Crystal.html