12 ms·

Single cortical neurons as deep artificial neural networks

164 points by nabla9 5y ago

cblconfederate 5y ago
"The spiking times of L5 cortical neurons without NMDA receptors can sufficiently be approximated with a 5-8 layer DNN" This isn't as novel as it sounds, previous work has modeled e.g. firing rates with 2 or 3 layer ANNs, nor does it provide some fundamental insight imho
zetalyrae 5y ago
ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastically. And then there's the glia. This is why I dismiss any claims about the computational power of the human brain, most of which seem to begin with the assumption that "1 spike = 1 FLOP", and that'll all that goes on in the brain. [0]: https://faculty.washington.edu/chudler/facts.html https://faculty.washington.edu/chudler/facts.html and https://www.wolframalpha.com/input/?i=1E-06+grams+in+dalton https://www.wolframalpha.com/input/?i=1E-06+grams+in+dalton
- singularity2001 5y ago
  Not to speak of all the 'weights' which are stored internaly in DNA methylation, among other memory forming mechanisms (proven recently). (part of the quadrillion atoms system of course)
  - zetalyrae 5y ago
    And if you need ab-initio quantum chemistry (or equivalent) simulations of every atom in a human brain, the timelines for both AGI and whole brain emulation get pushed far, far into the future. Assuming AGI requires about as much hardware as it'd take to emulate a brain, at least.
    - greens 5y ago
      My 2c (apologies for the aggressive tone -- I'm just excited about AGI): That's a very very weak upper bound on how much hardware it takes. I think it's not all that different from emulating a Nintendo64 with a quantum simulation of the hardware. For complex systems to work (not to mention evolve), they need to be robust to small perturbations -- there's no way the computation the brain is doing is sensitive to the details of particular atoms. There has to be redundancy, modularity, etc. These things aren't human inventions so much as they are the only way to meaningfully move in a 2^|giant-number| state-space.
      - zetalyrae 5y ago
        DNA/RNA is near-atomic level storage. There's redundancy in that there are multiple instances of each neuron|receptor|etc.
        greens 5y ago
        That's a great point. You could argue despite the huge number of physical degrees of freedom, the operations on DNA can be reduced to copy, repair, express, suppress. On the other hand, there's still a ton of intrinsic complexity in storing a huge amount of data, and yeah some nucleotides are totally essential. The other thing a wonder about: sure, maintaining a proteome is hugely complicated, but how much of this complexity goes into maintaining homeostasis (e.g. metabolism, cytoskeloton and membrane maintenance, replication,...) vs. enabling computation. Seems like silicon has the advantage here.
- nabla9 5y ago
  Your terminology is inaccurate. Threshold neurons like Pitts-McCulloch neurons are rarely used in artificial neural network architectures. They can't learn. ANN's uses a nonlinear activation function that is differentiable (at least in practice). That small change makes huge difference.
  - zetalyrae 5y ago
    I stand corrected. But the point is that artificial neurons are computationally simple.
  - l33t2328 5y ago
    > They can't learn Is there a good (i.e. theoretical) reason for this?
    - Arech 5y ago
      Non-differentiability, obviously
    - shmageggy 5y ago
      No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good activation functions but they don't prevent a network from learning.
      - sdenton4 5y ago
        Sorry, but I think I disagree... If I understand the PM neuron, it's outputs are boolean. This means there's no backprop signal, despite differentiability, since the outputs are constant (thus, gradient zero) in any neighborhood, which in turns zeros out any learning signal you would want to backprop through them. So you need a different learning strategy than backprop to use them. (see also: the 'dead neuron' problem/phenomenon with ReLU activations.)
        orbifold 5y ago
- adrianN 5y ago
  Transistors, while dramatically simpler than neurons, also have fairly complicated physics. It's quite hard to determine the computational power of a microprocessor when you only know roughly how transistors work and have no idea how microprocessors work.
  - yarcob 5y ago
    But digital chips are designed to use transistors as on/off switches, so the complicated physics don't matter for understanding a microprocessor. (It's different for analog ICs, where the characteristics of the individual transistors do matter)
    - adrianN 5y ago
      Yes, but you wouldn't know that if the best knowledge you had about the inner workings of microchips is essentially looking which parts get hot while they perform different tasks.
    - titzer 5y ago
      Wouldn't it suck if brains are just reservoirs used by the internet that can store roughly 3 memes at a time?
      - taneq 5y ago
        holds up a Duracell battery "You think thermodynamics means using humans as batteries is a dumb idea? Where do you think you learned thermodynamics?" faint nyan cat music in the background
        cwillu 5y ago
        "May I have a physics textbook?" "No. The real world doesn't run on math."
  - zetalyrae 5y ago
    True, but transistors are designed by humans (for now), so you have an upper bound on complexity as well as a certain degree of modularity. Whereas biology has no such limitations because evolution doesn't care about the understandability of its designs. So I tend to err on the side of biology being more complex than not. But it's absolutely possible that the high-level behaviour is simple while the underlying implementation is complex and chaotic, as in the gas laws.
- IshKebab 5y ago
  Isn't that the point of this research? To find out roughly how many ANN neurons a real neuron corresponds to? The number of atoms is irrelevant.
- nootropicat 5y ago
  To add to that, there's evidence that may mean that memories are stored in DNA, shuffled between neurons in RNA capsids, evolutionary borrowed from retroviruses. That would make each neuron a complicated computer, functionally equivalent to one pc in capability and data storage. "The neuronal gene Arc is essential for long-lasting information storage in the mammalian brain, mediates various forms of synaptic plasticity, and has been implicated in neurodevelopmental disorders. However, little is known about Arc’s molecular function and evolutionary origins. Here, we show that Arc self-assembles into virus-like capsids that encapsulate RNA. Endogenous Arc protein is released from neurons in extracellular vesicles that mediate the transfer of Arc mRNA into new target cells, where it can undergo activity-dependent translation. Purified Arc capsids are endocytosed and are able to transfer Arc mRNA into the cytoplasm of neurons. These results show that Arc exhibits similar molecular properties to retroviral Gag proteins. Evolutionary analysis indicates that Arc is derived from a vertebrate lineage of Ty3/gypsy retrotransposons, which are also ancestors to retroviruses. These findings suggest that Gag retroelements have been repurposed during evolution to mediate intercellular communication in the nervous system." https://www.cell.com/cell/fulltext/S0092-8674(17)31504-0 https://www.cell.com/cell/fulltext/S0092-8674(17)31504-0
  - twic 5y ago
    The passage you quote very much does not amount to "memories are stored in DNA", and as far as i know, memories are not stored in DNA sequence, and would be very surprised if they were. What this mechanism does is take the transcription level of a gene in one cell, and induce a proportional signal in another cell. There are numerous mechanisms which do that in cells. Most of them are nowhere near as weird as Arc, but the net result is similar.
    - caddemon 5y ago
      It's arguable that some memories are stored via epigenetic modifications to DNA, depending on how you define "storing memories". It is very unlikely it is storing memories in the way humans typically think of memories. But I don't think "in DNA" necessarily implies "in DNA sequence".
- causi 5y ago
  Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know. There are hard limits. No matter how you try you can't perfectly simulate three atoms using two atoms. If it turns out we have to, in software, represent fifty percent of neuronal activity to create consciousness we're in real trouble. A dragonfly can take inputs from thousands of ommatidia and use them to track targets in space using only sixteen neurons. How many transistors would it take us to do the same? Take that ratio and apply it to the 86 billion neurons in the human brain and you have a rough idea of what it will take to create strong AI. The numbers aren't promising.
  - sdenton4 5y ago
    /2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know./ I thought this at least was fairly well understood: We do in fact use our whole brains, as anything less would be a fantastic waste of resources, which evolution would have taken care of long ago. We have numerous human-specific adaptations to deal with the relatively massive brains we're carrying around.
    - causi 5y ago
      When you're building an AI you may not need the neurons involved with, for example, breathing. That's what I'm talking about. I'm not a neuroscientist so I don't know for sure whether all the neurons we use for muscle and organ control do double-duty to help us cogitate.
      - sdenton4 5y ago
        Depending on what you think about the AGI-ness of other mammals, this could be helpful to meditate upon: https://en.wikipedia.org/wiki/Brain-to-body_mass_ratio https://en.wikipedia.org/wiki/Brain-to-body_mass_ratio "In animals, it is thought that the larger the brain, the more brain weight will be available for more complex cognitive tasks. However, large animals need more neurons to represent their own bodies and control specific muscles;[clarification needed][citation needed] thus, relative rather than absolute brain size makes for a ranking of animals that better coincides with the observed complexity of animal behaviour. The relationship between brain-to-body mass ratio and complexity of behaviour is not perfect as other factors also influence intelligence, like the evolution of the recent cerebral cortex and different degrees of brain folding,[5] which increase the surface of the cortex, which is positively correlated in humans to intelligence."
- fnord77 5y ago
  what's also interesting is non-neuronal computational abilities. Allosteric regulation on receptors chained together seem like they have computational abilities. And they're on all sorts of cells I'm sure there's others.
- gnramires 5y ago
  > A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms You may be implying there could be information stored in all those atoms, but I'm not sure that's possible. We live in a thermal bath, this means the behavior of atoms is usually stochastic. Their position cannot reliably hold information without dispersing it rapidly into the thermal environment. One way to get around this is to form chemical bonds, like in the DNA, where some kind of order or position is stable. But it necessitates this chemical structure, and importantly it necessitates as well a chemical reading mechanism and a chemical writing mechanism (or some kind of kinetic activation). Most parts of a cell are not prepared for any of that as far as I can tell. Information really should be carried by discrete elements such as neurotransmitters, as well as continuous but temporary (unstable) elements such as electric impulses (potentially caused by complex responses to electric potential and current inside the neuron). Even in the electric case, the fact about information stands; electric state is also encoded in atoms. In other words, we almost certainly don't need the full fidelity to reproduce the behavior of a neuron. I would need more rigorous examination of the neuron to give a confident estimate, but as a rule of thumb the concentration of relevant information everywhere is much less than DNA's (and mostly negligible everywhere) -- certainly a very interesting research program. From a quick googling, E. Coli DNA has about 4.6 x 10^6 bp, so I would be reasonably confident in an upper bound to neuron information as (volume of neuron/volume of e. coli) x 5 x 10^6 bits (i.e. 1 megabit, 125 kb). The reality is probably much less. DNA is so dense because the reading and replication time is relatively slow. It's not made for rapid, random access at the speed of thought. If I were to guess, I'd say long term information is probably retained within the concentration of compounds that can be read electro-chemically. The question of information then is how sensitive the neural system as a whole is to differences in concentration and differences in timing and amplitude of neural impulses. Again given thermal noise in the brain and limitations of amplitude, you can give strict upper bounds on neural communication (I'd be surprised at sensitivities more than a few ppm). So essentially 1 neuron <= log2(distinguishable chemical state) (memory) And also 1 spike <= log2(electrical degrees of freedom) (bandwidth) Again, I'd require more information on spikes, but they carry maybe 20-40 bits at most -- so not more than a double f.p., although in ANNs again because of low sensitivities due to architecture most of LSB information doesn't contribute significantly to the computation (whereas the brain could multiplex information more effectively). So it's still likely in the order of 1 spike <= 10 flops.
- api 5y ago
  Older estimates are probably short by multiple orders of magnitude. We can see this in practice by looking at e.g. a self-driving Tesla vs. a mouse. Watch a frightened mouse run across an uneven landscape, climbing and clambering and avoiding obstacles as it controls four independent limbs and countless small muscles in real time. A Tesla's auto-drive is nowhere near that good in spite of having only a few levers to control: accelerate, brake, left, and right. It also has far better sensors than the mouse including better eyesight, a wider field of vision, etc., and last I checked mice do not have access to a cellular network supplying them with a heads-up macro view of the local environment. They don't have "fleet-wide learning" either. The Tesla's AI uses over a hundred watts of power. The mouse's brain uses milliwatts. As I wrote this my brain was consuming between 30 and 60 watts. My laptop peaks out at 80. We are not even close to what biological neurons accomplish in raw compute, and while we are getting pretty good at training giant regression models that we call AI I am not convinced we really understand things at the algorithmic level yet either.
  - datameta 5y ago
    Perhaps Spot from Boston Dynamic is a more fair SotA comparison. Though even then, the magnitude of scale is off and still the gulf in ability is wide. I wonder how our artificial insects stack up in comparison to organic ones computationally. I think there is much to learn from the bee brain, for instance. Relevant read: "The thermodynamic efficiency of computations made in cells across the range of life" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/
- taneq 5y ago
  > A biological neuron is 6e17 Daltons[0] A grain of sand is around 6.02E19 Daltons, imagine how much computational power there is in a sand castle! On the other hand, snails have been shown to use just two neurons to generate their complex (relatively, I mean they're snails after all) feeding behaviour with just two neurons: https://neurosciencenews.com/neurons-decision-making-4370/ https://neurosciencenews.com/neurons-decision-making-4370/ (Veering slightly offtopic here but this is the first I've heard the term 'Dalton', I've always just heard them called AMUs.)
londons_explore 5y ago
This is trying to train a neural network to match the behaviour of various mathematical models of biological neurons. Said models still don't seem to be very close to being able to predict the behaviour of biological neurons though. If they did, then we'd have invented artificial human-level intelligence.
- hoseja 5y ago
  That doesn't follow at all. Human neurons aren't much different from mouse neurons, maybe chicken neurons, mosquito neurons. After you faithfully model a neuron you still need untold myriads of them and their interconnects to get human brain.
  - raducu 5y ago
    Also, sensations and feelings are not a logical/mathematical byproduct of the neurons; no matter how "well" you simulate "neurons", feelings and sensations will not emerge.
    - hoseja 5y ago
      I don't think we have the faintest clue what subjective experience of self (whatever you call it, qualia?) actually is, to be able to say it isn't artificially reproducible.
      - raducu 5y ago
        We know that computers are absolutely not the kind of machinery to produce sensations. We know non-biological organisms don't produce qualia. We know it involves chemical reactions, because we can change qualia with drugs.
    - simiones 5y ago
      Unless you believe in a transcendent soul that could be the source of these sensations or feelings, this assertion doesn't make sense. Assuming there is no supernatural soul, it's logically impossible for anything humans experience to not arise from the human body. This entire notion of qualia is a philosophical quagmire predicated on the idea that if you can imagine something, it must be true ("we can imagine a zombie that behaves exactly like a human, but doesn't have qualia at all"). It's actually as laughable as the "argument from perfection" for the existence of a god.
Qu3tzal 5y ago
5-8 layers but how many weights per layer? I can’t access the article but it says it’s a deep CNN so probably in the order of a few thousands weights per layer.
- in3d 5y ago
  Up to 256.
  - danwills 5y ago
    Is that 256 weights per-cell per-layer, or 256 in total? Is "how many cells in each layer" a valid question too?
    - in3d 5y ago
      Their pretrained 128-wide net has a total of 9.2 million parameters: https://www.kaggle.com/selfishgene/single-neuron-as-deep-net-replicating-key-result https://www.kaggle.com/selfishgene/single-neuron-as-deep-net...
      - bmh100 5y ago
        9.2 million parameters per neuron. There an estimated 86 billion neurons in the human brain [0] and 19 billion in the neocortex [1]. That means that, for this strategy to emulate a human brain or neocortex would require 791 quadrillion parameters or 175 quadrillion parameters, respectively. The largest ANN built so far, GPT-3, has 175 billion parameters [2]. We are 6 orders of magnitude from being able to pull it off. [0]: https://en.wikipedia.org/wiki/Human_brain https://en.wikipedia.org/wiki/Human_brain [1]: https://pubmed.ncbi.nlm.nih.gov/9215725/ https://pubmed.ncbi.nlm.nih.gov/9215725/ [2]: https://siliconangle.com/2021/08/04/microsoft-researchers-reveal-neural-network-135b-parameters/ https://siliconangle.com/2021/08/04/microsoft-researchers-re...
        mattkrause 5y ago
        This is assuming there's no weight sharing. Some of those weights presumably go to reproducing highly-conserved features, like the kinetics of particular ion channels. These are "tied" via the genome, in the sense that there's one KCNC1 gene, but millions of neurons express the ion channel it encodes. On the other hand, this model is also missing all sorts of other interactions: hormones and other neuromodulators, ephaptic coupling, etc. It's so complicated I would venture that no one even has a reasonable guesstimate of how close we are, beyond "Not very."
freemint 5y ago
> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overhead". Does someone know of studies of this emulation overhead, even outside biology? Even between ARM and x86 there is an emulation overhead due to different memory models while both are register machines.
- laurent92 5y ago
  That’s true, the brain uses a VM to run maths or science, only the best scientists succeed at understanding some of the concepts natively. Same for music: A student runs the music sheet in a VM, and progressively JIT makes the movements native, which allows much faster execution, and which allows building on top of the base layer. Maybe we’re doing it all wrong writing programs in assembler. We should give them to a VM, the VM should see the similarity between various pieces of the programs, make them inline, and we could teach the machine faster. </daydreaming abstract=“true”>
  - titzer 5y ago
    > Maybe we’re doing it all wrong writing programs in assembler. We should give them to a VM, This is what compilers do. Their input is a program in a more abstract language, either bytecode, an intermediate representation, or a source language. The problem is that damn undecidability, which is like a minefield of rakes. It's undecidable for a compiler to tell if a program will do anything (e.g. halt). It's undecidable for a compiler to tell if two programs are equivalent. It's undecidable for a compiler to tell if a program is minimal. So compilers have to well, be dumber. They approximate a lot.
    - freemint 5y ago
      In fact finding the for loops to do tensor contractions (think matrix multiply but with many more dimensions) alone was something in NP range. Converting for loops to assembly as is done by https://polly.llvm.org/ https://polly.llvm.org/ is equivalent to Mixed-Integer Linear Programming, is equivalent to MaxSat is equivalent to Sat in a for a loop. In these domains there is a definition of minimal and they are still hard.
MAXPOOL 5y ago
Biology/biochemistry can squeeze a good amount of computation from just 20 watts. Even if we could get equal computation from silicon and have an algorithm to run near human-level general AI, humans can maintain the comparative advantage as long as the cost of hardware is more than the cost of raising and educating a human, and the operating cost is more than wages for the same task.
- vlovich123 5y ago
  Human-level intelligence is highly variable. If we’re talking about something that can intelligently and independently make discoveries in math science and engineering, then the comparative advantage can’t stay long because you’d just pose the problem of “make yourself but faster” to it. It doesn’t even need discoveries. Imagine an AI that could manually transcode your high level code into absolutely optimal assembly, simplifying your design, removing unnecessary code, optimizing the code in response to observed behavior, etc. I would guess we have several orders of magnitude of power efficiency loss just from building a system that has understandable and flexible layers of abstraction. A sufficiently powerful AI could work to automatically remove those abstractions and even operate at a higher level of abstraction. The real question is whether we’re at all on the right track. ¯\_(ツ)_/¯
  - adrianN 5y ago
    > Imagine an AI that could manually transcode your high level code into absolutely optimal assembly The Halting Problem prevents such an AI from existing, but your point still stands of course.
    - vlovich123 5y ago
      How so?
      - adrianN 5y ago
        Suppose you write a program that simulates a TM and then prints "Halt" when the TM halts. The magic AI could optimize this program to just a print (or an infinite loop). This requires solving the halting problem.
JulianMorrison 5y ago
I'm not sure if I'm reading this right, but, they made a predictive model of a biological neuron that works? Setting aside the how, the achievement is also a thing. How much work would it take to transcode, say, a nematode into assemblages of these things?
- cblconfederate 5y ago
  Compartmental modeling is the standard method for simulating neural tissue by breaking it down to electrical cylinders. There is already openworm.org which is a full cell-by-cell simulation of the c.elegans Larger projects like the Human Brain Project exist, but despite the existence of large scale compartmental simulations, we ve gained little insight about how brains work
- mr_toad 5y ago
  > How much work would it take to transcode, say, a nematode into assemblages of these things? You also need to find a way to simulate input (an environment) for this simulated “brain”, otherwise I suspect it would suffer the problems associated with sensory deprivation.
tambourine_man 5y ago
The recent episode of the Lex Fridman podcast with Jeff Hawkins touches on this issue of the computational capabilities of single neurons: https://youtube.com/watch?v=Z1KwkpTUbkg https://youtube.com/watch?v=Z1KwkpTUbkg Great episode all around.
- dwd 5y ago
  He also appeared recently on Sam Harris's podcast to a bit more more skepticism. I thought he didn't break much new ground from his previous appearance on Lex's podcast in 2019, which I went back and listened to before listening to the latest one. But...in saying that - listening to him again triggered a multitude of aha moments for me, and got me thinking about things like: "what is a dream", "how does deja-vu work" in the context of cortical columns, 1000 brain theory, predictive voting, etc. I though his most recent explanation on how creating an artificial neocortex is not a risk in itself, but rather what you ask it to do. He was also adamant that giving an AI the ability to self-replicate should be where we draw the line. This was very interesting when they previously discussed that topic but was months before SARS-CoV-2: https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/jeff-hawkins-thousand-brains-theory-of-intelligence#paragraph_7047 https://www.happyscribe.com/public/lex-fridman-podcast-artif...
joe_the_user 5y ago
Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers This is a link to the abstract so you don't get to find out the width of these networks. But given that neural networks are universal approximators, it seems to me that all that's being said is a neuron is a very complicated thing. And so the phrasing seems to give an unjustified impression you still best think of a biological neural network as just a larger artificial neural network.
armatav 5y ago
Finally someone understands!

164 points by nabla9 5y ago

"The spiking times of L5 cortical neurons without NMDA receptors can sufficiently be approximated with a 5-8 layer DNN" This isn't as novel as it sounds, previous work has modeled e.g. firing rates with 2 or 3 layer ANNs, nor does it provide some fundamental insight imho

zetalyrae 5y ago

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastically. And then there's the glia. This is why I dismiss any claims about the computational power of the human brain, most of which seem to begin with the assumption that "1 spike = 1 FLOP", and that'll all that goes on in the brain. [0]: https://faculty.washington.edu/chudler/facts.html https://faculty.washington.edu/chudler/facts.html and https://www.wolframalpha.com/input/?i=1E-06+grams+in+dalton https://www.wolframalpha.com/input/?i=1E-06+grams+in+dalton

singularity2001 5y ago
Not to speak of all the 'weights' which are stored internaly in DNA methylation, among other memory forming mechanisms (proven recently). (part of the quadrillion atoms system of course)
- zetalyrae 5y ago
  And if you need ab-initio quantum chemistry (or equivalent) simulations of every atom in a human brain, the timelines for both AGI and whole brain emulation get pushed far, far into the future. Assuming AGI requires about as much hardware as it'd take to emulate a brain, at least.
  - greens 5y ago
    My 2c (apologies for the aggressive tone -- I'm just excited about AGI): That's a very very weak upper bound on how much hardware it takes. I think it's not all that different from emulating a Nintendo64 with a quantum simulation of the hardware. For complex systems to work (not to mention evolve), they need to be robust to small perturbations -- there's no way the computation the brain is doing is sensitive to the details of particular atoms. There has to be redundancy, modularity, etc. These things aren't human inventions so much as they are the only way to meaningfully move in a 2^|giant-number| state-space.
    - zetalyrae 5y ago
      DNA/RNA is near-atomic level storage. There's redundancy in that there are multiple instances of each neuron|receptor|etc.
      - greens 5y ago
        That's a great point. You could argue despite the huge number of physical degrees of freedom, the operations on DNA can be reduced to copy, repair, express, suppress. On the other hand, there's still a ton of intrinsic complexity in storing a huge amount of data, and yeah some nucleotides are totally essential. The other thing a wonder about: sure, maintaining a proteome is hugely complicated, but how much of this complexity goes into maintaining homeostasis (e.g. metabolism, cytoskeloton and membrane maintenance, replication,...) vs. enabling computation. Seems like silicon has the advantage here.
nabla9 5y ago
Your terminology is inaccurate. Threshold neurons like Pitts-McCulloch neurons are rarely used in artificial neural network architectures. They can't learn. ANN's uses a nonlinear activation function that is differentiable (at least in practice). That small change makes huge difference.
- zetalyrae 5y ago
  I stand corrected. But the point is that artificial neurons are computationally simple.
- l33t2328 5y ago
  > They can't learn Is there a good (i.e. theoretical) reason for this?
  - Arech 5y ago
    Non-differentiability, obviously
  - shmageggy 5y ago
    No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good activation functions but they don't prevent a network from learning.
    - sdenton4 5y ago
      Sorry, but I think I disagree... If I understand the PM neuron, it's outputs are boolean. This means there's no backprop signal, despite differentiability, since the outputs are constant (thus, gradient zero) in any neighborhood, which in turns zeros out any learning signal you would want to backprop through them. So you need a different learning strategy than backprop to use them. (see also: the 'dead neuron' problem/phenomenon with ReLU activations.)
      - orbifold 5y ago
adrianN 5y ago
Transistors, while dramatically simpler than neurons, also have fairly complicated physics. It's quite hard to determine the computational power of a microprocessor when you only know roughly how transistors work and have no idea how microprocessors work.
- yarcob 5y ago
  But digital chips are designed to use transistors as on/off switches, so the complicated physics don't matter for understanding a microprocessor. (It's different for analog ICs, where the characteristics of the individual transistors do matter)
  - adrianN 5y ago
    Yes, but you wouldn't know that if the best knowledge you had about the inner workings of microchips is essentially looking which parts get hot while they perform different tasks.
  - titzer 5y ago
    Wouldn't it suck if brains are just reservoirs used by the internet that can store roughly 3 memes at a time?
    - taneq 5y ago
      holds up a Duracell battery "You think thermodynamics means using humans as batteries is a dumb idea? Where do you think you learned thermodynamics?" faint nyan cat music in the background
      - cwillu 5y ago
        "May I have a physics textbook?" "No. The real world doesn't run on math."
- zetalyrae 5y ago
  True, but transistors are designed by humans (for now), so you have an upper bound on complexity as well as a certain degree of modularity. Whereas biology has no such limitations because evolution doesn't care about the understandability of its designs. So I tend to err on the side of biology being more complex than not. But it's absolutely possible that the high-level behaviour is simple while the underlying implementation is complex and chaotic, as in the gas laws.
IshKebab 5y ago
Isn't that the point of this research? To find out roughly how many ANN neurons a real neuron corresponds to? The number of atoms is irrelevant.
nootropicat 5y ago
To add to that, there's evidence that may mean that memories are stored in DNA, shuffled between neurons in RNA capsids, evolutionary borrowed from retroviruses. That would make each neuron a complicated computer, functionally equivalent to one pc in capability and data storage. "The neuronal gene Arc is essential for long-lasting information storage in the mammalian brain, mediates various forms of synaptic plasticity, and has been implicated in neurodevelopmental disorders. However, little is known about Arc’s molecular function and evolutionary origins. Here, we show that Arc self-assembles into virus-like capsids that encapsulate RNA. Endogenous Arc protein is released from neurons in extracellular vesicles that mediate the transfer of Arc mRNA into new target cells, where it can undergo activity-dependent translation. Purified Arc capsids are endocytosed and are able to transfer Arc mRNA into the cytoplasm of neurons. These results show that Arc exhibits similar molecular properties to retroviral Gag proteins. Evolutionary analysis indicates that Arc is derived from a vertebrate lineage of Ty3/gypsy retrotransposons, which are also ancestors to retroviruses. These findings suggest that Gag retroelements have been repurposed during evolution to mediate intercellular communication in the nervous system." https://www.cell.com/cell/fulltext/S0092-8674(17)31504-0 https://www.cell.com/cell/fulltext/S0092-8674(17)31504-0
- twic 5y ago
  The passage you quote very much does not amount to "memories are stored in DNA", and as far as i know, memories are not stored in DNA sequence, and would be very surprised if they were. What this mechanism does is take the transcription level of a gene in one cell, and induce a proportional signal in another cell. There are numerous mechanisms which do that in cells. Most of them are nowhere near as weird as Arc, but the net result is similar.
  - caddemon 5y ago
    It's arguable that some memories are stored via epigenetic modifications to DNA, depending on how you define "storing memories". It is very unlikely it is storing memories in the way humans typically think of memories. But I don't think "in DNA" necessarily implies "in DNA sequence".
causi 5y ago
Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know. There are hard limits. No matter how you try you can't perfectly simulate three atoms using two atoms. If it turns out we have to, in software, represent fifty percent of neuronal activity to create consciousness we're in real trouble. A dragonfly can take inputs from thousands of ommatidia and use them to track targets in space using only sixteen neurons. How many transistors would it take us to do the same? Take that ratio and apply it to the 86 billion neurons in the human brain and you have a rough idea of what it will take to create strong AI. The numbers aren't promising.
- sdenton4 5y ago
  /2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know./ I thought this at least was fairly well understood: We do in fact use our whole brains, as anything less would be a fantastic waste of resources, which evolution would have taken care of long ago. We have numerous human-specific adaptations to deal with the relatively massive brains we're carrying around.
  - causi 5y ago
    When you're building an AI you may not need the neurons involved with, for example, breathing. That's what I'm talking about. I'm not a neuroscientist so I don't know for sure whether all the neurons we use for muscle and organ control do double-duty to help us cogitate.
    - sdenton4 5y ago
      Depending on what you think about the AGI-ness of other mammals, this could be helpful to meditate upon: https://en.wikipedia.org/wiki/Brain-to-body_mass_ratio https://en.wikipedia.org/wiki/Brain-to-body_mass_ratio "In animals, it is thought that the larger the brain, the more brain weight will be available for more complex cognitive tasks. However, large animals need more neurons to represent their own bodies and control specific muscles;[clarification needed][citation needed] thus, relative rather than absolute brain size makes for a ranking of animals that better coincides with the observed complexity of animal behaviour. The relationship between brain-to-body mass ratio and complexity of behaviour is not perfect as other factors also influence intelligence, like the evolution of the recent cerebral cortex and different degrees of brain folding,[5] which increase the surface of the cortex, which is positively correlated in humans to intelligence."
fnord77 5y ago
what's also interesting is non-neuronal computational abilities. Allosteric regulation on receptors chained together seem like they have computational abilities. And they're on all sorts of cells I'm sure there's others.
gnramires 5y ago
> A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms You may be implying there could be information stored in all those atoms, but I'm not sure that's possible. We live in a thermal bath, this means the behavior of atoms is usually stochastic. Their position cannot reliably hold information without dispersing it rapidly into the thermal environment. One way to get around this is to form chemical bonds, like in the DNA, where some kind of order or position is stable. But it necessitates this chemical structure, and importantly it necessitates as well a chemical reading mechanism and a chemical writing mechanism (or some kind of kinetic activation). Most parts of a cell are not prepared for any of that as far as I can tell. Information really should be carried by discrete elements such as neurotransmitters, as well as continuous but temporary (unstable) elements such as electric impulses (potentially caused by complex responses to electric potential and current inside the neuron). Even in the electric case, the fact about information stands; electric state is also encoded in atoms. In other words, we almost certainly don't need the full fidelity to reproduce the behavior of a neuron. I would need more rigorous examination of the neuron to give a confident estimate, but as a rule of thumb the concentration of relevant information everywhere is much less than DNA's (and mostly negligible everywhere) -- certainly a very interesting research program. From a quick googling, E. Coli DNA has about 4.6 x 10^6 bp, so I would be reasonably confident in an upper bound to neuron information as (volume of neuron/volume of e. coli) x 5 x 10^6 bits (i.e. 1 megabit, 125 kb). The reality is probably much less. DNA is so dense because the reading and replication time is relatively slow. It's not made for rapid, random access at the speed of thought. If I were to guess, I'd say long term information is probably retained within the concentration of compounds that can be read electro-chemically. The question of information then is how sensitive the neural system as a whole is to differences in concentration and differences in timing and amplitude of neural impulses. Again given thermal noise in the brain and limitations of amplitude, you can give strict upper bounds on neural communication (I'd be surprised at sensitivities more than a few ppm). So essentially 1 neuron <= log2(distinguishable chemical state) (memory) And also 1 spike <= log2(electrical degrees of freedom) (bandwidth) Again, I'd require more information on spikes, but they carry maybe 20-40 bits at most -- so not more than a double f.p., although in ANNs again because of low sensitivities due to architecture most of LSB information doesn't contribute significantly to the computation (whereas the brain could multiplex information more effectively). So it's still likely in the order of 1 spike <= 10 flops.
api 5y ago
Older estimates are probably short by multiple orders of magnitude. We can see this in practice by looking at e.g. a self-driving Tesla vs. a mouse. Watch a frightened mouse run across an uneven landscape, climbing and clambering and avoiding obstacles as it controls four independent limbs and countless small muscles in real time. A Tesla's auto-drive is nowhere near that good in spite of having only a few levers to control: accelerate, brake, left, and right. It also has far better sensors than the mouse including better eyesight, a wider field of vision, etc., and last I checked mice do not have access to a cellular network supplying them with a heads-up macro view of the local environment. They don't have "fleet-wide learning" either. The Tesla's AI uses over a hundred watts of power. The mouse's brain uses milliwatts. As I wrote this my brain was consuming between 30 and 60 watts. My laptop peaks out at 80. We are not even close to what biological neurons accomplish in raw compute, and while we are getting pretty good at training giant regression models that we call AI I am not convinced we really understand things at the algorithmic level yet either.
- datameta 5y ago
  Perhaps Spot from Boston Dynamic is a more fair SotA comparison. Though even then, the magnitude of scale is off and still the gulf in ability is wide. I wonder how our artificial insects stack up in comparison to organic ones computationally. I think there is much to learn from the bee brain, for instance. Relevant read: "The thermodynamic efficiency of computations made in cells across the range of life" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/
taneq 5y ago
> A biological neuron is 6e17 Daltons[0] A grain of sand is around 6.02E19 Daltons, imagine how much computational power there is in a sand castle! On the other hand, snails have been shown to use just two neurons to generate their complex (relatively, I mean they're snails after all) feeding behaviour with just two neurons: https://neurosciencenews.com/neurons-decision-making-4370/ https://neurosciencenews.com/neurons-decision-making-4370/ (Veering slightly offtopic here but this is the first I've heard the term 'Dalton', I've always just heard them called AMUs.)

londons_explore 5y ago

This is trying to train a neural network to match the behaviour of various mathematical models of biological neurons. Said models still don't seem to be very close to being able to predict the behaviour of biological neurons though. If they did, then we'd have invented artificial human-level intelligence.

hoseja 5y ago
That doesn't follow at all. Human neurons aren't much different from mouse neurons, maybe chicken neurons, mosquito neurons. After you faithfully model a neuron you still need untold myriads of them and their interconnects to get human brain.
- raducu 5y ago
  Also, sensations and feelings are not a logical/mathematical byproduct of the neurons; no matter how "well" you simulate "neurons", feelings and sensations will not emerge.
  - hoseja 5y ago
    I don't think we have the faintest clue what subjective experience of self (whatever you call it, qualia?) actually is, to be able to say it isn't artificially reproducible.
    - raducu 5y ago
      We know that computers are absolutely not the kind of machinery to produce sensations. We know non-biological organisms don't produce qualia. We know it involves chemical reactions, because we can change qualia with drugs.
  - simiones 5y ago
    Unless you believe in a transcendent soul that could be the source of these sensations or feelings, this assertion doesn't make sense. Assuming there is no supernatural soul, it's logically impossible for anything humans experience to not arise from the human body. This entire notion of qualia is a philosophical quagmire predicated on the idea that if you can imagine something, it must be true ("we can imagine a zombie that behaves exactly like a human, but doesn't have qualia at all"). It's actually as laughable as the "argument from perfection" for the existence of a god.

Qu3tzal 5y ago

5-8 layers but how many weights per layer? I can’t access the article but it says it’s a deep CNN so probably in the order of a few thousands weights per layer.

in3d 5y ago
Up to 256.
- danwills 5y ago
  Is that 256 weights per-cell per-layer, or 256 in total? Is "how many cells in each layer" a valid question too?
  - in3d 5y ago
    Their pretrained 128-wide net has a total of 9.2 million parameters: https://www.kaggle.com/selfishgene/single-neuron-as-deep-net-replicating-key-result https://www.kaggle.com/selfishgene/single-neuron-as-deep-net...
    - bmh100 5y ago
      9.2 million parameters per neuron. There an estimated 86 billion neurons in the human brain [0] and 19 billion in the neocortex [1]. That means that, for this strategy to emulate a human brain or neocortex would require 791 quadrillion parameters or 175 quadrillion parameters, respectively. The largest ANN built so far, GPT-3, has 175 billion parameters [2]. We are 6 orders of magnitude from being able to pull it off. [0]: https://en.wikipedia.org/wiki/Human_brain https://en.wikipedia.org/wiki/Human_brain [1]: https://pubmed.ncbi.nlm.nih.gov/9215725/ https://pubmed.ncbi.nlm.nih.gov/9215725/ [2]: https://siliconangle.com/2021/08/04/microsoft-researchers-reveal-neural-network-135b-parameters/ https://siliconangle.com/2021/08/04/microsoft-researchers-re...
      - mattkrause 5y ago
        This is assuming there's no weight sharing. Some of those weights presumably go to reproducing highly-conserved features, like the kinetics of particular ion channels. These are "tied" via the genome, in the sense that there's one KCNC1 gene, but millions of neurons express the ion channel it encodes. On the other hand, this model is also missing all sorts of other interactions: hormones and other neuromodulators, ephaptic coupling, etc. It's so complicated I would venture that no one even has a reasonable guesstimate of how close we are, beyond "Not very."

freemint 5y ago

> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overhead". Does someone know of studies of this emulation overhead, even outside biology? Even between ARM and x86 there is an emulation overhead due to different memory models while both are register machines.

laurent92 5y ago
That’s true, the brain uses a VM to run maths or science, only the best scientists succeed at understanding some of the concepts natively. Same for music: A student runs the music sheet in a VM, and progressively JIT makes the movements native, which allows much faster execution, and which allows building on top of the base layer. Maybe we’re doing it all wrong writing programs in assembler. We should give them to a VM, the VM should see the similarity between various pieces of the programs, make them inline, and we could teach the machine faster. </daydreaming abstract=“true”>
- titzer 5y ago
  > Maybe we’re doing it all wrong writing programs in assembler. We should give them to a VM, This is what compilers do. Their input is a program in a more abstract language, either bytecode, an intermediate representation, or a source language. The problem is that damn undecidability, which is like a minefield of rakes. It's undecidable for a compiler to tell if a program will do anything (e.g. halt). It's undecidable for a compiler to tell if two programs are equivalent. It's undecidable for a compiler to tell if a program is minimal. So compilers have to well, be dumber. They approximate a lot.
  - freemint 5y ago
    In fact finding the for loops to do tensor contractions (think matrix multiply but with many more dimensions) alone was something in NP range. Converting for loops to assembly as is done by https://polly.llvm.org/ https://polly.llvm.org/ is equivalent to Mixed-Integer Linear Programming, is equivalent to MaxSat is equivalent to Sat in a for a loop. In these domains there is a definition of minimal and they are still hard.

MAXPOOL 5y ago

Biology/biochemistry can squeeze a good amount of computation from just 20 watts. Even if we could get equal computation from silicon and have an algorithm to run near human-level general AI, humans can maintain the comparative advantage as long as the cost of hardware is more than the cost of raising and educating a human, and the operating cost is more than wages for the same task.

vlovich123 5y ago
Human-level intelligence is highly variable. If we’re talking about something that can intelligently and independently make discoveries in math science and engineering, then the comparative advantage can’t stay long because you’d just pose the problem of “make yourself but faster” to it. It doesn’t even need discoveries. Imagine an AI that could manually transcode your high level code into absolutely optimal assembly, simplifying your design, removing unnecessary code, optimizing the code in response to observed behavior, etc. I would guess we have several orders of magnitude of power efficiency loss just from building a system that has understandable and flexible layers of abstraction. A sufficiently powerful AI could work to automatically remove those abstractions and even operate at a higher level of abstraction. The real question is whether we’re at all on the right track. ¯\_(ツ)_/¯
- adrianN 5y ago
  > Imagine an AI that could manually transcode your high level code into absolutely optimal assembly The Halting Problem prevents such an AI from existing, but your point still stands of course.
  - vlovich123 5y ago
    How so?
    - adrianN 5y ago
      Suppose you write a program that simulates a TM and then prints "Halt" when the TM halts. The magic AI could optimize this program to just a print (or an infinite loop). This requires solving the halting problem.

JulianMorrison 5y ago

I'm not sure if I'm reading this right, but, they made a predictive model of a biological neuron that works? Setting aside the how, the achievement is also a thing. How much work would it take to transcode, say, a nematode into assemblages of these things?

cblconfederate 5y ago
Compartmental modeling is the standard method for simulating neural tissue by breaking it down to electrical cylinders. There is already openworm.org which is a full cell-by-cell simulation of the c.elegans Larger projects like the Human Brain Project exist, but despite the existence of large scale compartmental simulations, we ve gained little insight about how brains work
mr_toad 5y ago
> How much work would it take to transcode, say, a nematode into assemblages of these things? You also need to find a way to simulate input (an environment) for this simulated “brain”, otherwise I suspect it would suffer the problems associated with sensory deprivation.

tambourine_man 5y ago

The recent episode of the Lex Fridman podcast with Jeff Hawkins touches on this issue of the computational capabilities of single neurons: https://youtube.com/watch?v=Z1KwkpTUbkg https://youtube.com/watch?v=Z1KwkpTUbkg Great episode all around.

dwd 5y ago
He also appeared recently on Sam Harris's podcast to a bit more more skepticism. I thought he didn't break much new ground from his previous appearance on Lex's podcast in 2019, which I went back and listened to before listening to the latest one. But...in saying that - listening to him again triggered a multitude of aha moments for me, and got me thinking about things like: "what is a dream", "how does deja-vu work" in the context of cortical columns, 1000 brain theory, predictive voting, etc. I though his most recent explanation on how creating an artificial neocortex is not a risk in itself, but rather what you ask it to do. He was also adamant that giving an AI the ability to self-replicate should be where we draw the line. This was very interesting when they previously discussed that topic but was months before SARS-CoV-2: https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/jeff-hawkins-thousand-brains-theory-of-intelligence#paragraph_7047 https://www.happyscribe.com/public/lex-fridman-podcast-artif...

joe_the_user 5y ago

Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers This is a link to the abstract so you don't get to find out the width of these networks. But given that neural networks are universal approximators, it seems to me that all that's being said is a neuron is a very complicated thing. And so the phrasing seems to give an unjustified impression you still best think of a biological neural network as just a larger artificial neural network.

armatav 5y ago

Finally someone understands!