10 ms·
Computational Power Found in the Arms of Neurons
- tudorw 7y ago“Brains may be far more complicated than we think,” as said by anyone who has worked with brains! Good stuff though, very interesting :)
- prox 7y agoAs said by a brain studying brains
- tudorw 7y agoDon't get me started, if a complex brain can finally understand the brain, does that not mean we will still be missing the special sauce that was the cause of the complex comprehension, er, I'm out!
- prox 7y agoThis is why in some philosophies, there is a big difference between experiential knowledge and awareness and empirical ‘measurement’ knowledge.
- deleted 7y ago[deleted]
- mrow84 7y agoOT, but "They obtained slices of brain tissue from layers 2 and 3 of the human cortex" has a slightly uncomfortable air to it.
- leemailll 7y agohttps://www.youtube.com/watch?v=Zj3RxtJ_Ljc https://www.youtube.com/watch?v=Zj3RxtJ_Ljc
- haffi112 7y agoI wonder if this is also the case for neurons in other animals. Probably it is, but it would be a stunning revelation if it is not the case.
- buboard 7y agonotably this kind of dendritic spikes has not been found in other animals which have been investigated for dendritic spikes extensively (mice, rats, monkeys)
- chmod775 7y agoFrom the top reddit comment: > This paper is amazing. What is missing from the description above is that this is the first example of how human neurons are qualitatively different than rodent neurons (not only more computation power, but categorically different computation). > ELI5: the way the biological human neuron implements XOR is by a formerly unknown type of local response to inputs, which is low below the threshold, maximal at the threshold and decreases as the input intensifies above the threshold. We never saw anything like that in any other animal. (link to the relevant figure from the paper) Rodents being presumably the go-to non-human object of study.
- dr_dshiv 7y agoOther differences of human neurons. I saw a presentation recently that said that populations of human neurons can synchronize to far higher frequencies than mouse neurons. Into the thousands kd hertz. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5100995/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5100995/
- dharma1 7y agoCould you link the Reddit comment please?
- chmod775 7y agoOh. The link was changed on HN. Here is the reddit comment: https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_single_biological_neuron_can_compute_xor/fcx1ocb/ https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_s...
- ASlave2Gravity 7y agoSo does this point to us being biological machines? Is it already accepted that we are biological machines?
- kaffeemitsahne 7y agoI would guess that this is near-universally accepted among neuroscientists, and people in the exact sciences in general. Although I don't see how it would make a difference to this question where in the brain the computational power arises from.
- valvar 7y agoYou'll have to be more specific with what you mean by "machines". In what ways would machines be different from non-machines?
- ASlave2Gravity 7y agoThat's a good question. I guess I'd say a Turing machine is my definition of a machine. And I suppose the implication is that if we are machines, operating on a stream of input, does that make us deterministic? EDIT: What I'm ultimately interested in is if we, as humans, are operating in a realm that logic cannot operate in. If that makes any sense? I'll probably have to define realm!
- Elbrasch 7y agoAs always, being deterministic may make a system predictable in theory, through in practice there are to many variables for it to matter. To soften the existential dread.
- ASlave2Gravity 7y agoExistential dread is most definitely softened! Thank you.
- nabla9 7y ago
- bonoboTP 7y agoKnowing little about neuroscience outside of biology class, I've always wondered why neurons would be so simple as to just compute weighted sums and thresholds. I mean there are unicellular creatures that can do incredibly complex things, and neurons, as all cells, are ultimately the descendants of such unicellular organisms, "teamed up" into a multi-cellular creature. They have a lot of internal structure, organelles, whatnot. Modeling them as some simple switches seems a stretch. My point is, overall this doesn't seem so surprising to me as a layman. (Which doesn't mean more than just literally that. I'm not claiming laypeople should decide scientific questions from their armchairs.)
- svara 7y agoThis is an intuition that many neuroscientists have shared, it's just that direct experimental evidence of 1) it happening at all in a non-trivial way and 2) it being relevant to the behavior of an animal is somewhat scarce. Plus, the sum and threshold model actually gets you pretty far, as evidenced by artificial neural networks, so there might be some resistance to adding complexity that might not be necessary. Edit: Google "dendritic computation" if you want to dig deeper.
- buboard 7y agoclarifying that dendritic computation has been studied for decades, this one is just a new type of dendritic spike
- sansnomme 7y agoAlso: spiking neural networks. Doing useful stuff with them has been a work-in-progress so far though.
- nravic 7y agoYeah, spiking neural networks are a tough nut to crack. Check out Nengo if you're interested in learning more
- buboard 7y ago
- anentropic 7y agoI remember reading this before, in a comment somewhere here on HN
- eismcc 7y agoAs the saying goes, “if a brain were easy enough to understand, we’d be too stupid to understand it.”
- Galaxeblaffer 7y agoAnybody know where these super neurons are supposed to be located in the brain ? Are they exclusively located in the Neo Cortex ?
- buboard 7y agoneocortex, from the anterior temporal lobes of epileptic patients . This kind of dendritic response has not been found the dendrites of other mammals so far. it's not a new type of neuron
- deckar01 7y ago> It may also prompt some computer scientists to reappraise strategies for artificial neural networks, which have traditionally been built based on a view of neurons as simple, unintelligent switches. It has been suggested that neural networks have diverged from biological principles to the point where biological research does not provide any useful improvement to current machine learning techniques. Back propagation of errors was a major advancement in the 1970s, and it was developed purely based on mathematical principles like statistics, differential equations, and calculus. As far as I have read, error back propagation is not how biological NNs learn, and it seems to be a much more efficient strategy. Biology seems much more brute force in comparison.
- telmo 7y ago> Biology seems much more brute force in comparison. Biology is more decentralized for sure, but I wouldn't be so fast with calling it "brute force". One interesting feature of biological neural networks is the variety of neurotransmitters. In very simple terms, they are not only capable of exciting or inhibiting neurons from firing, but they are also capable of regulating how neurons adapt to the stimulae they have been exposed to. In other words, the biological network not only learns, but it runs the learning algorithm itself. It appears to do this by creating various layers of communication with different effects on the computing nodes. In my view, we have not been smart enough yet to figure out nature's algorithm.
- radarsat1 7y ago> It may also prompt some computer scientists to reappraise strategies for artificial neural networks, which have traditionally been built based on a view of neurons as simple, unintelligent switches. I find this is a conveniently simplistic view of the current state of machine learning, which they wiggle out of by saying "traditionally built". Anyone working on ML and neural networks these days knows that modern networks are built out of building blocks -- certain standard architecture archetypes (MLP vs CNN, autoencoders, GANs, etc), resnet blocks, training regimes (layer pre-training, progressive training, etc), normalization methods, and so on. Sure, the term "neural networks" originally implied that each linear-non-linear operation pair represented a single neuron, but it's not a stretch to say that if neurons are more powerful than we thought, then each neuron represents, say, a small MLP block, or a resnet block. The biological analogy still stands. Citing ideas of ANNs from two decades ago to promise how new results in biology may change "current" thinking in ML (by citing decades-old ideas while calling them "traditional" no less!) is disingenuous.
- whatshisface 7y agoQuestion for ML people: would backpropagation work if some of the neurons had nonmonotonic activation functions (for example exp(-x²)), or would the gradient descent get stuck on one side or the other?
- p1esk 7y agoIt would probably work with careful choice of learning rate, initialization, and weight decay to keep signals small. Batch norm would play a larger role (probably want to use it after the activation fn). I don't see why it would get stuck on either side, but it could obviously get stuck if enough signals grow too large on both sides.
- cyorir 7y agoGradient descent should have no issues with a function like exp(-x^2). Actually, softmax (softmax(x_i) = exp(x_i)/sum_j(exp(x_j))) is sometimes used as an activation function. It could make sense to modify the softmax function to use -x^2 in place of x, for some use case. However, it doesn't always make sense as a drop-in replacement for other activation functions like ReLu or Sigmoid. It really depends on your use case.
- orbifold 7y agoFor those of you that want to be horrified / amused by code, check out the materials and methods of the article itself. It contains a link to the full source code used to generate the modeling figures: https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model=254217&file=/GidonEtAl2020_fig3andS9/_mod/Traub.mod#tabs-1 https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model... The simulator used is called Neuron. Code for it is written in a custom language called Hoc and models are implemented in yet another domain specific language. Hardcoded parameters galore including whole sections of code that are prefaced with: this has been copied from this other publication. https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model=254217&file=/GidonEtAl2020_fig3andS9/_mod/Traub.mod#tabs-2 https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model... The dendritic tree geometry is specified as hardcoded set of point and branch declarations https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model=254217&file=/GidonEtAl2020_fig3andS9/_morph/171122005.hoc#tabs-2 https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model... (apparently that morphology was reconstructed from imaging data?)
- mattmanser 7y agoWhy horrified? A quick scan and it's actually nicely laid out and indented, split into logical sections and commented. Without wanting to spend a lot of time on it, why is it horrific? There's nothing intrinsically wrong with hardcoded values as they're biological constants, and nicely identified by the standard convention of using ALL_CAPS_VARIABLES. Edit: Scanning a bit more, he's probably referring to this file, which is pretty horrific: https://github.com/ModelDBRepository/254217/blob/master/_morph/171122005.hoc https://github.com/ModelDBRepository/254217/blob/master/_mor...
- cafebeen 7y agoAlas, the possibility of comments like this are one reason that some researchers hesitate to release their code. Even if there are issues, it's tremendously valuable that they shared this, so others can reproduce and build on their work. That said, most scientists would like to have better quality code in science as well, but I think the incentive structures and other demands that make it challenging...
- buboard 7y ago
- RocketSyntax 7y agoAh, it's spiking edges as opposed to spiking node activations. I like this because it applies to 1 of many connections uniquely, rather than dividing the entire graph by spiking neurons. Exponentially more edges than nodes in a dense net. Connections are always more important than entities. ```The dendrites generated local spikes, had their own nonlinear input-output curves and had their own activation thresholds, distinct from those of the neuron as a whole``` How would this work in practice? Apply activations to multiplication values of the weight, or just don't perform the multiplication if the activation of the node is low?
- AstralStorm 7y agoChange the simulation unit from neurons to dendrites. Miss different kinds of gate and nonlinear activations. Which is what's being done lately, but then we know next to nothing about the topology of real neural nets nor electrochemical communication inside a neuron.
- api 7y agoAs someone who studied neurobiology more deeply than most CS people I've been talking into the wind for a long time that neurons are not point objects that can be simply modeled by equations. They are organisms with incredibly complex behaviors and internal gene regulatory structures and other information processing capabilities. Neural networks are coarse grained models of large scale brain structure. The fact that these models can be taught to do very interesting brain-like things (especially pattern recognition) demonstrates that this structure is important and fundamental, but that doesn't mean it's the whole picture of what's going on in the brain. I say talking into the wind because in my experience most CS people tend to hand-wave away biology at the cellular level and below. There's this zealotry around us being "almost there" with AGI and the rest that blinds people to the real magnitude of the problem and how much is really happening in the brain. This likely includes a whole lot that we haven't even started to really understand.
- bordercases 7y agoAmen.
- PinkMilkshake 7y agoThe neurons in the Creatures video games come to mind. In Creatures Evolution Engine games such as Creatures 3, each neuron and dendrite is a fully functional register machine. https://creatures.wiki/Brain#SVRules https://creatures.wiki/Brain#SVRules
- neural_thing 7y agoI have written a short, clear, and dense book about this exact subject. It's free! Read it here: http://www.corticalcircuitry.com/ http://www.corticalcircuitry.com/
- neural_thing 7y agoA preprint that came out after I released the book that I would have mentioned: https://www.biorxiv.org/content/10.1101/613141v1.full https://www.biorxiv.org/content/10.1101/613141v1.full They model a single cortical neuron as a deep neural network with 7 hidden layers consisting of 128 hidden units each. Biological neurons are A LOT more powerful than "neurons" in neural networks. If you hear a claim about computer-brain parity being close - the people making it almost certainly don't understand the power of cortical neurons.