10 ms·
If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant
by jakobson14 3y ago
If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers")
Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong."
Neurologists seem to have a really hard time comprehending that researchers working on neural networks aren't as clueless about computers as neurology is about the brain. They also vastly overestimate how much engineers working on neural networks even care about how biological brains work.
Virtually every attempt at making neural networks mimic biological neurons has been a miserable failure. Neural networks, despite their name, don't work anything like biological neurons and their development is guided by a combination of
A) practical experimentation and refinement, and
B) real, actual understanding about how they work.
The concept of resnets didn't come from biology. It came from observations about the flow of gradients between nodes in the computational graph. The concept of CNNs didn't come from biology, it came from old knowledge of convolutional filters. The current form and function of neural networks is grounded in repeated practical experimentation, not an attempt to mimic the slabs of meat that we place on pedestals. Neural networks are deep because it turns out hierarchical feature detectors work really well, and it doesn't really matter if the brain doesn't do things that way.
And then you have the nitwits searching the brain for transformer networks. Might as well look for mercury delay line memory while you're at it. Quantum entanglement too.
- deleted 3y ago[deleted]
- junofan 3y agoI dunno, failure seems okay. Wouldn’t expect a better paradigm to beat SOTA at first. It’s totally plausible that neurons use eg. transposons in a way we don’t yet have the instrument resolution to characterize, which would suggest that you don’t need 1000 layers, but a lookup table or something.
- b33j0r 3y agoAgreed, but I do also think that order emerged from chaos. It’s an easy claim when order is defined by itself! But in reality, we’re equipped exactly to exist, and we still wonder why in a backwards way, even with education (guilty!) AI is the task of playing God like toddlers at recess, and LLMs the tower of babel. I still wanna play, it’s fun
- robbrown451 3y agoI can't agree with the dismissiveness of this comment, and frankly I find its tone out of line and not with the spirit of Hacker News. There are insights that can come from studying the brain, that do indeed apply. Some researchers may not glean anything from such studies, and some may. I have no doubt that as neural networks get more an more powerful, we will continue to find more ways they are similar to the brain, and apply things we've learned about the brain to them. I certainly prefer to see people making comparisons of neural networks to the brain, that the old "it's just a glorified autocomplete" and the like. Relax.
- krainboltgreene 3y agoWhat does this comment add to the discussion?
- robbrown451 3y agoI dunno. My comment complained about the parent comment not adding positively to the discussion. And gave at least a bit of support for that complaint. Would you have preferred I emulate your style, and complain while providing no support for my complaint? Ok.
- krainboltgreene 3y agoBeing positive is not a requirement of commenting on HN, but you should comment with something that is substantive, so yes I do think you shouldn't have commented at all. Tone policing is cringe.
- robbrown451 3y agoExactly what are you doing here then? But hey I guess I can do this too. How's this? Using cringe as an adjective is cringe.
- krainboltgreene 3y ago
- mrstone 3y agoA neurologist is a medical doctor. Neuroscientists are the PhDs who do the actual research.
- crustacean111 3y agoCNNs actually are biologically inspired. The receptive field in a CNN mimics the way that cortical neurons only respond to stimuli in a restricted region of the visual field. Different cortical neurons have receptive fields that partially overlap to cover the whole visual field [1]. [1] - https://en.wikipedia.org/wiki/Convolutional_neural_network https://en.wikipedia.org/wiki/Convolutional_neural_network
- jakobson14 3y agoYou're going to have to dig deeper. The concept of a receptive field goes all the way back to convolutional filters. It's not surprising that we found out later the brain also uses such a fundamental element of signal theory.
- SubiculumCode 3y agoOh good. So you do admit that there are useful parallels between signal processing, statistical processing, and the brain.
- vkou 3y agoSure, and airplanes are inspired by birds. That doesn't mean that detailed studies of the Boeing 747 are going to unlock a lot of hitherto unknown mysteries of heron behaviour.
- jacobsimon 3y agoI mean, I know you’re just providing an analogy, but people are still studying the physics of bird flight and we’re nowhere close to building machines yet that can maneuver the way birds can. https://www.quantamagazine.org/geometric-analysis-reveals-how-birds-mastered-flight-20220803/ https://www.quantamagazine.org/geometric-analysis-reveals-ho...
- ben_w 3y agoI could believe "we have more to learn", but not "we're nowhere close": https://youtu.be/w6VLzKACnS8?si=DZgOPuBRG4Vt98su https://youtu.be/w6VLzKACnS8?si=DZgOPuBRG4Vt98su
- jacobsimon 3y agoThis is a really weird take. There is such a long history of shared insights between biology and neural network research, and to say they’re unrelated or can’t take inspiration from one another is bizarre. > The concept of CNNs didn't come from biology I just opened a survey paper on CNNs and literally the first sentence of the paper reads: > “Convolutional Neural Network (CNN) is a well-known deep learning architecture inspired by the natural visual perception mechanism of the living creatures. In 1959, Hubel & Wiesel [1] found that cells in animal visual cortex are responsible for detecting light in receptive fields. Inspired by this discovery…” Source: https://arxiv.org/pdf/1512.07108.pdf%C3%A3%E2%82%AC%E2%80%9A https://arxiv.org/pdf/1512.07108.pdf%C3%A3%E2%82%AC%E2%80%9A
- jakobson14 3y agoThat's later backfill, a retroactive change to give a manufactured "biological" origin story. Whether they're real or not, researchers love a good "we took this from nature, isn't nature wonderful!" explanation. The C in CNN isn't "Convolution" for no reason. It came from work with convolutional filters (yay Sobel kernels!) which at it's height became filter banks and gabor filters and so on before neural networks pretty much killed off handcrafted feature development. Every explanation of how CNNs work still falls back to the original convolutional kernel intuition.
- dartos 3y agoYou can use that argument for anything you disagree with. Do you have a source or anything?
- jakobson14 3y agoHave a read through the first paper describing a convolutional neural network, from 1998: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf There's absolutely no mention of biological inspiration whatsoever. At the same time, one can point to a long and rich history of convolutional filters being used in signal processing. And then there's the name, Convolutional Neural Network. The entire concept of a CNN is framed as a series of learned filters.
- two_in_one 3y agoWhile I agree with this emotional post there is one nuance. Neural networks aren't intelligent, brain is. And that's where we want to be. Checking gradients and studying filters can get us only this far. So, using brain as inspiration looks like a good option. There are other, but nobody knows where next breakthrough will be. Like nobody knew five years back that transformers are so powerful. My guess next step to AGI will be a complex modular multi-modal system. With hierarchy, workers and controllers, complex signals.. Sound familiar? Brain is sort of it. This is need for embodied AI, obviously. But, interesting thing, it's needed even for body-less AGI too. I.e. AGI is not a big calculator (!), it's more like real-time system. One reason is that full search is impossible. So, in many cases requests will be like 'give the best answer you can find in 4 seconds'. 'and keep looking'. So far we have only real-time dumb robots and NN big calculators. And brains, of course.
- dilawar 3y ago> previously: comparing brains and "electronic computers") Before that: comparing brain with hydraulic machines. There has been tendency to compare brain with most complex machine known to us at that particular time. "Descartes was impressed by the hydraulic figures in the royal gardens, and developed a hydraulic theory of the action of the brain. We have since had telephone theories, electrical field theories, and now theories based on computing machines… . We are more likely to find out how the brain works by studying the brain itself, and the phenomenon of behavior, than by indulging in far-fetched physical analogies." -- Karl Lashley 1951
- spindle 3y agoAnd also comparing brains to clockwork.
- mjburgess 3y agoI cannot agree enough with Karl here. What is the brain? An organic system with deep roots in the organic body, with deep causal connections with its environment. There's little sense in ignoring the whole basic mode of operation, physics, chemistry and biology of the brain in order to analogise it to another system without any of those properties. This, at best, provides a set of inspirations for engineers -- it does nothing for science.
- ben_w 3y agoI mildly disagree (although your final conclusion is correct: it indeed does nothing for science). The deepest fundamental structures in the brain[0] are quantum fields, which are also the deepest fundamental structures in everything else. There is no known quantum field of "soul" or "intelligence". The right abstraction is higher, and could still be a whole lot of things; but as maths can be implemented in logic, which can be implemented in electronics or clockwork or hydraulics, it doesn't matter what analogy is used — and my mild disagreement here is that such inspiration has been useful and gotten us this far. [0] that we know of
- mjburgess 3y ago
- wslh 3y agoOnly an observer of the topic but I think it is good to review Koch's book about the real complexity of a single neuron [1]. [1] https://www.amazon.com/Biophysics-Computation-Information-Computational-Neuroscience/dp/0195181999/ref=sr_1_1?keywords=Biophysics+of+Computation%3A+Information+Processing+in+Single+Neurons&qid=1561229482&s=books&sr=1-1 https://www.amazon.com/Biophysics-Computation-Information-Co...
- SubiculumCode 3y agoIf you read this article, I think most would understand that it is primarily aimed at other neuroscientists, and only using ML structures an an analogy only, and I think a somewhat useful one to boot. The real point of the article was to propose a general hierarchy for how information flows in the brain, to emphasize the importance of subcortical brain even in higher order cognition, and proposes how simultaneous processing of multiple levels of representation can inform action and thought. As a developmental neuroscientist, I found the article insightful and thought provoking. Further, it is quite consistent with major hypotheses in psychology, how the hippocampus works (a subcortical structure) and combines information into memories: See fuzzy trace theory [1], for example. Your dismissive tone is unappreciated, ill-informed, and crass. [1] https://en.wikipedia.org/wiki/Fuzzy-trace_theory https://en.wikipedia.org/wiki/Fuzzy-trace_theory
- visitor4711 3y agofully agree
- bjourne 3y agoFirst, I wonder how you got access to the article? It is behind a paywall and not yet uploaded to the sites I usually find paywalled articles on. Second, there is no need to compare brains to neural networks because brains are neural networks. Neurons form vertices and axons edges connecting the aforementioned. What you are perhaps thinking of are artificial neural networks - most of which are very dissimilar to brains. But even then you are wrong. Artificial Izhikevich and Hodgkin-Huxley neural networks attempts to closely mimic the behavior of real neurons. While deep, hierarchical artificial neural networks have been more successful than biologically plausible ones, that may be because the technology isn't ready yet. After all, the perceptron was invented in the 1950's but didn't become prominent until the 2010's (or so). Perhaps we need new memories that better map to (real) neural network topologies, or perhaps 3d chips that can pack transistors in the same way brains pack neurons.
- mjan22640 3y agoA neuron is analogous to a 3d integrated circuit rather to a transistor. A molecule acts like a transistor https://medium.com/the-physics-arxiv-blog/the-origin-of-life-and-the-hidden-role-of-quantum-criticality-ca4707924552 https://medium.com/the-physics-arxiv-blog/the-origin-of-life... Changes in mechanical pressure, electric field, other molecules attachment, photon absorption, can control the conductivity. Organic semiconductors designed to fit like lego bricks to naturally build the desired structure are IMHO the way to go to produce 3d circuits, rather than layered silicone litography.
- ben_w 3y ago> silicone I've seen this particular mistake a lot recently. New and exciting auto-corrupt from the latest version of iOS? Given that our brains rewire themselves live, which ANNs can only do by being excessively connected and updating weights to/from zero, silicone (I'm thinking mainly the oil form) may be a better inspiration than lego. https://en.wikipedia.org/wiki/Silicone https://en.wikipedia.org/wiki/Silicone
- mjan22640 3y ago
- __loam 3y agoAs a biomedical engineer who went into software, thank you for this comment lol. So tired of rehashing this.
- andromaton 3y agoBooks and articles I was reading in the 80s (eg Minsky and Papert, Byte magazine) were referring to Rosenblatt and retinas.
- blovescoffee 3y agoDude. What holy and special work do you do? There's nothing dumb or dull in searching for analogous structure between two effective machines, neither of which we understand.
- nathias 3y agoMetaphores and analogies are important tools of thinking, even in science, some bear fruits some lead to errors, but we can't know in advance.
- hliyan 3y ago"brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong" Please don't claim things the author didn't. What I read was "ergo (artificial) neural networks may be missing a trick"
- deleted 3y ago[deleted]
- radarsat1 3y ago> every time some neurologist tried to compare brains to neural networks Value of this comment aside, it kind of makes me chuckle how casually it (and other comments in this thread) just drops the word "artificial" from neural networks here, specifically when comparing with neurology. The irony is funny. Like, somehow we've forgotten why we call them that in the first place, exactly when talking about the thing that inspired the approach.
- svara 3y agoDoesn't know what a neurologist is, knows they do shit work.
- NoToP 3y agoI disagree profoundly. There are things the brain does we have not yet been able to reproduce with a neural network, or to the extent we have seemingly with excessive resources of training and network size. Therefore there is some salient feature of neurology which has been overlooked. I don't think it is necessary to mimic biology down to the exact function of real neurons, but there must in fact be something we are neglecting to mimic.
- ben_w 3y agoPossibly, but it may also be that we're training them wrong. "Book smart, not street smart" (to use a catchphrase) would apply perfectly to GPT models: brain the size of a rodent's, with 50,000 year's experience of reading Reddit, Wikipedia, and StackOverflow, but no "real life" experiences of its own.