12 ms·
The cortex is a neural network of neural networks
- coinward 8y agoSo are we going to be able to use any of this for a new deeper deep learning framework?
- deleted 8y ago[deleted]
- davesque 8y agoWell, a neutral network is a neutral network of neutral networks.
- jarfil 8y agoI think the point of the article is that brain neurons are not equivalent to the ML representation of neurons, so that count if 17 billion in the cortex would actually require many more "ML neurons" to be simulated. It also explains how we can keep adding complexity even if no new neurons are being created, since the branches themselves act like extra neurons.
- taneq 8y agoWouldn't that just mean the brain's more like a small-world network, with small highly connected 'blobs' tied together in a sparser, larger network?
- shagie 8y agoThat is one theory. Multiple "agents" is something that has been proposed. I don't have the background to read beyond the abstract of https://eml.berkeley.edu/~webfac/malmendier/e218_sp06/Carrillo.pdf https://eml.berkeley.edu/~webfac/malmendier/e218_sp06/Carril... > We model the brain as a multi-agent organization. Based on recent neuroscience evidence, we assume that different systems of the brain have different time-horizons and different access to information. Introducing asymmetric information as a restriction on optimal choices generates endogenous constraints in decision-making. There's also Society of the Mind ( https://en.wikipedia.org/wiki/Society_of_Mind https://en.wikipedia.org/wiki/Society_of_Mind ) by Marvin Minsky (which is very readable) > A core tenet of Minsky's philosophy is that "minds are what brains do". The society of mind theory views the human mind and any other naturally evolved cognitive systems as a vast society of individually simple processes known as agents. These processes are the fundamental thinking entities from which minds are built, and together produce the many abilities we attribute to minds. The great power in viewing a mind as a society of agents, as opposed to the consequence of some basic principle or some simple formal system, is that different agents can be based on different types of processes with different purposes, ways of representing knowledge, and methods for producing results.
- mattkrause 8y agoThe brain is a small world network[0], but article’s thesis is that a neuron (a single vertex in that small world) can actually compute quite a bit on its own—-it’s not an integrate-and-fire unit or a ReLU. I guess you could expand things out and treat each synapse, rather than each neuron, as as a unit, in which case there’s an extra level in your “world”. This is a little different from how people normally think of “connectivity” because the way information is routed within a cell is much more flexible than between them. Still, it’s not totally bonkers. [0] Just so we’re on the same page, when people talk about the brain as a small world network, the general idea is that a neuron in (say) auditory cortex mostly communicates with other auditory neurons, locally and in higher/lower auditory areas. It doesn’t talk to neurons representing (say) touch in the small of your back. Thus, there are far fewer than 86B! connections in the brain, but there are still an awful lot!
- dschuetz 8y ago"... Our analogies often look to artificial neural networks: for neural networks compute, and they are made of up neuron-like things; and so, therefore, should brains compute. But if we think the brain is a computer, ..." OK, enough of this. Neurons are not computers. There is nothing what can be compared to actual neurons. "Artificial" neurons are just reduced models of the real ones, so that only the "compute" parts are used to calculate input vectors. It's only a fraction of what the real neurons actually do. While I appreciate the article trying to actually understand what's really going on in neural networks, let's not make unnecessary dumbed-down assumptions. At least the subtitle of the article is actually correct. The main title is sensationalist "...17 billion computers!!".
- carlsborg 8y agoAll models are wrong, some are useful.
- feanaro 8y agoNot all models are wrong, since some may be modelling exactly the aspects of importance and not modelling exactly those which are incidental and of no consequence.
- joesb 8y agoThen they are still wrong. It's just that they are wrong in the part that you don't care about.
- feanaro 8y agoIs it wrong, or is it simply an instance of the fact that a model is not the thing it models? The use of the word wrong sounds wrong to me here. The wrongness should be about whether the model accurately, or just approximately, or not at all, shares with the modelled object that part of the behaviour we are interested in.
- danmaz74 8y ago
- anonoholic 8y agoSo, a deep neural network?
- ianai 8y agoIs this the first time the concept of neural networks of neural networks has been proposed? I think it’s close to an idea I’d been knocking around in my head but never studied NNs deep enough to encounter. I wouldn’t be shocked if consciousness were composed of hundreds or even thousands of NNs. Or even a tree thousands of levels deep.
- buildbot 8y agoThere's network in networks: https://arxiv.org/abs/1312.4400 https://arxiv.org/abs/1312.4400 And network ensembles in general: https://en.wikipedia.org/wiki/Ensemble_learning https://en.wikipedia.org/wiki/Ensemble_learning
- dredmorbius 8y agoMarvin Minsky's Society of Mind was published in 1986. A core tenet of Minsky's philosophy is that "minds are what brains do". The society of mind theory views the human mind and any other naturally evolved cognitive systems as a vast society of individually simple processes known as agents. These processes are the fundamental thinking entities from which minds are built, and together produce the many abilities we attribute to minds. The great power in viewing a mind as a society of agents, as opposed to the consequence of some basic principle or some simple formal system, is that different agents can be based on different types of processes with different purposes, ways of representing knowledge, and methods for producing results. https://en.wikipedia.org/wiki/Society_of_Mind https://en.wikipedia.org/wiki/Society_of_Mind
- guskel 8y agoAs far as I know, nobody takes the "Society of Mind" theory seriously anymore and none of it lives in present day AI work. There was never a clear algorithm that could be constructed from the chapters on K-lines.
- imglorp 8y agoWe have a fair idea how many layers are involved in some activities, such as eye-hand response times, because we can divide the observed delay by the speed of a single layer. It's not a lot of layers. Reference needed.
- est31 8y agoThe model of the technical neuron is only inspired by the biological model, but not meant as approximation. Instead the goal is to obtain good results in actual applications.
- arcanus 8y agoYes, and the sooner we dispel with this absurd notion that we have any evidence we are closely modeling the human brain, the better.
- armada651 8y agoSure, but it's helpful to compare our models of artificial intelligence with biological intelligence to see if there's anything to be learned. We learned how to make airplane wings from the shape of a bird's wing. Of course we should not model our artificial wings so closely as to make a plane with wings that flap. But there was still plenty of stuff to learn by asking the question "why does a bird fly and my contraption doesn't?"
- FakeComments 8y agoAs an explicit example, winglets on airplanes were conceptualized from watching the way bird wings flap, observing the curl on the outer edge of the wing, discovering that it controls vortex formation, and then applying the same concepts to fixed wings. That kind of thing happens all the time in aerodynamics, fluid dynamics, mechanics, etc — precisely because evolution is a pretty good optimization function, and so “natural” solutions can often be very close to optimal, but using hard-to-discover quirks of physics.
- trentlott 8y agoOne of my favored arguments for maintaining as much biodiversity as we possibly can. Each species' death is millions of years of labwork trashed
- 8y ago
- casual_slacker 8y agoNot sure if I understand, but it seems the dendrites are "grouped" in a way, and their influence on the output is a function of the group? Is this functionally equivalent to having a two layer mini-network (that represents one brain neuron), with one neuron on top, and "child" neurons on bottom that mimic the grouping behavior? If this is true, then I would suspect our networks are already doing something like this automatically.
- buboard 8y agoyes, the linked papers deal with this 2-layer abstraction of a single neuron. In reality, the neuron-to-neuron connections however are different from dendrite-soma coupling and those levels (Dendrite and soma) differ in terms of their ability to integrate synaptic inputs and undergoing plasticity, so they re not really equivalent. This is still an active area of research with a lot of unknowns.
- xenadu02 8y agoThis seems to make intuitive sense. If we ever create a true AI it will probably be on the order of billions of neural networks connected together.
- User23 8y agoI've never understood the almost religious devotion many hackers have to the idea that the brain is a computer. The brain, or more practicably the brain, body, and a pencil and paper, can slowly simulate a Turing machine without great difficulty. But a Turing machine can simulate a DFA too and that doesn't make it one. This should not be construed as denigrating the wonderful achievements of AI researchers. Just because what they do is inspired by the brain rather than isomorphic to the brain doesn't mean it isn't great work.
- urgoroger 8y agoIn accordance with the Church-Turing thesis, the Turing machine stands to be capable of doing anything that should be called computation. It follows that if the brain is capable of simulating of a Turing machine (this is called a universal Turing machine, by the way), then it too can do any computation. So then the class of things that both can do are the same, and so it is reasonable to call them the same thing, in some sense.
- FakeComments 8y agoThis only shows computers are a subset of what brains can do, not that Turing machines can do whatever brains can do.
- User23 8y agoAmazing how many people miss this. It’s like confusing the implication with the equivalence.
- urgoroger 8y agoWhat I was trying to communicate was that the Turing machine believed to represent the limits of what is physically possible. So then you have turing machines >= brain (since a brain is physical) and brain >= turing machine (by simulation argument) The conclusion is the brain and Turing machine can do the same thing (brain = turing machine).
- kristianov 8y agoAt which point should we start to call it neural internet?
- skywhopper 8y agoThis is a good article for deepening the complexity model of the brain, but mere physical structure and behavior of subcomponents of neurons is only a tiny piece of the puzzle. To understand the workings of the brain you will also need to add in the complex interactions of hormones and other chemical agents in the brain and throughout the nervous system and the feedback loops they establish with the other systems of the body and through them, with the external world. Neural networks of neural networks doesn’t begin to describe it. We haven’t even scratched the tip of the tip of the iceberg in understanding this stuff.
- ltbarcly3 8y agoI appreciate what you are saying, but it may be that it's premature to try to list things necessary to understand something when it's not understood by anyone. In fact, I think you're probably dead wrong. Certainly hormone interactions and feedback loops in the body have some consequence on an organism, but these things take time. Lets say that some feedback loop is extremely fast, say 10 seconds. It seems reasonable to assume that you can't have a complex feedback loop with the body much faster than that, just because it takes time for chemicals to physically move. Cognition is much, much faster than that. There are many situations where you can learn, and then apply that learning in less than a second. Now if we were instead talking about what it takes to build a complex organism that can satisfy it's own needs, then certainly these complex hormonal feedback loops are essential. You can't have an organism survive when it doesn't look for food when it's body is out of fuel or keeps eating poisonous berries because it's body can't send the reasonable feedback to it's brain, and you can forget about dropping everything and competing for mates at the correct time without some kind of behavior modifying hormonal signals! I would say that your assertion is not significantly different from saying "you can't understand the workings of the brain without considering the complex interactions of the brain with the lungs, since without oxygen the brain can't work". So while I agree that the brain exists as part of the body, I don't see why you would automatically assume that the hormonal feedback loops with the body are at all necessary for cognition rather than a way to tell the brain when it's necessary to find food, when it's adaptive to conserve energy and be lethargic, or what have you. My point is that we may or may not find that there is some critical ingredient of cognition hiding in this place or that, and you can't very well tell us where that will be if nobody knows if it's there.
- iandanforth 8y agoI take issue with the "dendrites know more than neurons" bit. The fact that they respond to almost all inputs suggests they are performing a different function that a somatic spike. My preferred explanation for that is that any type of input can be predictive of a somatic spike and that has to be transduced somewhere. Specific patterns of concurrent input on a dendrite drive sub-threshold depolarization which is theorized to be key for sequence prediction.
- westurner 8y agoMetadata is not just data
- orbifold 8y agoAnother thing that is not well known about the brain (among non specialists) is that there are roughly one order of magnitude more glia cells than neurons in the brain, which while non-spiking definitely also respond to synaptic activity and could be involved in computation.
- _Schizotypy 8y agoThis cannot be overstated. Glial cells do seem to communicate with traditional neurons
- buboard 8y agoSo, a neural network of a neural network is just a deeper neural network. The big question in dendritic processing is whether it is used (conflicting information about that, e.g. Jia&Konnerth's work), whether it represents anything, and how it is learned. Plasticity is all over the place in neurons and takes place also at the dendritic level with cooperation & competition between synapses, temporal dynamics and neuromodulation. The credit assignment problem is hard to solve at the circuit/population level, but dendrites offer an intriguing alternative, as it is possible for them to bidirectionally communicate with the spike initiation site.
- svantana 8y agoIndeed, the Network-in-Network architecture [1] was a compelling idea to get complex activations, until it was realised that it's just a standard neural network which is not fully connected. Since neural networks are universal approximators, it's a bit silly to talk about something else being more powerful, it's all about the prior, bias, and training, which are all subject to the No Free Lunch theorem. [1] https://arxiv.org/abs/1312.4400 https://arxiv.org/abs/1312.4400
- empath75 8y agoYou can also compose functions together but there’s a reason that programmers don’t generally jam everything into a single function or think about programs that way.
- deleted 8y ago[deleted]
- svantana 8y agoRight, which is one of the main critiques against deep learning, there is no separation of concerns or encapsulation, just a single function matching input to output. But at the end of most days, performance is what matters. Similarly, the brain hardly has a "clean" structure, it's seemingly spaghetti code even though there is some structure to it.
- novaRom 8y agoOff topic, but it is really uncomfortable to read anything on medium because 25% of my screen is covered by header containing 'Sign In' and 'Get Started' and by footer with 'Get Updates' button.
- niceworkbuddy 8y agoFortunately there is an extension for Chrome solving that problem. https://chrome.google.com/webstore/detail/make-medium-readable-agai/kljjfejkagofbgklifblndjelgabcmig https://chrome.google.com/webstore/detail/make-medium-readab...
- speakeron 8y agoThe 'element zapper' (or 'picker' if you want it made permanent) in uBlock Origin also works great for these situations.
- azthecx 8y agoCheck outline out. https://outline.com/czFN3X https://outline.com/czFN3X I believe it was made by someone here on HN, or at least I found it through it.
- paol 8y agoIn addition to the solutions in the sibling comments, let me offer my favorite one: a simple bookmarklet that zaps all sticky elements on the page. Works wonders in Medium and a lot of other sites. javascript:(function()%7B(function%20()%20%7Bvar%20i%2C%20elements%20%3D%20document.querySelectorAll('body%20*')%3Bfor%20(i%20%3D%200%3B%20i%20%3C%20elements.length%3B%20i%2B%2B)%20%7Bif%20(getComputedStyle(elements%5Bi%5D).position%20%3D%3D%3D%20'fixed')%20%7Belements%5Bi%5D.parentNode.removeChild(elements%5Bi%5D)%3B%7D%7D%7D)()%7D)()
- 3PS 8y agoThank you, this works like a charm! I tried it on tapas.io where it got rid of both the annoyingly thick header as well as the sidebar.
- xpuente 8y agometabotropic channels/synapses? ... sadly the most frequent are missing there.
- deleted 8y ago[deleted]
- kingkawn 8y agoThe takeaway seems to be more about degrees of complexity than any particular structural component taking precedence.
- voidmain 8y agoA neural network of neural networks is... a bigger neural network. Having two or three layers of nonlinearities per "neuron" doesn't do anything qualitatively different. There are probably lots of huge differences between NNs and brains but this article is really making the case that the brain can be modeled as a big NN, just with a few thousand times more activations than neural cells.
- rusticpenn 8y agoBrain can indeed be modelled by NNs, however the neurons in brain are more complex and requires use of more complex neural models like Hodgkin–Huxley model or can be approximated by more simplified models like Integrate and Fire models. ( ref: https://www.humanbrainproject.eu/en/ https://www.humanbrainproject.eu/en/ )
- j7ake 8y agoWhat's the evidence that "brain" (whatever that means) can be modelled by NNs? What features of the brain can NNs model?
- johnnycab 8y agoThis article probably serves as an amuse bouche in the fluid world of mapping or replicating functions of wetware to algorithms and vice-versa; the top highlight: 17 billion neurons, almost sounds like one of those sampled, haunting soliloquies in prog or psy-trance tracks, which are usually restricted to snippets from sci-fi movies or taxonomy of the universe e.g. there are billions and billions of stars.. This blog post via the Human-Centred AI research from The Stanford Institute, dealing with a similar subject matter, is wide-ranging, incisive and replete with sources. https://hai.stanford.edu/news/intertwined-quest-understanding-biological-intelligence-and-creating-artificial-intelligence https://hai.stanford.edu/news/intertwined-quest-understandin...
- sica07 8y agoWhat I don't understand is the part about the supralinear/sublinear particularity of the dendrite. First, the article explains that: " If enough inputs are activated in the same small bit of dendrite then the sum of those simultaneous inputs will be bigger than the sum of each input acting alone (...) A bit of dendrite is “supralinear”: within a dendrite, 2+2=6." Further in the article, I find this explanation: "Because dendrites are naturally not linear: in their normal state they actually sum up inputs to total less than the individual values. They are sub-linear. For them 2+2 = 3.5". What makes the difference between a bit of dendrite spiting a sublinear vs. supralinear "result"? I feel that the difference lays in the 'if enough inputs are activated' vs. 'in their normal state'. If that's the case, what's the "normal state"? Could anybody help me understand this part?
- OnlyRepliesToBS 8y agoyo dawg...