5 ms·
The key to the claim that other computational estimates are way off is essentially that there's a lot of data crunching happening within a single neuron, rather
by ewjordan 9y ago
The key to the claim that other computational estimates are way off is essentially that there's a lot of data crunching happening within a single neuron, rather than it being something we can model as collecting a bunch of inputs and either firing or not. He's arguing that each neuron does a ton of internal computation that can itself be modeled as a (sometimes very large) convolutional network.
I think in a sense this is well-established when looking at real neurons, but several times in the article he uses phrases like "shown to be important for information processing", and that's where I get off the boat a bit. When you're saying that it's so important for information processing that it warrants a 1000x or more increase in the computational power necessary to implement an algorithm, I think it's necessary to dig into what the actual work being done there is, not just that there's some non-trivial transformation. A lot of interesting and extremely tough to model fluid and chemical dynamics are in play when I drink too much water and have to pee, but that doesn't mean that we need to understand them to build a waste disposal system using pipes.
In particular, does the within-neuron processing actively tune itself based on the data it processes to an extent on-par with inter-neuron connections (in which case the argument that it's fundamental to the learning process would hold a lot more weight), or is it mostly static? I think a lot of us consider "important for information processing" to mean "is a meaningfully dynamic parameter involved in a learning algorithm", rather than an accidental shmearing of data.
I'd really love more info on what the actual processing that's happening is.
- Seanny123 9y agoThat's basically what the entire field of computational/theoretical neuroscience is trying to figure out right now. What's being computed? What's the right level of abstraction? Basically, it depends on what you're modelling: https://www.ncbi.nlm.nih.gov/pubmed/24709593 https://www.ncbi.nlm.nih.gov/pubmed/24709593
- Lost_BiomedE 9y agoI worked in this field a bit at Baylor. There are those who want to model the 'hardware' down to every complexity and those who want to figure out the 'software' that emerges from the physical hardware. The groups communicate but are fairly divided. Does it tune itself or is static? It definitely tunes, but it can be meaningful to the algorithm or ignored.
- visarga 9y ago> there's a lot of data crunching happening within a single neuron I'll just leave this here: https://en.wikipedia.org/wiki/Gene_regulatory_network https://en.wikipedia.org/wiki/Gene_regulatory_network Inside any cell there is a system that works like a neural network - the gene regulatory network. Each gene acts like a neuron, with chemical inputs and outputs. That would make the processing power of any cell on par with that of a small neural net.
- shirakawasuna 9y agoGene regulation is very slow, on the order of minutes to hours. It's involved in our brains, for sure, but it's just way too slow to be a dynamic part of our brains' processing of information.
- mannykannot 9y agoWe should distinguish between the information processing of the machinery of the brain and the information processing of the mind, even though the issue of what makes up the latter is an open question. It would not be a useful measure of a program's complexity to simulate the hardware running the program, and calculate the complexity of that simulation (even if we didn't follow the implicit infinite recursion in that approach...)
- taneq 9y agoMy non-specialist take on it is that, even if what neurons really do is hugely complicated, we should be able to approximate it with something pretty simple and anything sorta close should work. I believe this because real brains are incredibly fault tolerant and can take an absolute beating while still continuing to function 'well enough'. I'd be willing to bet that once we understand the basic functioning, we'll be able to build a working brain using a few different types of neuron which operate on relatively simple rules-of-thumb, arranged in a few different basic structures which are internally relatively homogeneous.
- ewjordan 9y agoI think I tend to agree (and actually, the author does, as well), the interesting question is whether we've been underestimating the number of artificial "neurons" we'll need. If each biological neuron is actually essentially implementing an entire neural net internally, then we might be a lot further from achieving that amount of computation than we expected. We absolutely need to start doing what you're suggesting, and figuring out how to derive useful basic structures, represent them, and construct networks using those as the building blocks. My prediction is that's where the field will go over the next 5-10 years, a much deeper dive into how to specify connectivity and usefully control it than has been done so far (we're still using fully connected layers, for the most part, which we know is not a scalable approach as we go from thousands to millions of nodes).