3 ms·

I suspect if you rely on certain analog properties (with some good precision) then the elements won't scale down very well (down to current transistor scale). A

I suspect if you rely on certain analog properties (with some good precision) then the elements won't scale down very well (down to current transistor scale). At the very limit of element size I would think reliable elements inevitably degrade to binary gates, which are the simplest elements in behavior (at one point even binary gates become impossible of course due to phenomena like leakage, quantum tunelling, etc). The brain as far as I understand does so much with large, slow elements (neurons) by having them fill a volume, be sparsely activated (i.e. mostly a huge memory), and other advanced communication methods (temporal pulse position modulation/frequency from spiking? neurotransmitters?). Current ML is more densely activated, high-frequency networks. I'm not sure we could revert to the brain-like architecture unless we could get the cost of sillicon manufacturing several orders of magnitude down, enough that we could just fabricate a large block of stacked complex elements. A large part of the philosophy of nodes would need to be reworked (much lower frequency, lower leakeage, lower power consumption), as processes are optimized for >100MHz freqs; just so internal memory elements would keep at acceptable temperatures. Currently you could fit about 2000 GPUs in a 10cm^3 space (assuming 1mm die thickness), which would cost about $1.5M usd. And couldn't do much, because it would quickly overheat on reasonable loads, and because I don't think we have the technology to interconnect it all.

dkfellows 8y ago
The brain definitely is able to use not just spiking frequency (which is approximately the same as EEG voltage, though not really) but also spiking patterns to encode information. We've observed some highly interesting phase locked loops showing various higher-order patterns in simulations (we simply don't have fine enough tools to look for the equivalent in the biology). The variation in neurotransmitters allows for different sorts of activation, typically with different physical parameters (size of activation, time over which it decays) and multiple ways they interact with the other neurotransmitters. Sparse networks are not understood to anything like the same extent as dense matrices. And another key property that most ML is missing is large numbers of feedback loops. Again, that makes predicting behaviours extraordinarily difficult.
p1esk 8y ago
And couldn't do much It couldn't do much mainly because we don't know what it should be doing (to emulate brain). In this case, the software challenge is far greater than the hardware challenge.

dkfellows 8y ago

The brain definitely is able to use not just spiking frequency (which is approximately the same as EEG voltage, though not really) but also spiking patterns to encode information. We've observed some highly interesting phase locked loops showing various higher-order patterns in simulations (we simply don't have fine enough tools to look for the equivalent in the biology). The variation in neurotransmitters allows for different sorts of activation, typically with different physical parameters (size of activation, time over which it decays) and multiple ways they interact with the other neurotransmitters. Sparse networks are not understood to anything like the same extent as dense matrices. And another key property that most ML is missing is large numbers of feedback loops. Again, that makes predicting behaviours extraordinarily difficult.

p1esk 8y ago

And couldn't do much It couldn't do much mainly because we don't know what it should be doing (to emulate brain). In this case, the software challenge is far greater than the hardware challenge.