4 ms·

It's not that no one understands, it's that journalists don't know that their's research that has addressed this question. The problem with intuitively understa

It's not that no one understands, it's that journalists don't know that their's research that has addressed this question. The problem with intuitively understanding neural networks is that most of the weighted edges are spurious. Meaning you can remove most of them (WiJ -> 0) without negatively affecting the mapping function solution. Failing to do this we stare at a fully connected neural network which gives us no clues why it works. However , once you get rid of these spurious interactions (through perturbation analysis), then This will reveal the underlying circuit diagram for the network. Often these threshold integration neurons are just approximating a simpler Boolean function so it really comes down to digital logic. Techniques for pruning networks can be found here: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/

smaddox 10y ago
Pruning works for understanding sparse networks, but not for holographic networks, which can encode a much higher information density. Understanding a holographic network requires a large amount of effort, similar to determining the content of an optical holograph. You can try to develop a model, but if the information density is very high, it's no faster than observing the response to various inputs. The problem becomes even more difficult if there is no clear distinctions between layers in the network, as in the brain.
- rdlecler1 10y ago
  A network should be no more complex than it needs to be. If you're not pruning spurious interactions then you're implicitly assuming that each interaction is information bearing. If you're not pruning spurious interactions from holographic networks then you're assuming that they're all functionally relevant. I promise you they're not. You're also assuming that holographic networks are somehow necessary for most of the tasks we require. The human brain, while having billions of neurons is still quite sparsely connected.
p1esk 10y ago
Pruning the network leads to a smaller network. How does that help with understanding? Does compressing the network from 100M parameters to 10M parameters reveal "the underlying circuit diagram" that you can understand?

smaddox 10y ago

Pruning works for understanding sparse networks, but not for holographic networks, which can encode a much higher information density. Understanding a holographic network requires a large amount of effort, similar to determining the content of an optical holograph. You can try to develop a model, but if the information density is very high, it's no faster than observing the response to various inputs. The problem becomes even more difficult if there is no clear distinctions between layers in the network, as in the brain.

rdlecler1 10y ago
A network should be no more complex than it needs to be. If you're not pruning spurious interactions then you're implicitly assuming that each interaction is information bearing. If you're not pruning spurious interactions from holographic networks then you're assuming that they're all functionally relevant. I promise you they're not. You're also assuming that holographic networks are somehow necessary for most of the tasks we require. The human brain, while having billions of neurons is still quite sparsely connected.

p1esk 10y ago

Pruning the network leads to a smaller network. How does that help with understanding? Does compressing the network from 100M parameters to 10M parameters reveal "the underlying circuit diagram" that you can understand?