3 ms·

The brain being poised towards criticality is a serious meme in the neuro community. Not that it's an uninteresting idea or one unworthy of pursuit. Just a bit

The brain being poised towards criticality is a serious meme in the neuro community. Not that it's an uninteresting idea or one unworthy of pursuit. Just a bit hard to have people to take you seriously.

longtom 7y ago
See also: https://twitter.com/farlkriston https://twitter.com/farlkriston Well, these ideas are very likely accurate, but so general that they are disconnected from solving practical problems. Fine, the brain is a prediction machine and it optimizes over some program space by annealing doing homeostasis/regulation, and maybe tends to occupy certain kinds of states now and then. This, however, tells us very little what the learning rules for the synaptic weights should be and how we should wire things up. In fact, I believe human-relevant problems are best solved by such a special subregion of program space that one needs pretty specific architectural priors as otherwise search will take too long. These are unlikely to be derived from general concepts, but are more likely evolved, either literally by evolutionary algorithms or by people doing the trial and error. The issue being that general concepts about prediction errors and program spaces know nothing about our specific world. E.g. none of these general concepts predict the usefulness of CNNs. CNNs exploit fairly specialized priors about object translation invariance and locality in image statistics, which are specific computations occurring in our universe when parts of it are perceived by geometric projections of EM rays onto an image plane with sensors. Hinton's capsules go into the right direction exploiting some more priors about spatial reference point invariance, but we need to go deeper. The brain disassembles the world into stable episodic chunks and operates on them, and it manages to backpropagate values through such episodic memories. Currently, no neural architecture does something like this.

longtom 7y ago

See also: https://twitter.com/farlkriston https://twitter.com/farlkriston Well, these ideas are very likely accurate, but so general that they are disconnected from solving practical problems. Fine, the brain is a prediction machine and it optimizes over some program space by annealing doing homeostasis/regulation, and maybe tends to occupy certain kinds of states now and then. This, however, tells us very little what the learning rules for the synaptic weights should be and how we should wire things up. In fact, I believe human-relevant problems are best solved by such a special subregion of program space that one needs pretty specific architectural priors as otherwise search will take too long. These are unlikely to be derived from general concepts, but are more likely evolved, either literally by evolutionary algorithms or by people doing the trial and error. The issue being that general concepts about prediction errors and program spaces know nothing about our specific world. E.g. none of these general concepts predict the usefulness of CNNs. CNNs exploit fairly specialized priors about object translation invariance and locality in image statistics, which are specific computations occurring in our universe when parts of it are perceived by geometric projections of EM rays onto an image plane with sensors. Hinton's capsules go into the right direction exploiting some more priors about spatial reference point invariance, but we need to go deeper. The brain disassembles the world into stable episodic chunks and operates on them, and it manages to backpropagate values through such episodic memories. Currently, no neural architecture does something like this.