4 ms·

> Networks trained on a path integration task almost always learn to optimally encode position, but almost never learn lattice cells (hexagonal or square) to do

> Networks trained on a path integration task almost always learn to optimally encode position, but almost never learn lattice cells (hexagonal or square) to do so... Our main message is that it is highly improbable that DL models of path integration would have produced grid cells as a novel prediction simply from task-training, had grid cells not already been known to exist. This carries on into an extremely nuanced and technical discussion of the architecture of specific models used. It's basically pointing out that in some previous papers, authors thought that grid cells always arose when solving this problem, but in fact this only occurs when specific implementation choices are made. So those papers were incomplete, and the phenomenon isn't as clearcut as before. However! This new complexity still tells us something; if only certain architectural choices produce grid cells, then brains must (in some sense) implement those architectural choices. And the models that don't produce grid cells must be doing something differently to how the brain does things. In summary I think this paper is probably saying a lot less than most people here are reading into it; some papers are accidentally oversimplifying, and we've found more complexity that needs to be explained. More thorough hyperparameter-space exploration can identify brittle results. It's not some deep point about whether it is philosophically or logically consistent to compare deep NNs to the brain.

danielheath 4y ago
In wet-brains: Interlacing isn't 4-way or 6-way, it's 10e3-way, and each interlaced connection has a weight that's nonlinearly time-dependant based on how long since last firing. Every cyclic connection is potentially a self-sustaining oscillator. None of these features are efficiently implemented in current silicon. "Caution when comparing neural networks to brains" is underselling it. They're profoundly different kinds of network; nobody is building (or even publicly planning) silicon that has that kind of interconnect breadth.
- isaacfrond 4y ago
  What do you mean? Any image classifier will use way more than 4 kernels for convolution. All those layers are interlaced. Furthermore they also contain fully connected layers, with neuron integrating way more than 10^3 signals. The reasons that there aren't much more fully connected layers, is that this doesn't work. Actually, one of the key developments in NNs is architectural, minpools, ReLus, U-net. All are key for modern networks, all architectural.
  - danielheath 4y ago
    Sorry, that was hastily written & could have been clearer. It is, of course, very common to perform (eg) convolution with a larger kernel size, or to use a dense layer. However, unlike wet-neurons: * Convolution has the same local shape for each cell. * Convolution has no self-suppression for recent activation, vs time-dependent, nonlinear response in wet cells. * Current silicon offers no performance advantage for interconnect to adjacent cells (could be done). With ~80 billion neurons in a brain, 1000 is not like a dense layer at all.

danielheath 4y ago

In wet-brains: Interlacing isn't 4-way or 6-way, it's 10e3-way, and each interlaced connection has a weight that's nonlinearly time-dependant based on how long since last firing. Every cyclic connection is potentially a self-sustaining oscillator. None of these features are efficiently implemented in current silicon. "Caution when comparing neural networks to brains" is underselling it. They're profoundly different kinds of network; nobody is building (or even publicly planning) silicon that has that kind of interconnect breadth.

isaacfrond 4y ago
What do you mean? Any image classifier will use way more than 4 kernels for convolution. All those layers are interlaced. Furthermore they also contain fully connected layers, with neuron integrating way more than 10^3 signals. The reasons that there aren't much more fully connected layers, is that this doesn't work. Actually, one of the key developments in NNs is architectural, minpools, ReLus, U-net. All are key for modern networks, all architectural.
- danielheath 4y ago
  Sorry, that was hastily written & could have been clearer. It is, of course, very common to perform (eg) convolution with a larger kernel size, or to use a dense layer. However, unlike wet-neurons: * Convolution has the same local shape for each cell. * Convolution has no self-suppression for recent activation, vs time-dependent, nonlinear response in wet cells. * Current silicon offers no performance advantage for interconnect to adjacent cells (could be done). With ~80 billion neurons in a brain, 1000 is not like a dense layer at all.