3 ms·

This is good, I think research in this direction will yield the next breakthrough in ML. The current hierarchical feed-forward model of neural networks is what'

This is good, I think research in this direction will yield the next breakthrough in ML. The current hierarchical feed-forward model of neural networks is what's limiting our advances. If you look at ResNet or DenseNet, their skip-connections are hacks to bypass that limitation and it brings great improvements. And if you look at the rooting by agreement technique for capsule networks, it's clear that getting biases from later layers to earlier layers improves things as well. And even in our own virtual cortex, it's only up to V1 that things are remotely hierarchical, after that it's a mess of looping wiring between areas. We need to throw away the current neural network and adopt a new paradigm that can express both feed-forward and feed-backward connections innately.

IdiocyInAction 6y ago
It's not like that hasn't been tried before though. Boltzmann machines have all-to-all connectivity and RNNs have feedback connections (and sometimes backward connections). NNs have been studied since the 90s and a lot has been tried already. I think one should also keep the bitter lesson in mind.
- ad404b8a372f2b9 6y ago
  Convolutional neural networks were also tried and failed to fulfill their potential for 20 years (or 50 depending on how far you go back) until they didn't and now they're ubiquitous. You cannot discard a whole area of research because one or some implementations of a concept have so far failed to become competitive. All to all connectivity is possibly the worst paradigm unless you want to do architecture search so I wouldn't hold it as an example of anything wrt this concept. As for recurrent neural networks to my knowledge they've only been used for recurrence in time or space not recurrence between layers for a single sample, though I might be wrong, so they're not relevant to what I'm speaking of. Though there is some work on skip connections (forward and backward) which takes some inspiration from their gating mechanisms.

IdiocyInAction 6y ago

It's not like that hasn't been tried before though. Boltzmann machines have all-to-all connectivity and RNNs have feedback connections (and sometimes backward connections). NNs have been studied since the 90s and a lot has been tried already. I think one should also keep the bitter lesson in mind.

ad404b8a372f2b9 6y ago
Convolutional neural networks were also tried and failed to fulfill their potential for 20 years (or 50 depending on how far you go back) until they didn't and now they're ubiquitous. You cannot discard a whole area of research because one or some implementations of a concept have so far failed to become competitive. All to all connectivity is possibly the worst paradigm unless you want to do architecture search so I wouldn't hold it as an example of anything wrt this concept. As for recurrent neural networks to my knowledge they've only been used for recurrence in time or space not recurrence between layers for a single sample, though I might be wrong, so they're not relevant to what I'm speaking of. Though there is some work on skip connections (forward and backward) which takes some inspiration from their gating mechanisms.