5 ms·
this would be the dream, but biological neural networks don't do gradient decent, so have little to do with ANNs until we discover how they really train.
by make3 8y ago
this would be the dream, but biological neural networks don't do gradient decent, so have little to do with ANNs until we discover how they really train.
- jacquesm 8y agoIt is perfectly possible to build neural networks with other functions besides gradient descent.
- make3 8y agoI agree. I'm saying we don't know how biological neural networks are trained, which is a pretty important part of any learning construct, so we can't really know how actually similar artificial neural nets are.
- taneq 8y agoWhat other methods are there for training a neural network? Even blind evolutionary algorithms (assuming each generation is initialized to be somewhat 'near' or 'similar' to its progenitors) are effectively descending a gradient.
- luk32 8y agoTo my knowledge they explore solution space but not through a gradient descent but by probing. How can you perform a GD on non continuous function? Genetic algorithms and evolutionary methods can explore disjoint domains and functions hard from analytical pov. Just because they can be used for gradient descend doesn't mean they use it.
- akyu 8y agoIt would be remarkable if biological neural networks don't use gradient descent. They very likely do even if it is a highly approximate gradient. Neurons are optimizing for some objective, and the gradient is usually a very efficient way to do that optimization. It would be remarkable if evolution missed out on such a thing. Even simple Hebbian learning can be viewed as a gradient descent technique. Gradients can be calculated in remarkably simple ways. Look at Hinton's recirculation networks for example. I think you meant to say backprop.
- gnulinux 8y agoHow would brain do gradient descent without backpropagation?
- eli_gottlieb 8y agoMicrocircuits could descend the gradient of a function they can evaluate locally.