5 ms·

Also "dropout".

by gojomo 6y ago

Also "dropout".

sdenton4 6y ago
Yeah, dropout is conspicuously missing. For those who haven't heard: Dropout zeros out random activations, creating a random subnetwork to make a prediction. Each training step updates a different random subnetwork. (...Keeping in mind that two random subnetworks can share lots of weights...) Then at inference time, you use the full network, which now acts as an ensemble of all of the subnetworks. You could view it as 'thousand brains' with lots of weight sharing for efficiency+regularization.
ur-whale 6y ago
Much like backprop, I'm not sure dropout can be mapped back to the way an actual brain works.
- dcx 6y ago
  Wait, why can't you? Backprop is basically adjusting weights to get the model closer to making correct predictions. Isn't that basically how the brain does it? (Though I'm sure the way the brain adjusts its weights uses a different approximation of differentiation)
  - abeppu 6y ago
    I think the main issues are: - there's not a "correct" training prediction - there's not a "final layer"
  - ur-whale 6y ago
    Backprop is not simply about adjusting the weights (as a matter of fact, you can argue that any training method is about adjusting the weights). It's about computing the amount by which you adjust the weight. And unless there's been a major development in neuroscience that I'm not aware of, backprop is not the way the brain does it.
  - andyxor 6y ago
    Backprop requires global communication but biological brain works with local communication in neural assemblies, where signals spread in a way somewhat similar to diffusion. There is no global “teaching signal” or “delta rule” error correction. Learning via reward and punishment is a wrong level of abstraction for fundamental cognitive tasks like visual object recognition or “parsing” auditory signal. Sometime people mention dopamine as a kind of reinforcement signal but it operates on completely different time scale, orders of magnitude slower than any iterative optimization model would require. And the energy and time spent on iterative optimization in ANNs is not available to living organisms with constrained resources. If you’re interested in authoritative opinion on what kind of learning is biologically plausible see e.g. prof. Edmund Rolls recent book called “Brain Computations”.

Also "dropout".

by gojomo 6y ago

Also "dropout".

Yeah, dropout is conspicuously missing. For those who haven't heard: Dropout zeros out random activations, creating a random subnetwork to make a prediction. Each training step updates a different random subnetwork. (...Keeping in mind that two random subnetworks can share lots of weights...) Then at inference time, you use the full network, which now acts as an ensemble of all of the subnetworks. You could view it as 'thousand brains' with lots of weight sharing for efficiency+regularization.

ur-whale 6y ago

Much like backprop, I'm not sure dropout can be mapped back to the way an actual brain works.

dcx 6y ago
Wait, why can't you? Backprop is basically adjusting weights to get the model closer to making correct predictions. Isn't that basically how the brain does it? (Though I'm sure the way the brain adjusts its weights uses a different approximation of differentiation)
- abeppu 6y ago
  I think the main issues are: - there's not a "correct" training prediction - there's not a "final layer"
- ur-whale 6y ago
  Backprop is not simply about adjusting the weights (as a matter of fact, you can argue that any training method is about adjusting the weights). It's about computing the amount by which you adjust the weight. And unless there's been a major development in neuroscience that I'm not aware of, backprop is not the way the brain does it.
- andyxor 6y ago
  Backprop requires global communication but biological brain works with local communication in neural assemblies, where signals spread in a way somewhat similar to diffusion. There is no global “teaching signal” or “delta rule” error correction. Learning via reward and punishment is a wrong level of abstraction for fundamental cognitive tasks like visual object recognition or “parsing” auditory signal. Sometime people mention dopamine as a kind of reinforcement signal but it operates on completely different time scale, orders of magnitude slower than any iterative optimization model would require. And the energy and time spent on iterative optimization in ANNs is not available to living organisms with constrained resources. If you’re interested in authoritative opinion on what kind of learning is biologically plausible see e.g. prof. Edmund Rolls recent book called “Brain Computations”.