3 ms·
I understand the fragment you quoted to be accurate, and the problem described is mostly ameliorated by optimizers. But, I don't understand your criticism of th
by CyberShadow 5y ago
I understand the fragment you quoted to be accurate, and the problem described is mostly ameliorated by optimizers. But, I don't understand your criticism of the quoted text. Would you mind elaborating?
- jmmcd 5y agoThe idea of the kangaroo being blind and "feeling around" might be a good metaphor for an optimiser which uses numerical estimation of the gradient, but backprop does not do that. At a stretch, one could argue that being blind is a good metaphor for metaheuristic optimisation, but "feeling around" on the ground is not a good metaphor for metaheuristic optimisation. If we are trying to find a metaphor for backprop, I think we are in trouble, because the central idea in backprop is that we have to calculate the gradient for some variables (eg the north-south variable) before we can calculate the gradient for some others (eg the east-west variable).