4 ms·
It's not the backprop that is holding back ML - it takes about as much time as the forward pass and requires 3x the memory. Backprop is necessary for almost all
by visarga 7y ago
It's not the backprop that is holding back ML - it takes about as much time as the forward pass and requires 3x the memory. Backprop is necessary for almost all the deep neural nets that have state of the art results, attempting to replace it would push us back a decade or more.
The main problem with ML is the separation of data and compute. It takes a lot of time and energy to move data around. We need 'in memory compute'.
- cr0sh 7y agoSo you're in disagreement with Hinton? https://www.axios.com/artificial-intelligence-pioneer-says-we-need-to-start-over-1513305524-f619efbd-9db0-4947-a9b2-7a4c310a28fe.html https://www.axios.com/artificial-intelligence-pioneer-says-w... I've heard the argument made about "in memory compute" - and I do know that various companies have made hardware toward that direction (and some of it has very low power requirements). I just find it compelling that Hinton - at least as of 2 years ago - has this viewpoint that we should be rethinking the concept of backprop...
- bllguo 7y agothat article is way too vague for agreement/disagreement to be a productive point of discussion imo. is he talking about new optimization methods?
- jasallen 7y agoI'd say this is a difference of scale. Hinton's comments are meant to inspire and encourage _The Next Great Thing_. We could read the paper that broadly, but they seem to be addressing a more immediate local minima.
- bigred100 7y agoIsn’t backpropagation just an efficient method for getting a gradient? It sounds like what he’s looking for an alternative to is the idea of adjusting parameters to match labeled training data.