3 ms·
Nitpick: newton's method is intractable in Deep learning since it's computationally intractable to compute the hessian matrix
by deepnotderp 10y ago
Nitpick: newton's method is intractable in Deep learning since it's computationally intractable to compute the hessian matrix
- montecarl 10y agoNewton's method might be, but quasi-Newton methods like L-BFGS[1] are usable if you can calculate the gradient of your problem and have enough memory to store several previous coordinates and gradients (so maybe 10x the memory requirement of gradient descent). [1] https://en.wikipedia.org/wiki/Limited-memory_BFGS https://en.wikipedia.org/wiki/Limited-memory_BFGS
- throwaway61AR 10y agoYep this was used by Jeff Dean et al. in one of Google Brain's earlier experiments https://research.google.com/archive/large_deep_networks_nips2012.html https://research.google.com/archive/large_deep_networks_nips...