4 ms·
Newton's method might be, but quasi-Newton methods like L-BFGS[1] are usable if you can calculate the gradient of your problem and have enough memory to store s
by montecarl 10y ago
Newton's method might be, but quasi-Newton methods like L-BFGS[1] are usable if you can calculate the gradient of your problem and have enough memory to store several previous coordinates and gradients (so maybe 10x the memory requirement of gradient descent).
[1] https://en.wikipedia.org/wiki/Limited-memory_BFGS https://en.wikipedia.org/wiki/Limited-memory_BFGS
- throwaway61AR 10y agoYep this was used by Jeff Dean et al. in one of Google Brain's earlier experiments https://research.google.com/archive/large_deep_networks_nips2012.html https://research.google.com/archive/large_deep_networks_nips...