4 ms·
> Directions we think are wide open > Second-order optimizers and natural gradient methods Do second order optimizers help improve data efficiency? I assumed
by bee_rider 8mo ago
> Directions we think are wide open
> Second-order optimizers and natural gradient methods
Do second order optimizers help improve data efficiency? I assumed they’d help you get to the same minimum faster (but this is way outside my wheelhouse).
- sdpmas 8mo agoyes! typically the optimizer that trains faster also get better data efficiency. it maybe not be absolutely true, but that has been my observation so far. also see https://arxiv.org/pdf/2510.09378 https://arxiv.org/pdf/2510.09378 for second-order methods.
- alyxya 8mo agoFundamentally I don't believe second-order methods get better data efficiency by itself, but changes to the optimizer can because the convergence behavior changes. ML theory lags behind the results in practice.
- vladf 8mo agoThat still looks like a “converge faster” paper. https://arxiv.org/abs/2006.10732 https://arxiv.org/abs/2006.10732 The above provides a nuanced theoretical view. GD inductive bias is probably better unless your model is misspecified