3 ms·
I tried it to train a CNN-based CIFAR10 classifier, which worked well (only a tiny bit worse than Adam, but the difference might go away with hyper parameter tu
by johndough 2y ago
I tried it to train a CNN-based CIFAR10 classifier, which worked well (only a tiny bit worse than Adam, but the difference might go away with hyper parameter tuning), but the optimizer totally failed (loss -> infinity) when training a U-Net for an image segmentation task. I had to increase eps to 1e-4 and decrease lr to 1e-3 so it would not explode, but that made it very slow to converge.
My summary is that the memory savings might be great if it works, but it does not work everywhere.
- eden-u4 2y agoah, numerical instability in the warmup stage might be the issue then?
- jszymborski 2y agoYah I mean that's the rub with SGD... you need to spend a non-trivial compute budget on hyperparam tuning, which sometimes beats Adam. Adam, on the other hand, generally gets you pretty good results without futzing too much with hyper params.