Yah I mean that's the rub with SGD... you need to spend a non-trivial compute budget on hyperparam tuning, which sometimes beats Adam. Adam, on the other hand,

Yah I mean that's the rub with SGD... you need to spend a non-trivial compute budget on hyperparam tuning, which sometimes beats Adam. Adam, on the other hand, generally gets you pretty good results without futzing too much with hyper params.