I haven't read more than the abstract of this particular paper, but it is expected that training behaves differently with/without adam. The problem with adam i

I haven't read more than the abstract of this particular paper, but it is expected that training behaves differently with/without adam. The problem with adam is that it keeps one more statistic (the same size as the model parameters) in memory. It also adds a little computation. The way to deal with it otherwise is to tune the momentum-parameters and clip/limit the gradient in various ways.