Adam functions as a low-pass filter (and/or "compressor") on the gradient. It filters out "noise", which is "wild" during start of training. This is basically

Adam functions as a low-pass filter (and/or "compressor") on the gradient. It filters out "noise", which is "wild" during start of training. This is basically what all "optimizers" achieve in various ways, including momentum. Adam uses several times more memory, and is slower, than momentum or just SGD. That's reason to not use if not needed.