Optimization as a tool is important and widely used, but... almost everything grabbing headlines uses some form of SGD + Momentum. Very little of the actual pro

Optimization as a tool is important and widely used, but... almost everything grabbing headlines uses some form of SGD + Momentum. Very little of the actual progress comes from better optimization.