I also agree with "Cheap Gradient Principle". I see it as a case of "width of two horse backs from Ancient Rome determine Shuttle buster width" (which is untrue

I also agree with "Cheap Gradient Principle". I see it as a case of "width of two horse backs from Ancient Rome determine Shuttle buster width" (which is untrue but cool as a reference). The very SGD thing was developed because it was the only way to train something like neural network with small memory and, more importantly, in reasonable time. Multiplying of training time by N (number of parameter) meant having good result in a year, not in a day. And today we have large batch training with complex synchronization systems to speed up training even more. Which bring us closer to the whole-dataset training and, I guess, second-order optimization as well.