I've always had this idea that you could do a dumb version of annealing by just progressively increasing the size of samples used in the minibatches. SGD gives

I've always had this idea that you could do a dumb version of annealing by just progressively increasing the size of samples used in the minibatches. SGD gives you noise on the gradient which should get smaller as you increase the batch size. If you just adjusted the batch size with some sort of MH accept/reject logic you should get a better optimzer for mode seeking than just vanilla SGD. Like all my ideas though it has probably been thought before or is just wrong :) too damn lazy to try it as well.