I want to add to this, that there's a lot of work that doesn't require pretraining. I recently implemented the more advanced hessian free optimization methods t

I want to add to this, that there's a lot of work that doesn't require pretraining. I recently implemented the more advanced hessian free optimization methods that don't require pretraining in my deep learning framework. The results are amazing. I'm hoping to demonstrate a lot of the tradeoffs of the different methods in a more comprehensive manner here shortly. This was an updated extension by some of hinton's students. The paper I implemented was: http://www.cs.toronto.edu/~rkiros/papers/shf13.pdf http://www.cs.toronto.edu/~rkiros/papers/shf13.pdf