More specifically techniques such as Momentumn https://www.willamette.edu/~gorr/classes/cs449/momrate.html https://www.willamette.edu/~gorr/classes/cs449/momrat

More specifically techniques such as Momentumn https://www.willamette.edu/~gorr/classes/cs449/momrate.html https://www.willamette.edu/~gorr/classes/cs449/momrate.html (great article visualizing the problem space discussed above) and thigns like RMSPROP http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slid... are used. The rectified linear unit also converges about 75% faster than the classic Sigmoid or Tanh. the ReLu is F(x) = max(0, x) so the gradient propogated is 1 rather than a more "curvy" value.