This reminds me of something that came up in Andrew Ng's online ML class. He said that it is important to check the correctness of your gradient calculation in

This reminds me of something that came up in Andrew Ng's online ML class. He said that it is important to check the correctness of your gradient calculation in backprop (by comparing it to a finite difference of the loss) because if you have a bug there, your algorithm might more or less work anyway, making it hard to tell that there was a bug. Apparently you can still get sensible output even with an incorrect gradient.