3 ms·
It seems like the implicit target in the document is to achieve a critically damped system with no ringdown on learning. However, if they're trying to go for s
by araes 12y ago
It seems like the implicit target in the document is to achieve a critically damped system with no ringdown on learning. However, if they're trying to go for speed, then it seems like they should accept possible overshoot, and use non-linear control theory for their weights so that they're underdamped during the initial descent, and then transition into critically damped gradient descent as they move into the flat zone. Something like a variable "damper" or weights/springs based on current error. Perhaps that is done elsewhere though, and just not described as a technique here.
- Houshalter 12y agoThat sort of sounds like Rprop which is supposedly the fastest learning algorithm. There are also other "adaptive learning rate" algorithms. https://en.wikipedia.org/wiki/Rprop https://en.wikipedia.org/wiki/Rprop
- araes 12y agoVery cool. I had never heard of Rprop, but that's a neat way to trigger your learning that it needs to rapidly damp. Kind of like a limiter in CFD.