7 ms·
Karpathy suggests the following error: def clipped_error(x): return tf.select(tf.abs(x) < 1.0, 0.5 * tf.square(x),
by WithinReason 11mo ago
Karpathy suggests the following error:
def clipped_error(x):
return tf.select(tf.abs(x) < 1.0,
0.5 * tf.square(x),
tf.abs(x) - 0.5) # condition, true, false
Following the same principles that he outlines in this post, the "- 0.5" part is unnecessary since the gradient of 0.5 is 0, therefore -0.5 doesn't change the backpropagated gradient. In addition, a nicer formula that achieves the same goal as the above is √(x²+1)
- macleginn 11mo agoIf we don't subtract from the second branch, there will be a discontinuity around x = 1, so the derivative will not be well-defined. Also the value of the loss will jump at this value, which will make it hard to inspect the errors, for one thing.
- WithinReason 11mo agoNo, that's not how backprop works. There will be no discontinuity in a backpropagated gradient.
- macleginn 11mo agoI did not say there will be a discontinuity in the gradient; I said that the modified loss function will not have a mathematically well-defined derivative because of the discontinuity in the function.
- WithinReason 11mo agoWhich is completely irrelevant to the point I was making
- kingstnap 11mo agoYou do that to make things smoother when plotted. You could in theory add some crazy stairstep that adds a hundred to the middle part. It would make your loss curves spike and increase towards convergence but then those spikes are just visual artifacts from doing weird discontinuous nonsense with yoru loss.
- slashdave 11mo agosquare roots are expensive
- WithinReason 11mo agothey are negligible, especially when the post was written when ops were not fused. The extra memory you need to store the extra tensors when you use the original version is more expensive