4 ms·
The derivative of x->x (identity) should be 1 regardless of your definition of derivative. Subgradients are only applicable when summing convex functions. Here
by throwawayiionqz 7y ago
The derivative of x->x (identity) should be 1 regardless of your definition of derivative.
Subgradients are only applicable when summing convex functions. Here relu(x) - relu(-x) is a sum of a convex function and a concave function.