3 ms·
I am wondering what is the state of things for fixing gradients are kinks, or if there is no hope for general automatic differentiation libraries. import j
by throwawayiionqz 7y ago
I am wondering what is the state of things for fixing gradients are kinks, or if there is no hope for general automatic differentiation libraries.
import jax.numpy as np
from jax import grad, jit, vmap
from jax import random
def relu(x):
return np.where(x>0, x, np.zeros_like(x))
def identity(x):
return relu(x) - relu(-x)
derivative_identity = grad(identity)
derivative_identity(0.0)
returns 1.0 or 0.0? It currently returns 0.0. (Edit: typo in sign)
- jl2718 7y ago0.0>0 is false.
- ajtulloch 7y agoSee https://en.m.wikipedia.org/wiki/Subderivative https://en.m.wikipedia.org/wiki/Subderivative
- throwawayiionqz 7y agoThe derivative of x->x (identity) should be 1 regardless of your definition of derivative. Subgradients are only applicable when summing convex functions. Here relu(x) - relu(-x) is a sum of a convex function and a concave function.
- tel 7y agoFor many kinks, it doesn't in practice seem to matter that much. Most applications are computing large stochastic derivatives which smooth out kinks through averaging. Critical kinks are those that affect the geometry of the gradient in systematic ways. For instance, a model with a mixture of discrete and continuous parameters. These are serious blockers and require more complex methods to solve such as Rao-Blackwellization (marginalizing out the discrete parameters). Generally this appears as model bias or substantially increased, often fatal variance in loss curves.