kokinon

3 ms·

1. 1.▲
   by kokinon 6y ago

   Nice hack! Although it doesn't work for modern networks, where the forward pass is done with int8 (not float, thus no +0/-0), and only the backward pass is float16/32. (common for networks used in mobile phones)
2. 2.▲
   by kokinon 6y ago

   To be fair, AI researchers used strictly differentiable functions (which is required for back-propagation) until recently. For example lenet5 uses the logistic function. Only in 2011, some smart-asses [1] :) experimented with rectifier unit
3. 3.▲
   by kokinon 6y ago

   Add: ah I see what you mean. In uni differential calculus we learned about differentiable from above(1), from below(0), so I assumed RELU is considered "differentiable", as opposed to Cantor set or Dirichlet / Thomae's f
4. 4.▲
   by kokinon 6y ago

   Do you mean symbolically non-differentiable? Because RELU is certainly numerically differentiable...