4 ms·

As a non-mathematician, how did the word "differentiable" change its meaning? Is it not still that one can compute the derivative (so that Gradient Descent can

As a non-mathematician, how did the word "differentiable" change its meaning? Is it not still that one can compute the derivative (so that Gradient Descent can be applied)? (honest question)

enriquto 6y ago
The "canonical" example of non-differentiable function (for a traditional mathematician) is precisely RELU.
- kokinon 6y ago
  Do you mean symbolically non-differentiable? Because RELU is certainly numerically differentiable...
  - enriquto 6y ago
    What is RELU'(0) ?
    - deleted 6y ago
      [deleted]
    - dpwm 6y ago
      Under floating point we have two zeros, so we can have RELU'(+0) != RELU'(-0). Hence, I propose: RELU'(-0) = 0 RELU'(+0) = 1
      - kokinon 6y ago
        Nice hack! Although it doesn't work for modern networks, where the forward pass is done with int8 (not float, thus no +0/-0), and only the backward pass is float16/32. (common for networks used in mobile phones)
      - gugagore 6y ago
        Signed zeros are a tricky design choice, for sure... https://www.johndcook.com/blog/2010/06/15/why-computers-have-signed-zero/ https://www.johndcook.com/blog/2010/06/15/why-computers-have... I like this comment: > To other commenters: It’s important not to confuse floating point with mathematics. In floating point, both zeroes are considered to be exactly equal to zero, to each other. The redundant sign bit is “piggybacked” information which retains a sign across a series of multiplies or divisions, and it only affects downstream results in a few exceptional cases (eg dividing non-zero by zero, yielding inf or -inf). The behaviour has been standardized based on various pragmatic considerations, and will not always be consistent with evaluating infinitesimals in mathematics. I can see the reasoning that log(-0.0) should give the same exceptional result as log(-1). But bear in mind it can arise from something like log( -(a+b)) where a+b evaluates to 0. Whereas ((-a)-b) would be +zero. I.e the sign of the zero is generally not meaningful when the zero originally arises from a sum or difference. Indeed, in such cases, if you consider a and b to carry “rounding fuzz”, then the proper sign of a+b is indeterminate, and not correlated to the sign bit generated by the add. When zeroes are generated by underflow of mul or div, at least the sign is “correct”.
  - kokinon 6y ago
    Add: ah I see what you mean. In uni differential calculus we learned about differentiable from above(1), from below(0), so I assumed RELU is considered "differentiable", as opposed to Cantor set or Dirichlet / Thomae's function...
- sabertoothed 6y ago
  But the definition of differentiable did not change, did it? RELU is not differentiable (or - in other words - is differentiable at all points except 0). Not nitpicking, just trying to improve my understanding.
  - enriquto 6y ago
    > But the definition of differentiable did not change, did it? It did! In the title of this article they are describing a blatantly non-differentiable function (the rasterizer) using the word "differentiable". This is indeed a new usage of the word, only seen since the advent of automatic differentiation a few years ago.
    - pixelpoet 6y ago
      How is it non-differentiable? They explicitly point out that after prefiltering (convolution with a pixel filter), it is differentiable. Moreover, these derivatives can be obtained via methods like Automatic Differentiation, so they are true derivatives, not finite difference approximations: https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation
    - kokinon 6y ago
      To be fair, AI researchers used strictly differentiable functions (which is required for back-propagation) until recently. For example lenet5 uses the logistic function. Only in 2011, some smart-asses [1] :) experimented with rectifier units and discovered they're even better [1] Xavier Glorot, Antoine Bordes and Yoshua Bengio - Deep sparse rectifier neural networks (2011)
    - DougBTX 6y ago
      They make the point in their presentation to say that their method replaces the non-differentiable step function with a differentiable sigmoid function to enable the rasterizer to be differentiable.
- im3w1l 6y ago
  Well neural networks based on relu are almost everywhere differentiable.
- IdiocyInAction 6y ago
  It is subdifferentiable though.
  - enriquto 6y ago
    Sure, and also differentiable in the sense of distributions. Still, not differentiable.
- beagle3 6y ago
  The really canonical non-differentiable function is the Takagi/Blancmange Function[0] if you like things deterministic, and the Wiener Process[1] if you like stochastic processes. Both are everywhere-continuous, nowhere-differentiable. [0] https://en.wikipedia.org/wiki/Blancmange_curve https://en.wikipedia.org/wiki/Blancmange_curve [1] https://en.wikipedia.org/wiki/Wiener_process https://en.wikipedia.org/wiki/Wiener_process
  - kwkelly 6y ago
    The canonical example in my experience is always the Weierstrass function

As a non-mathematician, how did the word "differentiable" change its meaning? Is it not still that one can compute the derivative (so that Gradient Descent can be applied)? (honest question)

enriquto 6y ago

The "canonical" example of non-differentiable function (for a traditional mathematician) is precisely RELU.

kokinon 6y ago
Do you mean symbolically non-differentiable? Because RELU is certainly numerically differentiable...
- enriquto 6y ago
  What is RELU'(0) ?
  - deleted 6y ago
    [deleted]
  - dpwm 6y ago
    Under floating point we have two zeros, so we can have RELU'(+0) != RELU'(-0). Hence, I propose: RELU'(-0) = 0 RELU'(+0) = 1
    - kokinon 6y ago
      Nice hack! Although it doesn't work for modern networks, where the forward pass is done with int8 (not float, thus no +0/-0), and only the backward pass is float16/32. (common for networks used in mobile phones)
    - gugagore 6y ago
      Signed zeros are a tricky design choice, for sure... https://www.johndcook.com/blog/2010/06/15/why-computers-have-signed-zero/ https://www.johndcook.com/blog/2010/06/15/why-computers-have... I like this comment: > To other commenters: It’s important not to confuse floating point with mathematics. In floating point, both zeroes are considered to be exactly equal to zero, to each other. The redundant sign bit is “piggybacked” information which retains a sign across a series of multiplies or divisions, and it only affects downstream results in a few exceptional cases (eg dividing non-zero by zero, yielding inf or -inf). The behaviour has been standardized based on various pragmatic considerations, and will not always be consistent with evaluating infinitesimals in mathematics. I can see the reasoning that log(-0.0) should give the same exceptional result as log(-1). But bear in mind it can arise from something like log( -(a+b)) where a+b evaluates to 0. Whereas ((-a)-b) would be +zero. I.e the sign of the zero is generally not meaningful when the zero originally arises from a sum or difference. Indeed, in such cases, if you consider a and b to carry “rounding fuzz”, then the proper sign of a+b is indeterminate, and not correlated to the sign bit generated by the add. When zeroes are generated by underflow of mul or div, at least the sign is “correct”.
- kokinon 6y ago
  Add: ah I see what you mean. In uni differential calculus we learned about differentiable from above(1), from below(0), so I assumed RELU is considered "differentiable", as opposed to Cantor set or Dirichlet / Thomae's function...
sabertoothed 6y ago
But the definition of differentiable did not change, did it? RELU is not differentiable (or - in other words - is differentiable at all points except 0). Not nitpicking, just trying to improve my understanding.
- enriquto 6y ago
  > But the definition of differentiable did not change, did it? It did! In the title of this article they are describing a blatantly non-differentiable function (the rasterizer) using the word "differentiable". This is indeed a new usage of the word, only seen since the advent of automatic differentiation a few years ago.
  - pixelpoet 6y ago
    How is it non-differentiable? They explicitly point out that after prefiltering (convolution with a pixel filter), it is differentiable. Moreover, these derivatives can be obtained via methods like Automatic Differentiation, so they are true derivatives, not finite difference approximations: https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation
  - kokinon 6y ago
    To be fair, AI researchers used strictly differentiable functions (which is required for back-propagation) until recently. For example lenet5 uses the logistic function. Only in 2011, some smart-asses [1] :) experimented with rectifier units and discovered they're even better [1] Xavier Glorot, Antoine Bordes and Yoshua Bengio - Deep sparse rectifier neural networks (2011)
  - DougBTX 6y ago
    They make the point in their presentation to say that their method replaces the non-differentiable step function with a differentiable sigmoid function to enable the rasterizer to be differentiable.
im3w1l 6y ago
Well neural networks based on relu are almost everywhere differentiable.
IdiocyInAction 6y ago
It is subdifferentiable though.
- enriquto 6y ago
  Sure, and also differentiable in the sense of distributions. Still, not differentiable.
beagle3 6y ago
The really canonical non-differentiable function is the Takagi/Blancmange Function[0] if you like things deterministic, and the Wiener Process[1] if you like stochastic processes. Both are everywhere-continuous, nowhere-differentiable. [0] https://en.wikipedia.org/wiki/Blancmange_curve https://en.wikipedia.org/wiki/Blancmange_curve [1] https://en.wikipedia.org/wiki/Wiener_process https://en.wikipedia.org/wiki/Wiener_process
- kwkelly 6y ago
  The canonical example in my experience is always the Weierstrass function