4 ms·

> Actually, that is the correct definition! Hmmm; no, it isn't? Your quotation is vague stuff and not a definition at all. The notion of differentiability ha

> Actually, that is the correct definition! Hmmm; no, it isn't? Your quotation is vague stuff and not a definition at all. The notion of differentiability has nothing to do with deep learning, rendering, nor neural networks. None of these terms should appear on a definition of the concept. The classical mathematical definition of differentiability concerns the function itself. For example, the function f(x)=x^2 is differentiable while the function g(x)=abs(x) isn't. On the other hand, the "modern" definition of differentiability concerns the computer implementation of the function. Given the same function f(x)=x^2, you can have a differentiable implementation of it (e.g., using a high level language) and a non-differentiable implementation (e.g., calling a library written in assembler that evaluates this function). At the same time, you can have a differentiable implementation of g(x)=abs(x), which is the sign function, and non-differentiable implementation as well. Thus, both concepts are really independent! I'm still longing for a formal, canonical, definition of the "modern" notion of differentiability.

andersource 4y ago
Parent is correct that the definition you gave is indeed broadly the correct definition, i.e. there's no principle difference between calculus "differentiable" and differentiable as in differentiable rendering, except maybe the modern sense is a bit more relaxed. The differentiability of the rendering process doesn't depend on the implementation - incidentally look at the first line of this paper's abstract: > Differentiable renderers provide a direct mathematical link between an object's 3D representation and images of that object When we call a process differentiable (in the "modern" sense you're referring to) we simply mean that we have a formulation of the process as a differentiable function. In rendering this means the function's domain is the set of tuples (3D object, camera parameters) and the function's range is a set of 2D images of some sort. These are very high-dimensional functions, but mathematical functions nonetheless, and calling them "differentiable" means the same as in the definition you linked to. Your examples re/ implementation make me think maybe you're conflating differentiability with "autodiff"[0] - languages / libraries that allow you to just write a function and automatically differentiate it for you. [0] https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation
- enriquto 4y ago
  > this means the function's domain is the set of tuples (3D object, camera parameters) and the function's range is a set of 2D images of some sort What would be an example of non-differentiable rendering according to this?
  - andersource 4y ago
    Most rendering pipelines, as they involve discrete operations. For example in mesh rendering, the step of polygon rasterization is typically non-differentiable, because every pixel is either inside some polygon or isn't.
    - enriquto 4y ago
      It would seem that you mean "continuous" here, instead of "differentiable".
      - andersource 4y ago
        No, I mean (non-)differentiable, although differentiable non-continuous is also not useful. If you scanned every pixel for every polygon and discretely determined whether it was inside or outside that would be as you describe, but that's rarely the case. Take a look at the typical line drawing algorithm[0] - it procedurally determines which pixels to draw as part of the line. Without modification such algorithms don't give you derivatives of how each pixel depends on the shape parameters. [0] https://en.m.wikipedia.org/wiki/Bresenham%27s_line_algorithm https://en.m.wikipedia.org/wiki/Bresenham%27s_line_algorithm
6gvONxR4sf7o 4y ago
You’re confusing “everywhere differentiable” with “differentiable at the points we care about.” g(x) = abs(x) is differentiable almost everywhere (not in the Cantor’s Staircase sense, but in a practical sense), and that’s enough for gradient descent to work in practice. And fitting models by gradient-descent-like methods is the single unifying feature of deep learning. I don’t understand the implementation thing you’re worrying about. Assuming it takes the same inputs to the same outputs, how you implement a function has no bearing on its limit properties, which is what differentiability is about.
HyperSane 4y ago
https://www.jeremyjordan.me/neural-networks-training/ https://www.jeremyjordan.me/neural-networks-training/
mrkeen 4y ago
What about differential equations? Should we stop calling them that if we can't find the slope of an equation?

andersource 4y ago

Parent is correct that the definition you gave is indeed broadly the correct definition, i.e. there's no principle difference between calculus "differentiable" and differentiable as in differentiable rendering, except maybe the modern sense is a bit more relaxed. The differentiability of the rendering process doesn't depend on the implementation - incidentally look at the first line of this paper's abstract: > Differentiable renderers provide a direct mathematical link between an object's 3D representation and images of that object When we call a process differentiable (in the "modern" sense you're referring to) we simply mean that we have a formulation of the process as a differentiable function. In rendering this means the function's domain is the set of tuples (3D object, camera parameters) and the function's range is a set of 2D images of some sort. These are very high-dimensional functions, but mathematical functions nonetheless, and calling them "differentiable" means the same as in the definition you linked to. Your examples re/ implementation make me think maybe you're conflating differentiability with "autodiff"[0] - languages / libraries that allow you to just write a function and automatically differentiate it for you. [0] https://en.wikipedia.org/wiki/Automatic_differentiation https://en.wikipedia.org/wiki/Automatic_differentiation

enriquto 4y ago
> this means the function's domain is the set of tuples (3D object, camera parameters) and the function's range is a set of 2D images of some sort What would be an example of non-differentiable rendering according to this?
- andersource 4y ago
  Most rendering pipelines, as they involve discrete operations. For example in mesh rendering, the step of polygon rasterization is typically non-differentiable, because every pixel is either inside some polygon or isn't.
  - enriquto 4y ago
    It would seem that you mean "continuous" here, instead of "differentiable".
    - andersource 4y ago
      No, I mean (non-)differentiable, although differentiable non-continuous is also not useful. If you scanned every pixel for every polygon and discretely determined whether it was inside or outside that would be as you describe, but that's rarely the case. Take a look at the typical line drawing algorithm[0] - it procedurally determines which pixels to draw as part of the line. Without modification such algorithms don't give you derivatives of how each pixel depends on the shape parameters. [0] https://en.m.wikipedia.org/wiki/Bresenham%27s_line_algorithm https://en.m.wikipedia.org/wiki/Bresenham%27s_line_algorithm

6gvONxR4sf7o 4y ago

You’re confusing “everywhere differentiable” with “differentiable at the points we care about.” g(x) = abs(x) is differentiable almost everywhere (not in the Cantor’s Staircase sense, but in a practical sense), and that’s enough for gradient descent to work in practice. And fitting models by gradient-descent-like methods is the single unifying feature of deep learning. I don’t understand the implementation thing you’re worrying about. Assuming it takes the same inputs to the same outputs, how you implement a function has no bearing on its limit properties, which is what differentiability is about.

HyperSane 4y ago

https://www.jeremyjordan.me/neural-networks-training/ https://www.jeremyjordan.me/neural-networks-training/

mrkeen 4y ago

What about differential equations? Should we stop calling them that if we can't find the slope of an equation?