17 ms·

One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x. A

by yarg 4y ago

One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x. And I don't think that the common differential equation is the ideal form when dealing with that situation. (Take for example g(x) = -f(-x), the gradients at x and -x are not equivalent for Δx, but would be for 𝛿x.) Anyway, (limit(h -> 0)((f(x + h) - f(x))/h) Why do we even need h? It's just (the infinitesimal) 𝛿x multiplied by an arbitrary finite constant. (You're dealing with a linear equivalent infinitesimal subsection of the graph - so 𝛿x and 𝛿y scale linearly with each other.) So remove h: ((f(x + 𝛿x) - f(x))/𝛿x) Looks cleaner. (f(x + 𝛿x) - f(x))/𝛿x ≡ (f(x) - f(x - 𝛿x))/𝛿x ≡ (f(x + 𝛿x) - f(x - 𝛿x))/(2 * 𝛿x) You can take the gradient from x to (x + 𝛿x) or from (x - 𝛿x) to x, or from (x - 𝛿x) to (x + 𝛿x). (The cube-root of the other three forms is also a valid differential equation, if you want to be evil about it.) (f(x + 𝛿x) - f(x - 𝛿x))/(2 * 𝛿x) (That's a nicer form for programmatic use; resolving the earlier mentioned gradient issue when dealing with finite deltas.) Another thing I noticed: the standard (h -> 0) form eliminates all parts of the gradient containing 𝛿x values - which is fine for infinitesimal 𝛿x, but is less ideal for finite Δx.

ngcc_hk 4y ago
Is h use instead of 𝛿 or Δ, because it is applicable to both partial and normal differential. 𝛿 Is always a partial symbol to me. d in dx whilst is not the same as h, as it stands its own way now as “operator”.
- hither_shores 4y ago
  No, the h here is a number, not an operator.
- yarg 4y ago
  I just find that it buries the lede.
joe__f 4y ago
If you're interested in how to best calculate derivatives numerically using a finite difference rather than an infinitesimal, try having a look at the Wikipedia page on numerical differentiation. The punchline is that you can control the error in f'(x) much better if you use many displacements, ie. f'(x) = a1 f(x + b1 h) + a2 f(x + b2 h) + a3 f(x + b3 h)..., where the bi are chosen and the ai depend on the bi and h.
- ogogmad 4y ago
  Such methods are doomed. Consider the function f(x) = a sin(x / a) for very small a. Finite differencing methods would approximate f'(0) as 0 as f(x) is very close to 0, but the actual value of f'(0) is 1. Differentiation is a discontinuous operator, and cannot be accurately estimated using numerical methods. Only exact methods, which include the dual numbers, can produce accurate results.
  - goldenkey 4y ago
    This is not true. Doomed is sensational. With proper perturbation analysis, one can choose an appropriate stencil and formula that has desired error bounds. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72.3529&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72... https://www.google.com/search?q=finite+difference+stencil+derivative+sensitive+functions https://www.google.com/search?q=finite+difference+stencil+de...
    - ogogmad 4y ago
      Won't work on my example. Given any differentiable function f:R -> R, you can add a very small function to it that dramatically changes its derivative. But maybe this is too pathological to bother about. Over the complex numbers, this phenomenon doesn't happen. This can be shown, for instance, using the maximum modulus principle. You can therefore use Cauchy's Integral Formula to accurately estimate derivatives. But why not then just use the dual numbers?
      - 4y ago
godelski 4y ago
You really just derived the central derivative. Yes, this derivative is more accurate. The issue is that you can't use it on the edges. For the boundaries you're stuck with lower precision. Here's a resource you might be interested in http://www2.math.umd.edu/~dlevy/classes/amsc466/lecture-notes/differentiation-chap.pdf http://www2.math.umd.edu/~dlevy/classes/amsc466/lecture-note...
- ogogmad 4y ago
  The central derivative is equal to Im(f(x + j h) / h) where j is the split-complex number which squares to 1, "Im" is the imaginary-part function, and h is a small real number. When j is replaced with the dual number epsilon, this instead equals the actual derivative, as is shown in the article. You can also replace j with the complex number i, producing the complex-step method. Central difference and complex-step are both bad approximations to the actual derivative. Of these, the complex-step method is the most ridiculous because it requires extending a function to act on complex numbers, while not extending it to the closely-related dual numbers.
  - vladf 4y ago
    "Ridiculous" is not a word I'd use... sometimes symbolic derivatives are intractable, in which case complex step derivatives provide high-precision numerical derivatives which are otherwise unattainable. https://vladfeinberg.com/2020/07/05/discrete-residues.html https://vladfeinberg.com/2020/07/05/discrete-residues.html
    - ogogmad 4y ago
      No. The dual numbers are very similar to the complex numbers and produce exact results. Ridiculous is true and fair. [edit] Also, I wouldn't count using Cauchy's Integral Formula as the complex-step method.
- data_maan 4y ago
  This! I always find it curious how computer scientists end up rediscovering things known for decades, and then get all hyped about it ...
goldenkey 4y ago
An entire subfield of analysis called Stenciling exists, for this purpose. Depending on the function or differential equation / system, different stencils are used. Stenciling doesn't just deal with the derivative but it tries to come up with approximations involving a fixed number of sample points, the stencil, for any differential operator, ie. the Laplacian, higher order derivatives, etc. > The term "stencil" was coined for such patterns to reflect the concept of laying out a stencil in the usual sense over a computational grid to reveal just the numbers needed at a particular step. [2] [0] https://en.wikipedia.org/wiki/Five-point_stencil https://en.wikipedia.org/wiki/Five-point_stencil [1] https://en.m.wikipedia.org/wiki/Finite_difference_coefficient#Arbitrary_stencil_points https://en.m.wikipedia.org/wiki/Finite_difference_coefficien... [2] https://en.wikipedia.org/wiki/Stencil_(numerical_analysis) https://en.wikipedia.org/wiki/Stencil_(numerical_analysis)
- yarg 4y ago
  Thanks for the links (and the flame war). Another thing that I'm wondering about is how to calculate the length of a function. What I'm thinking, is that you can treat a function like a piece of string, you take f(x) and generate a pair of functions, one above and one below f(x). You generate the functions at a fixed (infinitesimal) tangential distance 𝛿t and generate deltas for {x, y} mul = 𝛿t/((𝛿x)^2 + (𝛿y)^2)^.5 Δx = 𝛿y * mul Δy = 𝛿x * mul u(x - Δx) = f(x) + Δy l(x + Δx) = f(x) - Δy The idea being that the function length is the area ∫(u(x) - l(x)) divided by the thickness 2 * 𝛿t. I have no idea how wrong this is, but if you can point me toward any resources that'll help when I (inevitably) get stuck, it would be very much appreciated.
  - goldenkey 4y ago
    Not exactly sure what you are getting at but it does remind me of the Cauchy's Integral Formula which shows that differentiation is equivalent to integration. https://en.wikipedia.org/wiki/Cauchy%27s_integral_formula https://en.wikipedia.org/wiki/Cauchy%27s_integral_formula
    - yarg 4y ago

by yarg 4y ago

ngcc_hk 4y ago

Is h use instead of 𝛿 or Δ, because it is applicable to both partial and normal differential. 𝛿 Is always a partial symbol to me. d in dx whilst is not the same as h, as it stands its own way now as “operator”.

hither_shores 4y ago
No, the h here is a number, not an operator.
yarg 4y ago
I just find that it buries the lede.

joe__f 4y ago

If you're interested in how to best calculate derivatives numerically using a finite difference rather than an infinitesimal, try having a look at the Wikipedia page on numerical differentiation. The punchline is that you can control the error in f'(x) much better if you use many displacements, ie. f'(x) = a1 f(x + b1 h) + a2 f(x + b2 h) + a3 f(x + b3 h)..., where the bi are chosen and the ai depend on the bi and h.

ogogmad 4y ago
Such methods are doomed. Consider the function f(x) = a sin(x / a) for very small a. Finite differencing methods would approximate f'(0) as 0 as f(x) is very close to 0, but the actual value of f'(0) is 1. Differentiation is a discontinuous operator, and cannot be accurately estimated using numerical methods. Only exact methods, which include the dual numbers, can produce accurate results.
- goldenkey 4y ago
  This is not true. Doomed is sensational. With proper perturbation analysis, one can choose an appropriate stencil and formula that has desired error bounds. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72.3529&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72... https://www.google.com/search?q=finite+difference+stencil+derivative+sensitive+functions https://www.google.com/search?q=finite+difference+stencil+de...
  - ogogmad 4y ago
    Won't work on my example. Given any differentiable function f:R -> R, you can add a very small function to it that dramatically changes its derivative. But maybe this is too pathological to bother about. Over the complex numbers, this phenomenon doesn't happen. This can be shown, for instance, using the maximum modulus principle. You can therefore use Cauchy's Integral Formula to accurately estimate derivatives. But why not then just use the dual numbers?
    - 4y ago

godelski 4y ago

You really just derived the central derivative. Yes, this derivative is more accurate. The issue is that you can't use it on the edges. For the boundaries you're stuck with lower precision. Here's a resource you might be interested in http://www2.math.umd.edu/~dlevy/classes/amsc466/lecture-notes/differentiation-chap.pdf http://www2.math.umd.edu/~dlevy/classes/amsc466/lecture-note...

ogogmad 4y ago
The central derivative is equal to Im(f(x + j h) / h) where j is the split-complex number which squares to 1, "Im" is the imaginary-part function, and h is a small real number. When j is replaced with the dual number epsilon, this instead equals the actual derivative, as is shown in the article. You can also replace j with the complex number i, producing the complex-step method. Central difference and complex-step are both bad approximations to the actual derivative. Of these, the complex-step method is the most ridiculous because it requires extending a function to act on complex numbers, while not extending it to the closely-related dual numbers.
- vladf 4y ago
  "Ridiculous" is not a word I'd use... sometimes symbolic derivatives are intractable, in which case complex step derivatives provide high-precision numerical derivatives which are otherwise unattainable. https://vladfeinberg.com/2020/07/05/discrete-residues.html https://vladfeinberg.com/2020/07/05/discrete-residues.html
  - ogogmad 4y ago
    No. The dual numbers are very similar to the complex numbers and produce exact results. Ridiculous is true and fair. [edit] Also, I wouldn't count using Cauchy's Integral Formula as the complex-step method.
data_maan 4y ago
This! I always find it curious how computer scientists end up rediscovering things known for decades, and then get all hyped about it ...

goldenkey 4y ago

An entire subfield of analysis called Stenciling exists, for this purpose. Depending on the function or differential equation / system, different stencils are used. Stenciling doesn't just deal with the derivative but it tries to come up with approximations involving a fixed number of sample points, the stencil, for any differential operator, ie. the Laplacian, higher order derivatives, etc. > The term "stencil" was coined for such patterns to reflect the concept of laying out a stencil in the usual sense over a computational grid to reveal just the numbers needed at a particular step. [2] [0] https://en.wikipedia.org/wiki/Five-point_stencil https://en.wikipedia.org/wiki/Five-point_stencil [1] https://en.m.wikipedia.org/wiki/Finite_difference_coefficient#Arbitrary_stencil_points https://en.m.wikipedia.org/wiki/Finite_difference_coefficien... [2] https://en.wikipedia.org/wiki/Stencil_(numerical_analysis) https://en.wikipedia.org/wiki/Stencil_(numerical_analysis)

yarg 4y ago
Thanks for the links (and the flame war). Another thing that I'm wondering about is how to calculate the length of a function. What I'm thinking, is that you can treat a function like a piece of string, you take f(x) and generate a pair of functions, one above and one below f(x). You generate the functions at a fixed (infinitesimal) tangential distance 𝛿t and generate deltas for {x, y} mul = 𝛿t/((𝛿x)^2 + (𝛿y)^2)^.5 Δx = 𝛿y * mul Δy = 𝛿x * mul u(x - Δx) = f(x) + Δy l(x + Δx) = f(x) - Δy The idea being that the function length is the area ∫(u(x) - l(x)) divided by the thickness 2 * 𝛿t. I have no idea how wrong this is, but if you can point me toward any resources that'll help when I (inevitably) get stuck, it would be very much appreciated.
- goldenkey 4y ago
  Not exactly sure what you are getting at but it does remind me of the Cauchy's Integral Formula which shows that differentiation is equivalent to integration. https://en.wikipedia.org/wiki/Cauchy%27s_integral_formula https://en.wikipedia.org/wiki/Cauchy%27s_integral_formula
  - yarg 4y ago