4 ms·
I've always thought that this was a fun way to look at it: Suppose you solve the differential equation, x'(t) = -f'(x(t)) (1) Then,
by chestervonwinch 10y ago
I've always thought that this was a fun way to look at it:
Suppose you solve the differential equation,
x'(t) = -f'(x(t)) (1)
Then,
d/dt f(x(t)) = f'(x) x'(t) = - [f'(x)]^2 <= 0
In other words, if `x(t)` follows a path that solves (1), then `x(t)` follows a path that decreases the value of `f(x)`.
The gradient descent algorithm is a numerical approximation to solving (1) using the forward Euler method:
(x(t_(n+1)) - x(t_n)) / dt = -f'(x(t_n))
- RBerenguel 10y agoNice and clear way to think about it (and touching closer to the kind of numerical analysis I drilled most, too). I wonder, if having a good polynomial description of f then we can "just" use a Taylor integration method, approximating the inverses needed along the way with some high order approximation.
- dxbydt 10y agoPerhaps an example might help. Say x(t) = exp(-t) f(x(t)) = (x^2)/2 Then it satisfies conditions imposed by parent, namely x' = -x = -f'
- chestervonwinch 10y agoNice. I think you mean f(x) = 0.5*x^2 though.
- Chris2048 10y agoIs it not: d/dt f(x(t)) = f'(x(t)) x'(t) = -(-f'(x(t))) x'(t) = -(x'(t)) x'(t) = -[x'(t)]^2 = -[f'(x(t))]^2 ?