4 ms·

Yes sometimes it’s hard to measure a derivative. Eg when doing hyperparameter tuning in ML, you can read out a metric at a given choice of parameters, but it’s

by efavdb 3y ago

Yes sometimes it’s hard to measure a derivative. Eg when doing hyperparameter tuning in ML, you can read out a metric at a given choice of parameters, but it’s generally not easy to get a gradient. Shameless plug: I happen to have recently written a package for the opposite limit! It finds roots when you can only measure the derivative. https://github.com/EFavDB/inchwormrf https://github.com/EFavDB/inchwormrf

quickthrower2 3y ago
I assume backprop is still most efficient for the neural net itself, rather than one of these algorithms? I am doing a course and I am aware that there are other algorithms than gradient descent but haven’t seen the details yet on that. I know things like cross entropy loss are designed to be easy to differentiate and probably more efficient to do that than something that approximates it.
- efavdb 3y ago
  Yeah, backprop gives efficient gradient calcs to train at a given net architecture. To find the best architecture though people try various other methods, eg random search or Gaussian processes, which don’t evaluate derivatives wrt architecture Params.
imurray 3y ago
You might be interested in David MacKay's old conjugate gradient implementation in C that also only uses derivatives: https://www.inference.org.uk/mackay/c/macopt.html https://www.inference.org.uk/mackay/c/macopt.html (It was written in another time, don't expect it to be nice. My 20-year-old Octave/Matlab wrappers on that page have almost certainly bit-rotted, don't expect them to work.)
- efavdb 3y ago
  Thank you! I’ll definitely check this out.
enriquto 3y ago
> It finds roots when you can only measure the derivative. How is that even possible? The function f(x)=x-7 has a single root at x=7. Its derivative is f'(x)=1. How do you recover 7 from 1 ?
- CorrectHorseBat 3y ago
  You need one initial value to start from
  - deleted 3y ago
    [deleted]
- deleted 3y ago
  [deleted]

by efavdb 3y ago

quickthrower2 3y ago

I assume backprop is still most efficient for the neural net itself, rather than one of these algorithms? I am doing a course and I am aware that there are other algorithms than gradient descent but haven’t seen the details yet on that. I know things like cross entropy loss are designed to be easy to differentiate and probably more efficient to do that than something that approximates it.

efavdb 3y ago
Yeah, backprop gives efficient gradient calcs to train at a given net architecture. To find the best architecture though people try various other methods, eg random search or Gaussian processes, which don’t evaluate derivatives wrt architecture Params.

imurray 3y ago

You might be interested in David MacKay's old conjugate gradient implementation in C that also only uses derivatives: https://www.inference.org.uk/mackay/c/macopt.html https://www.inference.org.uk/mackay/c/macopt.html (It was written in another time, don't expect it to be nice. My 20-year-old Octave/Matlab wrappers on that page have almost certainly bit-rotted, don't expect them to work.)

efavdb 3y ago
Thank you! I’ll definitely check this out.

enriquto 3y ago

> It finds roots when you can only measure the derivative. How is that even possible? The function f(x)=x-7 has a single root at x=7. Its derivative is f'(x)=1. How do you recover 7 from 1 ?

CorrectHorseBat 3y ago
You need one initial value to start from
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]