3 ms·
What's your basis for claiming that Tinygrad can't compute 2nd order partial derivatives (i.e. Hessians) needed for LBFGS? Tinygrad like PyTorch uses automatic
by osipov 2y ago
What's your basis for claiming that Tinygrad can't compute 2nd order partial derivatives (i.e. Hessians) needed for LBFGS? Tinygrad like PyTorch uses automatic differentiation which has no problem supporting nth order derivatives.
- fjkdlsjflkds 2y agoOP does not (seemingly) claim that tinygrad can't compute hessians, only that a first-order optimization method was the only thing tried. Also, as a quasi-newton method, L-BFGS does not require explicit (pre-)computation of the hessian (it implicitly iteratively estimates its inverse in an online manner).
- thesz 2y agoAs someone with highly unpronoceable nickname said, my only complaint is that only first order methods are used. Second order methods are fun, actually. I like them. ;)