I can only state that KAN paper used LBFGS and that LBFGS is remarkably different from SGD. One of the differences is a dynamic learning rate guided by approxi

I can only state that KAN paper used LBFGS and that LBFGS is remarkably different from SGD. One of the differences is a dynamic learning rate guided by approximation of the local curvature.