4 ms·
You're saying that LBFGS is fundamental to the success of KANs? Why is this so?
by abetusk 2y ago
You're saying that LBFGS is fundamental to the success of KANs? Why is this so?
- buildbot 2y agoAnd if it is, good luck scaling LBFGS to anything useable, like vgg-16 scale…let alone a 7B param LLM. Back in grad school I tried to use LBFGS to optimize a small lenet network. I want to say it used over 128GB before OOM.
- thesz 2y agoThis is why I mentioned batch gradient line search. You can combine it with conjugate gradient. And small LeNet (I think it is first convolutional network that obtained good score on MNIST) is orders of magnitude bigger than KAN's in the original paper. And it will be, if we believe scaling claims from the KAN paper.
- thesz 2y agoI can only state that KAN paper used LBFGS and that LBFGS is remarkably different from SGD. One of the differences is a dynamic learning rate guided by approximation of the local curvature.
- deleted 2y ago[deleted]
- adw 2y agoWill deal better with less well-conditioned parameters, maybe? That's a bit of a wild-ass guess, but it's not an entirely uninformed one.