3 ms·
This is sort of my view as well, most of the hype and the criticisms of KANs seem to be fairly unfounded. I do think they have a lot of potential, but what ha
by Lerc 2y ago
This is sort of my view as well, most of the hype and the criticisms of KANs seem to be fairly unfounded.
I do think they have a lot of potential, but what has been published so far does not represent a panacea. Perhaps they will have an impact like transformers, perhaps they will only serve in a little niche. You can't really tell immediately how refinements will alter the usability.
Finding out what those refinements are and how they change things is what research is all about. I have been quite enjoying following https://github.com/mintisan/awesome-kan https://github.com/mintisan/awesome-kan progress and seeing the variety of things being tried. I have a few ideas of my own I might try at sometime.
Between KANs and fixed activation function networks there is an entire continuum of activation function tuning available for research.
Buckets of simple parameter activation functions something like xsigmoid(mx) ( ReLU when m is large, GeLU at m=1.7, SiLU at m=1). This adds a small number of parameters for presumably some game
Single activation functions as above per neuron.
Multi parameterizable activation functions, in batches, or per neuron.
Many parameter function approximators, in batches, or per neuron.
Full KANs without weights.
I can see some significant acclaim being awarded to the person who can calculate a unified formula for determining where additional parameters should go for the largest impact.
- sigmoid10 2y agoMy big issue with KANs is that MLPs can trivially be made functionally identical to them up to an arbitrarily small error. Just take a group of neurons/layers and thanks to the UAT you can get them to model any reasonably well behaved activation function. Now redefine that group as a KAN node and you have something that works exactly the same way. In that sense it is actually strange that KANs with the same number of parameters don't outperform MLPs. This could be seen as a hint that activation functions are not really what matters in the end. This is also something that the biology of real neural networks seems to suggest from experiments with rats. Although there is far too little conclusive research in that area. I'm about 50:50 on whether this question will be solved by biologists or computer scientists.
- Lerc 2y agoWell obviously a function approximator that uses a sum of functions can be produced by an assembly of function approximators. I don't think anyone is going to argue otherwise. The measure of parameter efficiency is the issue at hand. Given the relative newness of this approach even hitting the bar of "sometimes better, under specific circumstances" is impressive. I am less likely to believe in biological research revealing new techniques. I feel like the inverse is more likely, that discovering new techniques will allow biologists to identify mechanisms where it was already in use.
- sigmoid10 2y ago"Relative newess" is a very shaky foundation for any argument. After all, in practice you could use all of the gradient descent methods we have today, you only need to hardcode a few derivatives. On the other hand biology suggests that activation functions are kind of irrelevant for complex networks beyond a certain size. So even in the case KANs slightly better, MLPs would win due to their simplicity in the implementation.
- vinnyvichy 2y agoHaha you are wrong, I have a PhD, and a number of MSc, from the U.S.., institutions that have a number of Nobel laureates.. (but I don't have student debt!) Otherwise -- in a different world -- you and I could be brothers.. (If you judge me to be delusional.. however, erm.. frater quoque! The delusionality helps, sadly)
- sigmoid10 2y agoWrong comment chain?
- gradschoolfail 2y agoI hear you, brother ;) Will re-investigate your research pointers when i have time.