3 ms·

My big issue with KANs is that MLPs can trivially be made functionally identical to them up to an arbitrarily small error. Just take a group of neurons/layers a

My big issue with KANs is that MLPs can trivially be made functionally identical to them up to an arbitrarily small error. Just take a group of neurons/layers and thanks to the UAT you can get them to model any reasonably well behaved activation function. Now redefine that group as a KAN node and you have something that works exactly the same way. In that sense it is actually strange that KANs with the same number of parameters don't outperform MLPs. This could be seen as a hint that activation functions are not really what matters in the end. This is also something that the biology of real neural networks seems to suggest from experiments with rats. Although there is far too little conclusive research in that area. I'm about 50:50 on whether this question will be solved by biologists or computer scientists.

Lerc 2y ago
Well obviously a function approximator that uses a sum of functions can be produced by an assembly of function approximators. I don't think anyone is going to argue otherwise. The measure of parameter efficiency is the issue at hand. Given the relative newness of this approach even hitting the bar of "sometimes better, under specific circumstances" is impressive. I am less likely to believe in biological research revealing new techniques. I feel like the inverse is more likely, that discovering new techniques will allow biologists to identify mechanisms where it was already in use.
- sigmoid10 2y ago
  "Relative newess" is a very shaky foundation for any argument. After all, in practice you could use all of the gradient descent methods we have today, you only need to hardcode a few derivatives. On the other hand biology suggests that activation functions are kind of irrelevant for complex networks beyond a certain size. So even in the case KANs slightly better, MLPs would win due to their simplicity in the implementation.
vinnyvichy 2y ago
Haha you are wrong, I have a PhD, and a number of MSc, from the U.S.., institutions that have a number of Nobel laureates.. (but I don't have student debt!) Otherwise -- in a different world -- you and I could be brothers.. (If you judge me to be delusional.. however, erm.. frater quoque! The delusionality helps, sadly)
- sigmoid10 2y ago
  Wrong comment chain?
  - gradschoolfail 2y ago
    I hear you, brother ;) Will re-investigate your research pointers when i have time.

Lerc 2y ago

Well obviously a function approximator that uses a sum of functions can be produced by an assembly of function approximators. I don't think anyone is going to argue otherwise. The measure of parameter efficiency is the issue at hand. Given the relative newness of this approach even hitting the bar of "sometimes better, under specific circumstances" is impressive. I am less likely to believe in biological research revealing new techniques. I feel like the inverse is more likely, that discovering new techniques will allow biologists to identify mechanisms where it was already in use.

sigmoid10 2y ago
"Relative newess" is a very shaky foundation for any argument. After all, in practice you could use all of the gradient descent methods we have today, you only need to hardcode a few derivatives. On the other hand biology suggests that activation functions are kind of irrelevant for complex networks beyond a certain size. So even in the case KANs slightly better, MLPs would win due to their simplicity in the implementation.

vinnyvichy 2y ago

Haha you are wrong, I have a PhD, and a number of MSc, from the U.S.., institutions that have a number of Nobel laureates.. (but I don't have student debt!) Otherwise -- in a different world -- you and I could be brothers.. (If you judge me to be delusional.. however, erm.. frater quoque! The delusionality helps, sadly)

sigmoid10 2y ago
Wrong comment chain?
- gradschoolfail 2y ago
  I hear you, brother ;) Will re-investigate your research pointers when i have time.