> And here's a neural network/multi-layer perceptron with the same number of layers and nodes: One big difference to note is that there are far fewer connection

> And here's a neural network/multi-layer perceptron with the same number of layers and nodes: One big difference to note is that there are far fewer connections between nodes in KANs compared to neural networks/MLPs. I think it’s probably worth clarifying a little here that a Bspline is essentially a little MLP, where, at least for uniform Bsplines, the depth is equal to the polynomial degree of the spline. (That’s also the width of the first layer.) So those two network diagrams are only superficially similar, but the KAN is actually a much bigger network if degree > 1 for the splines. I wonder if that contributed to the difficulty of training it. It is possible some of the “code smell” you noticed and got rid of is relatively important for achieving good results. I’d guess the processes for normalizing inputs and layers of a KAN need to be a bit different than for standard nets.