I'm not sure it's true that symbolic AI is dead, but I think Gary Marcus style symbolic AI is. His "next decade in AI" paper doesn't even mention symbolic regre

I'm not sure it's true that symbolic AI is dead, but I think Gary Marcus style symbolic AI is. His "next decade in AI" paper doesn't even mention symbolic regression. For those who don't know, linear regression tries to find a linear fit against a dataset. Quadratic regression a quadratic fit. And so on. Symbolic regression tries to find a symbolic expression that gives you an accurate data fit. Symbolic regression has an extremely obvious advantage over neural networks, which is that it learns parameters and architecture simultaneously. Having the correct architecture means that the generalization power is greater and that the cost of evaluation due to redundant parameters is lower. The crippling downside is that the search space is so vast that it is only applicable to toy problems. But Gary Marcus is in favour of hybrid architectures, so what would that look like? In the SymbolNet paper, they have essentially decided to keep the overall neural network architecture, but replaced the activation functions with functions that take multiple inputs aka symbols. The network can then be pruned down into a symbolic expression. That in itself is actually a pretty damning blow to Gary Marcus, because now you have most of the benefits of symbolic AI with only a tiny vestige of investment into it. What this tells us is that fixed activation functions with a single input appear to be a no go, but nobody ever said that biological neurons implement sigmoid, relu, etc in the first place. It's possible that the spike encoding already acts as a mini symbolic regressor and gives each neuron its own activation function equivalent. The "Neuronal arithmetic" paper has shown that biological neurons can not only calculate sums (ANNs can do this), but also multiply their inputs, which is something the activation functions in artificial neural networks cannot do. LLMs use gating in their MLPs and attention to explicitly model multiplication. There is also the fact that biological neurons form loops. The memory cells in LSTMs perform a similar function, but in the brain there can be memory cells everywhere whereas in a fixed architecture like LSTMs they are only where the designer put them. It seems as if the problem with neural networks is that they're too static and inflexible and contain too much influence from human designers.