Math is helpful for neural networks, but it seems our mathematical sophistication is immature. For example, why can't a MLP (the most basic neural network from

Math is helpful for neural networks, but it seems our mathematical sophistication is immature. For example, why can't a MLP (the most basic neural network from the 1960s) do the same things as a Transformer (the successful new architecture from 2017)? We've cited some possible reasons in retrospect, but we're not skilled enough to predict the performance of a neural network without trying it.