I hate this as being too wooish. There first example, arithmetic, we know actually benefits from over fitting, but even then it isn't great. We now know that

I hate this as being too wooish. There first example, arithmetic, we know actually benefits from over fitting, but even then it isn't great. We now know that over Parameterization and double decent is less lossy compression. We know that feed forward networks are TC^0 and we know that CoT etc... are limited to polynomial reductions. The fact that we can't predict exactly what is learned in a specific instance is purely a function of computational complexity theory. Attention we know is equivalent to exhaustive search. Yes interpretability is hard and math explains that. We know that the curse of dimensionality means LLMs are using extrapolation as an example. https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 This isn't meant to diminish the very impressive advances that have been made. But it is not a complete god box, we can understand the underlying properties that lead to models being found. But IMHO pretending that it is something that is magical seriously diminishes the hard work that was required to get where we are today.