To me the odd takeaway here is that the model eventually converges to a close approximation but never really lands at the core of the problem. Loss never reache

To me the odd takeaway here is that the model eventually converges to a close approximation but never really lands at the core of the problem. Loss never reaches zero even for the memorized set, and the "grokked" loss is usually higher. Wouldn't you expect something totally bounded like "addition mod 113" to be trainable to the point of zero loss? I think the field is looking for that final phase change that takes us from statistical methods to actual understanding.