4 ms·

There seems to be some sort of flaw or misalignment in the hypothesis. By my reading, it seems this trio is saying LLMs do arithmetic by predicting the digits,

There seems to be some sort of flaw or misalignment in the hypothesis. By my reading, it seems this trio is saying LLMs do arithmetic by predicting the digits, and at intermediate stages there are actually partial predictions, and so there isn't a huge jump, which would then be classifiable as emergent. But I'm thinking that in order for an LLM to generally predict arithmetic results in the way they are designed, there would be an infinite memory requirement as it'd have to "store" an infinite set of completions. So the key may be to find sets of operands that GPT4/Claude Opus are unable to solve, even with hints similar to what a teacher may give a student learning the topic. Until such a set is found, I'd say this capability continues to meet the "emergent" definition, as the only other explanation - that I can think of - is that the models "discovered" how to do arithmetic in a general way.

famouswaffles 3y ago
>that I can think of - is that the models "discovered" how to do arithmetic in a general way. I mean..yeah. Maybe i've simply misunderstood but It seems like you're presenting this as an outlandish idea but something like this is what Neural Networks regularly end up doing especially in the limit (as loss trends down to zero). https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura... https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec... Nobody is going to find any such set because time and time again, neural networks show us complete memorization is actually more difficult than just learning the distribution (even if imperfectly).