> but AFAIU LLMs will likely never rationalize or be able to "add numbers" in the normal sense; they can count only in as much they know that 1,2,3,4 is more p

> but AFAIU LLMs will likely never rationalize or be able to "add numbers" in the normal sense; they can count only in as much they know that 1,2,3,4 is more probably coming after "count to four" than 10,2,4. You might be interested in this experiment - https://thegradient.pub/othello/ https://thegradient.pub/othello/ The question is does a model trained on move sequences in a game of othello just learn "play X(t) after seeing the sequence X(t-3), X(t-2), X(t-1)" or does it build a representation of the board and use that to choose the next move to play.