Damn, how many problems with LLMs relate to the encoding of the token? Surely every symbolic manipulation task is getting thrown off by this. Memorizing the mul

Damn, how many problems with LLMs relate to the encoding of the token? Surely every symbolic manipulation task is getting thrown off by this. Memorizing the multiplication table of two three digit numbers is no easy task at all. That explains why the interpreter hack works so well. The python interpreter sees things digit by digit, but the LLM does it token by token. I've asked it so many times to count the number of words or letters and it was incredibly bad at it. Since it is capable of splitting large tokens into smaller tokens, the solution to this problem is to create additional training samples that perform "big token" to "small token" conversion and back, so that the model will learn to dynamically provide the most suitable encoding to itself.