LLMs started out as letter-by-letter. The issue is that this makes you context window much smaller than compressing words into tokens. Tokens make both training

LLMs started out as letter-by-letter. The issue is that this makes you context window much smaller than compressing words into tokens. Tokens make both training and inference much more effective and efficient, at the cost of trading away some capabilities. But I believe there is a lot of room in exploring better tokenization. An obvious example some models are playing with is how to tokenize numbers. In GPT3 10000 is a single token, while 10001 is two tokens (100-01), which obviously makes it harder for the model to learn math. Some models improved on this by special-casing numbers in the tokenizer to turn each digit into one token. I think GPT4 settled one one token for ever three digits. For pronunciation, maybe throwing some IPA dictionaries into the training set might already do the trick. The model can already infer facts across languages, so if we can somehow teach it a phonetic model it might be able to use that.