A syllabary or a phonetic encoding like IPA would patch the rhyme problem, but it would sabotage you in still other ways. People expect a language model to be a

A syllabary or a phonetic encoding like IPA would patch the rhyme problem, but it would sabotage you in still other ways. People expect a language model to be able to solve anagrams or reverse strings, for example. And there are a lot of things in written language which are connected to the exact spelling but not necessarily the phonetics; there is no list of what those things are or what not learning them would do to a model, and you would have to discover (or more likely, not discover, in the way that people keep not discovering the BPE sabotage) the drawbacks the hard way. So you have a Bitter Lesson tradeoff here: yeah, you can build in that concept instead of learning from raw Unicode/bytes, and it will work initially better, but you are handicapping yourself in the long run. So, I always say go for raw Unicode for the foreseeable future.