Tokenization is my pet peeve with LLMs as I suspect it makes it much harder for them to learn coding, but it's difficult to do anything about because doing bett

Tokenization is my pet peeve with LLMs as I suspect it makes it much harder for them to learn coding, but it's difficult to do anything about because doing better requires you to train a model from scratch. In turn that's so expensive that it hardly happens and the people who do it tend to focus on higher priorities, leaving tokenization alone. I guess at some point we'll see some progress here. Things I'd like to see tweaked: • All code being reformatted as part of the cleaning process to use a canonical style designed for LLMs. For example, reindenting to use tabs instead of spaces, unwrapping lines. Way too many tokens in the GPT4 token list have random bits of whitespace prepended to them meaning they can only be used in certain nesting levels, which must make it harder for the model to learn. • Forcing language keywords to be a single token. GPT4 has tokens like " public", " public", " Public" etc. • Special tokens being used to indicate start/end of identifiers, then decompounding and decasing words during tokenization, i.e. "void flipTheSwitch(" -> "void <id-start> flip the switch <id-end> (". This avoids tokens that are compound words which forces the model to relearn their meaning from scratch. The renderer would then need to do the reverse during inferencing. • Decompounding dereference operators like "." or "->". The GPT4 token list is full of tokens like ".square", ".attribute" etc. • Clearing away all the tokens that are just punctuation compounds like "))));\n" The above changes would slow down inference by requiring more tokens to be generated, and would require renderers to be taught how to reformat code on the fly to a form better suited for humans (e.g. shorter lines), but intuitively should make it far easier for smaller models to learn coding and that in turn might make things faster again. More speculative ideas include: • Reserve some special tokens for bracketing IDE-style features like inlining invisible JavaDocs, so inference can be rolled forward and backwards when the logprob distribution is too flat. • Ditto for inferred types if the cleaning process can be tightly integrated with an IDE or language server (e.g. "val a = b.c.d()" -> "val a /String/ = /Foo/b./Bar/c./String/d()") There's just so much that can be done here, the amount of low hanging fruit is enormous. But right now I suspect the industry is stuck in a kind of local minima where if you're willing and able to train an LLM from scratch you also aren't terribly price sensitive and would rather bitter-lesson your way to better results, or focus on core algorithmic improvements that affect more than just coding, rather than spend time improving tokenization schemes. I also suspect that the Python focus of the AI community means they just aren't thinking very hard about ways to help LLMs with strongly typed languages.