My point was that it doesn't matter so much what the tokens look like as long as the model has seen them often enough during training. English-in-base64 superfi

My point was that it doesn't matter so much what the tokens look like as long as the model has seen them often enough during training. English-in-base64 superficially appears completely different from English, but they're actually in 1-to-1 correspondence and follow the same rules. The same holds for misspelled or badly-OCR'd text; the tokenizer actually has tokens for them and the model has seen enough of them to handle heavily-distorted text. But represent the same text using different but theoretically valid tokens that would not normally be produced by the tokenizer and all bets are off.