[shameless plug] You can effectively train a token-free model if you somehow know how to segment the text into meaningful chunks. https://arxiv.org/abs/2211.12

[shameless plug] You can effectively train a token-free model if you somehow know how to segment the text into meaningful chunks. https://arxiv.org/abs/2211.12677 https://arxiv.org/abs/2211.12677