They won't move past tokenization, but you can take it down to the byte level and make it arbitrarily flexible and adaptive: https://ai.meta.com/research/publi

They won't move past tokenization, but you can take it down to the byte level and make it arbitrarily flexible and adaptive: https://ai.meta.com/research/publications/byte-latent-transformer-patches-scale-better-than-tokens/ https://ai.meta.com/research/publications/byte-latent-transf...