Google Research has a character-based transformer that learns to tokenize text rather than relying on hand coded tokenizers. It demonstrates superior performanc

Google Research has a character-based transformer that learns to tokenize text rather than relying on hand coded tokenizers. It demonstrates superior performance on a variety of LLM tasks. If you have the money, you can apply the transformer architecture to many different tasks and people are experimenting all the time. I think one of the big challenges is always to come up with methods for training such enormous models pragmatically without cost exploding. [1] https://huggingface.co/docs/transformers/model_doc/canine https://huggingface.co/docs/transformers/model_doc/canine