2 ms·
I predict eventual convergence between UTF-whatever and most popular tokenizer for whatever LLM escapes to become world-ruling AGI. I mean, if someone's seriou
by TeMPOraL 14d ago
I predict eventual convergence between UTF-whatever and most popular tokenizer for whatever LLM escapes to become world-ruling AGI.
I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.