4 ms·
>After spending a lot of time with language models, I have come to the conclusion that tokenization in general is insane and it is a miracle that language model
by msp26 3y ago
>After spending a lot of time with language models, I have come to the conclusion that tokenization in general is insane and it is a miracle that language models learn anything at all.
Truer words have never been spoken