3 ms·
Didn't tokenization already have one bitter lesson: that it's better to let simple statistics guide the splitting, rather than expert morphology models? Would t
by pona-a 1y ago
Didn't tokenization already have one bitter lesson: that it's better to let simple statistics guide the splitting, rather than expert morphology models? Would this technically be a more bitter lesson?
- matusp 1y agoAgreed completely. There is a ton of research into how to represent text, and these simple tokenizers are consistently performing on SOTA levels. The bitter lesson is that you should not worry about it that much.
- kingstnap 1y agoSimple statistics aren't some be all. There was a huge improvement in Python coding by fixing the tokenization of indents in Python code. Specifically they made tokens for 4,8,12,16 or something spaces.