4 ms·
I think what OP means is that the tokenization is throwing away encoded information. That is 11 should be two tokens representing 1 and 10. That way the relati
by gerad 4y ago
I think what OP means is that the tokenization is throwing away encoded information.
That is 11 should be two tokens representing 1 and 10. That way the relationship can be more easily learned.
Otherwise it needs to memorize the relationship between every pair of numbers. Which is exponentially more difficult.
- nl 4y agoYes this is a fair point, and if that is what they meant I could agree it is worth trying. As pointed out down-thread the Llama tokenizer is supposed to spilt numbers into their digits.