3 ms·
It doesn't actually have a token for Estonian dog, the word " koer" is tokenized as two tokens, " ko" and "er". In my language it's three tokens. In Russian it'
by a2128 3y ago
It doesn't actually have a token for Estonian dog, the word " koer" is tokenized as two tokens, " ko" and "er". In my language it's three tokens. In Russian it's seven(!). So any relations to dog and puppy in other languages would have to happen deeper within the model. Just as the tokenizer can't capture that a base64-encoded "dog" is related to a base64-encoded "puppy"