4 ms·
It mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilin
by espadrine 3y ago
It mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilingually): “説明書” (instruction manual) has its own token, and so does “Nixon”, “آباد” (a city suffix, I believe), and the HTML sequence "\"><!--".
- lalaithion 3y agoI understand the theory, I was looking for an example of the same text tokenized with the two different vocabularies.
- espadrine 3y agoDo you have an example text in mind? You can use this playground to test it out: https://huggingface.co/spaces/Xenova/the-tokenizer-playground https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...
- moffkalast 3y agoInteresting, it's actually worse than GPT-4s 100k tokenizer by quite a bit despite being over twice the size and only marginally better than LLama's 30k. At least for some random articles in English latin script that I tried anyway, but Llama and Gemma are English-only models so no point in testing anything else. Doesn't seem like a well made tokenizer at first glance or it's heavily biased towards languages the model can't even generate coherently, lol. If they really wanted it to be SOTA at something they could've at least made it the first open source truly multilingual model, but that's apparently more effort than the lame skin colour oriented virtue signalling Google wants to do.