4 ms·
You can partially tell by the tokeniser; which gives you some hint into the training corpus mix. </div> is four Gemma4 tokens, but one Qwen3.6 token.
by dannyw 2mo ago
You can partially tell by the tokeniser; which gives you some hint into the training corpus mix.
</div> is four Gemma4 tokens, but one Qwen3.6 token.
- venusenvy47 2mo agoWhere do you find this information for each model?
- ComputerGuru 2mo agoThe tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.
- adrian_b 2mo agoWhen you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json". In that file you can see all tokens and their corresponding numeric codes.
- deleted 2mo ago[deleted]
- stymaar 2mo agoLooks like we have a /r/localllama dweller here.