3 ms·Do they all use the same tokenizer? (I mean, Mistral vs GPT)by raverbashing 3y agoDo they all use the same tokenizer? (I mean, Mistral vs GPT)superkuh 3y agoNo. Mistral uses sentencepiece and the GPT use tiktoken.