3 ms·

is is possible for your tokenizer to give different tokenization ever then openai tokenizer? i am asking because there are multiple ways to tokenize the same st

is is possible for your tokenizer to give different tokenization ever then openai tokenizer? i am asking because there are multiple ways to tokenize the same string?? sry if i am mistaken

matthewolfe 1y ago
Should be the same. Both use Byte-Pair Encoding (BPE) as underlying algo.