3 ms·
Sometimes yes, but in this case you can verify that in the phrase "gpt-3.5-turbo", the only doubled up characters are "gpt-3.5-turbo", so the number can still b
by minimaxir 3y ago
Sometimes yes, but in this case you can verify that in the phrase "gpt-3.5-turbo", the only doubled up characters are "gpt-3.5-turbo", so the number can still be wrong by swapping a single token.
https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer