4 ms·
Correct me if I'm wrong—but we don't know if GPT-4 uses the same tokenizer as GPT-3.5, right?
by npsomaratna 3y ago
Correct me if I'm wrong—but we don't know if GPT-4 uses the same tokenizer as GPT-3.5, right?
- minimaxir 3y agoOpenAI's web tokenizer demo confirms it: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer