2 ms·
Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words. If y
by ralusek 2y ago
Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words.
If you want to just see the tokens for yourself, though, just enter some text here:
https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer