2 ms·
The OpenAI tokenizer tool agrees with you, unless it's at the start of the document. e.g. "An eagle flew into its nest ..."
by nerdponx 4y ago
The OpenAI tokenizer tool agrees with you, unless it's at the start of the document. e.g. "An eagle flew into its nest ..."
- vore 4y agoYup, if placed at the start of the document, "An eagle [...]" would be [2025, 31176] and "Anoxic [...]" would be [2025, 18047].