3 ms·
https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer Test with: 6f6b25515e18b4c41927e2dce3ffc589 You'll see that it spits out single character
by decremental 4y ago
https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer
Test with:
6f6b25515e18b4c41927e2dce3ffc589
You'll see that it spits out single character tokens.