2 ms·
Something in between. example: https://gpt-tokenizer.dev/ https://gpt-tokenizer.dev/
by neoromantique 2y ago
Something in between.
example: https://gpt-tokenizer.dev/ https://gpt-tokenizer.dev/
- UncleOxidant 2y agoLooks like for the most part it's words, but some longer words get broken at prefix/suffix boundaries.