3 ms·
The page does very poor job tokenizing phrase "Noinceolik fiyulnabmed fyvaproldge" into "Noinceolik fiyulnabm ed fyvaproldge", factoring only "ed" suffix. As if
by thesz 6mo ago
The page does very poor job tokenizing phrase "Noinceolik fiyulnabmed fyvaproldge" into "Noinceolik fiyulnabm ed fyvaproldge", factoring only "ed" suffix. As if made up words such as "noinceolik" are so common they are part of 100K token vocabulary.
The actual application of GPT-5 tokenizer at [1] to my made up phrase results in 14 tokens, only two of them are four characters long and there are tokens containing spaces.
[1] https://gpt-tokenizer.dev/ https://gpt-tokenizer.dev/
I will read along, though.
- ynarwal__ 6mo agoI appreciate the feedback, I did notice that as well and I had this thought perhaps this is not worth fixing since I have a link to tiktokenizer. I decided to remove it and just added a more prominent link to tiktokenizer.
- thesz 6mo agoBPE that is used in tokenization is very simple: https://en.wikipedia.org/wiki/Byte-pair_encoding https://en.wikipedia.org/wiki/Byte-pair_encoding