2 ms·
I appreciate the feedback, I did notice that as well and I had this thought perhaps this is not worth fixing since I have a link to tiktokenizer. I decided to r
by ynarwal__ 6mo ago
I appreciate the feedback, I did notice that as well and I had this thought perhaps this is not worth fixing since I have a link to tiktokenizer. I decided to remove it and just added a more prominent link to tiktokenizer.
- thesz 6mo agoBPE that is used in tokenization is very simple: https://en.wikipedia.org/wiki/Byte-pair_encoding https://en.wikipedia.org/wiki/Byte-pair_encoding