nispin

4 ms·

1. 1.▲
   28 Times faster than Google's LLM tokenizer(o200k-tokenizer-70fe25.gitlab.io)

   2 points by nispin 7mo ago|0 comments
2. 2.▲
   Tokenizer That Outperform Tiktoken with O200k_base(o200k-tokenizer-70fe25.gitlab.io)

   1 points by nispin 7mo ago|0 comments