3 ms·
Two of the most popular libraries for token creation (tokenization) are in fact in rust, with an interface in python: https://github.com/huggingface/tokenizers
by m3at 2y ago
Two of the most popular libraries for token creation (tokenization) are in fact in rust, with an interface in python:
https://github.com/huggingface/tokenizers https://github.com/huggingface/tokenizers
https://github.com/openai/tiktoken https://github.com/openai/tiktoken