4 ms·
It looks like TikToken is written in Rust (https://github.com/openai/tiktoken/tree/main/src https://github.com/openai/tiktoken/tree/main/src), are the gains her
by notatallshaw 1y ago
It looks like TikToken is written in Rust (https://github.com/openai/tiktoken/tree/main/src https://github.com/openai/tiktoken/tree/main/src), are the gains here actually from porting to C++?
- fhub 1y agoFrom the post Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.