There is at least some awareness already when it comes to the performance of the regex engine: https://github.com/openai/tiktoken/blob/main/src/lib.rs#L95-L110

There is at least some awareness already when it comes to the performance of the regex engine: https://github.com/openai/tiktoken/blob/main/src/lib.rs#L95-L110 https://github.com/openai/tiktoken/blob/main/src/lib.rs#L95-...