4 ms·NB: huggingface has long had a rust based tokenizer.by Vetch 4y agoNB: huggingface has long had a rust based tokenizer.