4 ms·
OpenAI's Tokenizer Page for OSS Models
- rgs224 3y agoI kept having to debug prompt issues with open models So I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc http://tokenwiz.rahul.gs http://tokenwiz.rahul.gs