3 ms·
Yes, tiktoken, here's a client side visualizer for it https://github.com/functorism/gpt4-tokenizer-visualizer https://github.com/functorism/gpt4-tokenizer-visu
by WilliamBerglund 3y ago
Yes, tiktoken, here's a client side visualizer for it
https://github.com/functorism/gpt4-tokenizer-visualizer https://github.com/functorism/gpt4-tokenizer-visualizer