4 ms·

Does anyone know of a chatgpt/gpt-4 tokenizer that can run client-side?

• WilliamBerglund 3y ago
  Yes, tiktoken, here's a client side visualizer for it https://github.com/functorism/gpt4-tokenizer-visualizer https://github.com/functorism/gpt4-tokenizer-visualizer
• YetAnotherNick 3y ago
  https://github.com/cogentapps/chat-with-gpt/blob/main/app/src/core/tokenizer/bpe.ts https://github.com/cogentapps/chat-with-gpt/blob/main/app/sr...
• sp332 3y ago
  https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken https://github.com/dqbd/tiktoken
  • oofsa 3y ago
    https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer is not for GPT-4 but GPT-3. https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/ supports GPT-4.
  • adroitboss 3y ago
    Thanks for the recommendations. I found the one I needed because of this comment thread.
• adroitboss 3y ago
  Thanks for the recommendations everyone.