4 ms·
OpenAI seems to use Tiktoken [0]. It also covers GPT-4 token encoding. [0] https://github.com/openai/tiktoken https://github.com/openai/tiktoken
by coffeeri 3y ago
OpenAI seems to use Tiktoken [0]. It also covers GPT-4 token encoding.
[0] https://github.com/openai/tiktoken https://github.com/openai/tiktoken
- deleted 3y ago[deleted]
- lspears 3y agoSeems odd they don't reference this on the page. Instead they list: "If you need a programmatic interface for tokenizing text, check out the transformers package for python or the gpt-3-encoder package for node.js." with the links: https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast https://huggingface.co/docs/transformers/model_doc/gpt2#tran... https://www.npmjs.com/package/gpt-3-encoder https://www.npmjs.com/package/gpt-3-encoder