3 ms·
I hadn't realized OpenAI's tiktoken Python library could work with other models outside of the OpenAI family, that's really useful: https://github.com/therealol
by simonw 2y ago
I hadn't realized OpenAI's tiktoken Python library could work with other models outside of the OpenAI family, that's really useful: https://github.com/therealoliver/Deepdive-llama3-from-scratch?tab=readme-ov-file#summary-of-the-steps-to-load-the-bpe-based-tokenizer https://github.com/therealoliver/Deepdive-llama3-from-scratc...
- therealoliver 2y agoI'm glad to have helped you :)
- moffkalast 2y agoIt's more than just that, practically every notable open model released in the past year or so uses tiktoken as the tokenizer.