3 ms·
New tokenizer has a much larger vocabulary (200k)[0]. [0] https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8#diff-0d973848bd229
by zackangelo 2y ago
New tokenizer has a much larger vocabulary (200k)[0].
[0] https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8#diff-0d973848bd229418209db2c46c86167000845592ca6b98fad215c21c317bc494R9 https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74c...