4 ms·
can we please have some good example input/outputs in the readme itself? what is the expected output of print(enc.encode("hello world")) ?
by Omie6541 4y ago
can we please have some good example input/outputs in the readme itself? what is the expected output of print(enc.encode("hello world")) ?
- stabbles 4y agohttps://github.com/openai/tiktoken/blob/main/tests/test_simple_public.py https://github.com/openai/tiktoken/blob/main/tests/test_simp...
- mnks 4y agoHave a look at https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer which uses javascript reimplementation of the GPT-2 / GPT-3 BPE tokenizer. In this case it's [31373, 995].