3 ms·
GPT-4o uses BPE (byte pair encoding). They released `tiktoken` which allows you to count tokens in strings in python. pip install tiktoken >>> import t
by conesus 2y ago
GPT-4o uses BPE (byte pair encoding). They released `tiktoken` which allows you to count tokens in strings in python.
pip install tiktoken
>>> import tiktoken
>>> encoding = tiktoken.encoding_for_model("gpt-4o")
>>> print(encoding.encode("hello marcellus"))
[24912, 2674, 10936, 385]