3 ms·

Here is the list of the 100k GPT-4 tokens as text file. https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a09193 https://gist.github.com/s-macke/ae83f6

Here is the list of the 100k GPT-4 tokens as text file. https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a09193 https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0... Yes, a lot of tokens are just for code. Edit: Here as raw link for the poor mobile devices: https://gist.githubusercontent.com/s-macke/ae83f6afb89794350f8d9a1ad8a09193/raw/7efdf09500e1865483e0b2a97f1c5408a540d7f5/gpt-4-tokens.txt https://gist.githubusercontent.com/s-macke/ae83f6afb89794350...

sillysaurusx 3y ago
Thanks for this! That was very nice and thoughtful. There’s something poetic about ULL being a token, but NULL not being one.
- thyselius 3y ago
  Saw many words missing their first letter. Realized it’s probably because it’s sometimes Null and sometimes null
- hammeiam 3y ago
  ULL is also used for designating Unsigned Long Long 64bit ints in c++, so it’s not just a part of the null symbol.
josecyc 3y ago
Where are all the tokens for other languages though? How are these getting tokenized?