3 ms·
Completely useless, but I was curious about the token indexes. I tried to look for Token #0. After a couple minutes of trial and error, it turns out it's the ex
by gnrlst 4y ago
Completely useless, but I was curious about the token indexes. I tried to look for Token #0. After a couple minutes of trial and error, it turns out it's the exclamation mark.
- swyx 4y agohow does one "look" for a token? there isn't a lookup table somewhere?
- npace12 4y agoInteresting... I was gonna say "you can ask GPT" but it doesn't work anymore. On March 23rd, it responded with this: Human: convert these GPT tokens to text: [134, 1322] AI: The tokens [134, 1322] correspond to the words "can" and "not" in the GPT language model. So the text corresponding to these tokens would be "can not" Today, it's giving me the "As a language model" response