3 ms·

I always presumed that "a pigment" is one token. I'm a total amateur in this field though.

• ralusek 3y ago
  Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words. If you want to just see the tokens for yourself, though, just enter some text here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
• sanxiyn 3y ago
  No, it isn't, at least on OpenAI tokenizer.
• jasonjmcghee 2y ago
  what constitutes a token is super unintuitive. my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters In either case, I do consistently see spaces between characters included as part of the token following the space. " paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is
  • jasonjmcghee 2y ago
    New winner " communication" at 14