3 ms·

Yeah, everybody agrees on what a character is, right? It's just {an ASCII byte|a UTF8 code unit|a UTF16 code unit|a Unicode code point|a Unicode grapheme}.

• ntonozzi 3y ago
  I’m not saying it’s easy but it’s much better than tokens IMO. I think bytes would be understandable too.
  • criddell 3y ago
    Bytes are understandable but make no sense from a business point of view. If you submit the same simple query with UTF-8 and UTF-32, the latter will cost 4x as much.
    • xyzzyz 3y ago
      No API accepts input in UTF-32. Nobody uses this on the internet.
• geysersam 3y ago
  At least there are standards for characters. Nothing like that for tokens.
• sheepscreek 3y ago
  And we think tokens solve that problem? Spoiler alert: they don’t https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_times_more_expensive_than_english_the/ https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_tim...
  • est31 3y ago
    They don't but Google could have been more precise with which of the definitions listed by GP they mean by "character".