4 ms·

Thanks, that's a good pickup -I'll refine my prompt.

• psychphysic 3y ago
  But LLM have little concept of tokens don't they? Or at least well not know what their tokenizes is like.
  • teaearlgraycold 3y ago
    It can understand word boundaries, though. A space is its own token and there are special tokens for common words prefixed with a space or common word prefixes with a space in front, ex “ a”