3 ms·
But LLM have little concept of tokens don't they? Or at least well not know what their tokenizes is like.
by psychphysic 3y ago
But LLM have little concept of tokens don't they? Or at least well not know what their tokenizes is like.
- teaearlgraycold 3y agoIt can understand word boundaries, though. A space is its own token and there are special tokens for common words prefixed with a space or common word prefixes with a space in front, ex “ a”