4 ms·
This is due to the tokenization scheme. These LLMs can’t see individual characters, they see chunks of characters that are glued together to form tokens. It’s
by gamegoblin 3y ago
This is due to the tokenization scheme. These LLMs can’t see individual characters, they see chunks of characters that are glued together to form tokens.
It’s impressive that they understand individual-character-level questions as well as they do.