The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!" We know e

The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!" We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer . When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens". But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens. Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.