3 ms·

I don’t think we’ve yet come to the point where, how an LLM end to end goes from prompt to output is blatantly obvious. LLMs operate with language at multiple

I don’t think we’ve yet come to the point where, how an LLM end to end goes from prompt to output is blatantly obvious. LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge. For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge. And of course LLMs famously have emergent properties, for which there’s not yet a precise mechanism to illuminate the results. We’re dealing with very complex systems that are still relatively poorly understood, and I believe the pool of concepts understood to the point of being blatantly obvious is still small.

hn_throwaway_99 2y ago
The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!" We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer . When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens". But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens. Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.
- homarp 2y ago
  teaching LLM to say 'i don't know' in general is hard, not just for counting letters.
  - theGnuMe 2y ago
    It can't without some Bayesian action.
- WhitneyLand 2y ago
  You seem pretty certain for having only the most rudimentary understanding. I’m still going to have to disagree. I’d describe the idea that token granularity is the cause of letter counting limitations as a hypothesis, not as something that’s been conclusively shown as far as I’m aware. I’m not discounting it, or even saying it’s unlikely, but its not hard to imagine other ways it could plausibly be happening. As a side note when I mentioned training data I was not suggesting anything related to “token metadata”, or to tokens in any regard. Rather, I think it might be possible to instead improve the learning around certain types of counting in a way that could generalize.
  - hn_throwaway_99 2y ago
    > You seem pretty certain for having only the most rudimentary understanding. Fine, I lied. I have a much better understanding than you. Happy?
    - WhitneyLand 2y ago
      I only wish you the best brother.

hn_throwaway_99 2y ago

The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!" We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer . When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens". But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens. Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.

homarp 2y ago
teaching LLM to say 'i don't know' in general is hard, not just for counting letters.
- theGnuMe 2y ago
  It can't without some Bayesian action.
WhitneyLand 2y ago
You seem pretty certain for having only the most rudimentary understanding. I’m still going to have to disagree. I’d describe the idea that token granularity is the cause of letter counting limitations as a hypothesis, not as something that’s been conclusively shown as far as I’m aware. I’m not discounting it, or even saying it’s unlikely, but its not hard to imagine other ways it could plausibly be happening. As a side note when I mentioned training data I was not suggesting anything related to “token metadata”, or to tokens in any regard. Rather, I think it might be possible to instead improve the learning around certain types of counting in a way that could generalize.
- hn_throwaway_99 2y ago
  > You seem pretty certain for having only the most rudimentary understanding. Fine, I lied. I have a much better understanding than you. Happy?
  - WhitneyLand 2y ago
    I only wish you the best brother.