4 ms·

There is some limited truth in this but we still see claims that LLMs are "just next token predictors" and "just regurgitate code they read online". These are j

by nl 10mo ago

There is some limited truth in this but we still see claims that LLMs are "just next token predictors" and "just regurgitate code they read online". These are just uninformed and wrong views. It's fair to say that these people were (are!) wrong.

zahlman 10mo ago
Objecting to these claims is missing their point. Saying these things is really about denying that the LLMs "think" in any meaningful sense. (And the retorts I've seen in those discussions often imply very depressing and self-deprecating views of what it actually means to be human.)
- emp17344 10mo ago
  Leave it to HN to be militantly misanthropic to sell chatbots.
mjr00 10mo ago
> we still see claims that LLMs are "just next token predictors" and "just regurgitate code they read online". These are just uninformed and wrong views. It's fair to say that these people were (are!) wrong. I don't think it's fair to say that at all. How are LLMs not statistical models that predict tokens? It's a big oversimplification but it doesn't seem wrong, the same way that "computers are electricity running through circuits" isn't a wrong statement. And in both cases, those statements are orthogonal to how useful they are.
- jcelerier 10mo ago
  > How are LLMs not statistical models that predict tokens? there's LLMs as in "the blob of coefficients and graph operations that runs on a gpu whenever there's an inference" which is absolutely "a statistical model that predict tokens" and LLMs as in "the online apps that iterates and have access to an entire automated linux environment that can run $LANGUAGE scripts and do web queries when an intermediary statistical output contains too much maybes and use the result to drive further inference.".
- threethirtytwo 10mo ago
  It's wrong because it’s deliberately used to mischaracterize the current abilities of AI. Technically it's not wrong but the context of usage in basically every case is that the person saying it is deliberately trying to use the concept to downplay AI as just a pattern matching machine.
  - yladiz 10mo ago
    I'm a bit confused. You say it's wrong, but then later say it's not wrong, and just because it can be used to downplay advancements in AI doesn't mean that it's wrong and saying it's wrong because it can be used that way is a bit disingenuous.
    - threethirtytwo 10mo ago
      [flagged]
      - jibal 10mo ago
        You don't understand the meaning of "technically". Also, don't use inflammatory language.
        threethirtytwo 10mo ago
        I am not using inflammatory language to hurt anyone. I am illustrating a point on the contrast between technical meaning and non-technical meanings. One meaning is offensive the other meaning is technically correct. Don't start a witch hunt by deliberately misinterpreting what I'm saying. So technical means something like this: in a technical sense you are a stochastic parrot. You are also technically an object. But in everyday language we don't call people stochastic parrots or objects because language is nuanced and the technical meaning is rarely used at face value and other meanings are used in place of the technical one. So when people use a term in conversation and go by the technical meaning it's usually either very strange or done deliberately to deceive. Sort of like how you claim you don't know what "technically" means and sort of how you deliberately misinterpreted my words as "inflammatory" when I did nothing of the sort. I hope you learned something basic about the English today! Good day to you sir!
- nl 10mo ago
  > I don't think it's fair to say that at all. How are LLMs not statistical models that predict tokens? It's a big oversimplification but it doesn't seem wrong Modern LLMs are trained via reinforcement learning where the training objective is no longer maximum next token probability. They still produce tokens sequentially (ignoring diffusion models for now) but since the objective is so different thinking of them as next token predictors is more wrong than right. Instead one has to think of them as trying to fit their entire output to the model learnt in the reinforcement phase. That's how reasoning in LLMs works so well.
- Libidinalecon 10mo ago
  It is just a tell that the person believes LLMs are more than what they are ontologically. No one says "computers are JUST electricity running through circuits" because no one tries to argue the computer itself is "thinking" or has some kind of being. No one tries to argue that when you put the computer to sleep it is actually doing a form of "sleeping". The mighty token though produces all kinds of confused nonsense.

by nl 10mo ago

zahlman 10mo ago

Objecting to these claims is missing their point. Saying these things is really about denying that the LLMs "think" in any meaningful sense. (And the retorts I've seen in those discussions often imply very depressing and self-deprecating views of what it actually means to be human.)

emp17344 10mo ago
Leave it to HN to be militantly misanthropic to sell chatbots.

mjr00 10mo ago

> we still see claims that LLMs are "just next token predictors" and "just regurgitate code they read online". These are just uninformed and wrong views. It's fair to say that these people were (are!) wrong. I don't think it's fair to say that at all. How are LLMs not statistical models that predict tokens? It's a big oversimplification but it doesn't seem wrong, the same way that "computers are electricity running through circuits" isn't a wrong statement. And in both cases, those statements are orthogonal to how useful they are.

jcelerier 10mo ago
> How are LLMs not statistical models that predict tokens? there's LLMs as in "the blob of coefficients and graph operations that runs on a gpu whenever there's an inference" which is absolutely "a statistical model that predict tokens" and LLMs as in "the online apps that iterates and have access to an entire automated linux environment that can run $LANGUAGE scripts and do web queries when an intermediary statistical output contains too much maybes and use the result to drive further inference.".
threethirtytwo 10mo ago
It's wrong because it’s deliberately used to mischaracterize the current abilities of AI. Technically it's not wrong but the context of usage in basically every case is that the person saying it is deliberately trying to use the concept to downplay AI as just a pattern matching machine.
- yladiz 10mo ago
  I'm a bit confused. You say it's wrong, but then later say it's not wrong, and just because it can be used to downplay advancements in AI doesn't mean that it's wrong and saying it's wrong because it can be used that way is a bit disingenuous.
  - threethirtytwo 10mo ago
    [flagged]
    - jibal 10mo ago
      You don't understand the meaning of "technically". Also, don't use inflammatory language.
      - threethirtytwo 10mo ago
        I am not using inflammatory language to hurt anyone. I am illustrating a point on the contrast between technical meaning and non-technical meanings. One meaning is offensive the other meaning is technically correct. Don't start a witch hunt by deliberately misinterpreting what I'm saying. So technical means something like this: in a technical sense you are a stochastic parrot. You are also technically an object. But in everyday language we don't call people stochastic parrots or objects because language is nuanced and the technical meaning is rarely used at face value and other meanings are used in place of the technical one. So when people use a term in conversation and go by the technical meaning it's usually either very strange or done deliberately to deceive. Sort of like how you claim you don't know what "technically" means and sort of how you deliberately misinterpreted my words as "inflammatory" when I did nothing of the sort. I hope you learned something basic about the English today! Good day to you sir!
nl 10mo ago
> I don't think it's fair to say that at all. How are LLMs not statistical models that predict tokens? It's a big oversimplification but it doesn't seem wrong Modern LLMs are trained via reinforcement learning where the training objective is no longer maximum next token probability. They still produce tokens sequentially (ignoring diffusion models for now) but since the objective is so different thinking of them as next token predictors is more wrong than right. Instead one has to think of them as trying to fit their entire output to the model learnt in the reinforcement phase. That's how reasoning in LLMs works so well.
Libidinalecon 10mo ago
It is just a tell that the person believes LLMs are more than what they are ontologically. No one says "computers are JUST electricity running through circuits" because no one tries to argue the computer itself is "thinking" or has some kind of being. No one tries to argue that when you put the computer to sleep it is actually doing a form of "sleeping". The mighty token though produces all kinds of confused nonsense.