4 ms·

> LLMs have no understanding of the underlying reality Not true. They are slowly gaining an understanding of reality by reverse engineering the relationships b

> LLMs have no understanding of the underlying reality Not true. They are slowly gaining an understanding of reality by reverse engineering the relationships built into human languages. The only reason LLMs are getting better is because they are better modeling the world. At some point the only way to improve token prediction is to gain an understanding of the world.

pdonis 3y ago
> They are slowly gaining an understanding of reality by reverse engineering the relationships built into human languages. Perhaps the people building LLMs are doing this, but the LLMs themselves are not. LLMs are just generating text. They aren't "reverse engineering" anything. > The only reason LLMs are getting better is because they are better modeling the world. No, they are getting better at generating text that seems fluent and coherent, as long as you don't inquire into any actual semantic relationships with the world. LLMs can't model the world because they don't even have a concept of "the world". All they have is text.
- teaearlgraycold 3y ago
  Here's a bit from an episode of This American Life that I think is relevant, cut into a post on my blog: https://danangell.com/blog/posts/gpt-understands/ https://danangell.com/blog/posts/gpt-understands/ > The idea is that back when GPT-4 was being trained for it to really consistently get the next word correct, to do that reliably, it had to do more than just bullshit. It had to do more than guess based on patterns. To get the next word right, it had to truly understand the words coming before it. Imagine if you were training a transformer to predict numbers in a sequence that came from a sine function. Eventually it would get a pretty good test score and you'd find that internally it had built a decent replication of the sine function. Now do the same with English text. Throw enough data and computation at it and eventually it'll model a rough understanding of gravity and the shapes of objects, as seen in the different answers between GPT-3.5 and GPT-4.
  - thfuran 3y ago
    Does an analog circuit understand differential equations?
    - teaearlgraycold 3y ago
      I think it's not really interesting to say so, but by the same definition it would be fair to describe an analog circuit that models some system as understanding the system in some capacity. It's more interesting when the thing doing the understanding is a massive system that has both mastered English and other domains and then incidentally modeled some systems English can describe. An analog circuit that models artillery trajectories "understands" one kinematic equation. But that's kind of obvious and mundane. Understanding just one equation barely registers as profound and without that understanding existing in a larger context the word barely seems applicable.
- PoignardAzur 3y ago
  This subjects comes up in every single discussion on LLMs, and every time someone asserts with perfect confidence that LLMs couldn't possibly have a world model. But that confidence is based on intuition, not on any empirical observations. All the research we've seen so far points in the other direction: sufficiently trained LLMs do have a world model that you can find in their weights, changing these weights changes their completions in a way consistent with the new world model, etc. See OthelloGPT for the most blatant example. I'm starting to wish we had something like a FAQ to point to, with a summary of all the research on the subject, because it's pretty settled by now.
  - happytiger 3y ago
    Any network of sufficiently complex connected nodes develops characteristics of complexity that mirror intelligence. This was always the conjecture of the “ghost in the machine” and mech futurist sci-fi writers. Would you be willing to provide some reading about this? I’d love to see 1-2 things that could help me (and others) know more about what you know.
  - pdonis 3y ago
    > LLMs couldn't possibly have a world model I'm not asserting that they "couldn't possibly". I'm asserting that they don't--as in, none of the LLMs we have today have a world model. Perhaps someone might invent a different something in the future that they call an "LLM" that does have a world model, but no such thing exists now. > that confidence is based on intuition, not on any empirical observations No, it's not based on either of those. It's based on the explicit descriptions of how existing LLMs work by the people who made them. Their descriptions make clear that LLMs are just confabulating text based on a given prompt and their training data. The LLM doesn't make any connection at all between the text and anything else. Contrast, for example, with Wolfram Alpha. If you give Wolfram Alpha the prompt "What is the distance from New York to Tokyo", it doesn't confabulate text based on that prompt and some corpus of training data. It first parses the text and infers that the correct response involves a lookup in its geographic database (note that you can skip this step by explicitly selecting the particular tool that does this); then it does the lookup; then it renders the response into text form and outputs it. That is what having a (rudimentary) world model looks like. LLMs do nothing of the sort.
- gaganyaan 3y ago
  > LLMs are just generating text. I really can't stand these low-effort takes. It's like saying "Human brains are just meat". You can summarize anything badly, but it's not helpful to the conversation. Whatever quibbles you have aside, LLMs are generating useful text, today. At some point you're just pointlessly privileging meat over silicon.
  - pdonis 3y ago
    > It's like saying "Human brains are just meat". No, it isn't, because human brains have rich semantic connections to the rest of the world. LLMs do not. So "LLMs are just generating text" is a justified description of their limitations, in a way that "human brains are just meat" is not.
    - gaganyaan 3y ago
      > LLMs do not Prove it
      - pdonis 3y ago
        I don't have to; the people who designed the LLMs have already done so by explicitly describing how they work. See, for example, my contrast of what an LLM does with what Wolfram Alpha does upthread.
        gaganyaan 3y ago
        Wolfram Alpha has a model of the world manually created by humans. LLMs generate a model of the world from training data. "Explicitly describing" how something works doesn't mean you wave your hands and say "therefore meat is magic". I have no idea why you think Wolfram Alpha is relevant. LLMs can use tools like langchain to do the exact same thing Wolfram Alpha does. What proof do you have for your original claim?

pdonis 3y ago

> They are slowly gaining an understanding of reality by reverse engineering the relationships built into human languages. Perhaps the people building LLMs are doing this, but the LLMs themselves are not. LLMs are just generating text. They aren't "reverse engineering" anything. > The only reason LLMs are getting better is because they are better modeling the world. No, they are getting better at generating text that seems fluent and coherent, as long as you don't inquire into any actual semantic relationships with the world. LLMs can't model the world because they don't even have a concept of "the world". All they have is text.

teaearlgraycold 3y ago
Here's a bit from an episode of This American Life that I think is relevant, cut into a post on my blog: https://danangell.com/blog/posts/gpt-understands/ https://danangell.com/blog/posts/gpt-understands/ > The idea is that back when GPT-4 was being trained for it to really consistently get the next word correct, to do that reliably, it had to do more than just bullshit. It had to do more than guess based on patterns. To get the next word right, it had to truly understand the words coming before it. Imagine if you were training a transformer to predict numbers in a sequence that came from a sine function. Eventually it would get a pretty good test score and you'd find that internally it had built a decent replication of the sine function. Now do the same with English text. Throw enough data and computation at it and eventually it'll model a rough understanding of gravity and the shapes of objects, as seen in the different answers between GPT-3.5 and GPT-4.
- thfuran 3y ago
  Does an analog circuit understand differential equations?
  - teaearlgraycold 3y ago
    I think it's not really interesting to say so, but by the same definition it would be fair to describe an analog circuit that models some system as understanding the system in some capacity. It's more interesting when the thing doing the understanding is a massive system that has both mastered English and other domains and then incidentally modeled some systems English can describe. An analog circuit that models artillery trajectories "understands" one kinematic equation. But that's kind of obvious and mundane. Understanding just one equation barely registers as profound and without that understanding existing in a larger context the word barely seems applicable.
PoignardAzur 3y ago
This subjects comes up in every single discussion on LLMs, and every time someone asserts with perfect confidence that LLMs couldn't possibly have a world model. But that confidence is based on intuition, not on any empirical observations. All the research we've seen so far points in the other direction: sufficiently trained LLMs do have a world model that you can find in their weights, changing these weights changes their completions in a way consistent with the new world model, etc. See OthelloGPT for the most blatant example. I'm starting to wish we had something like a FAQ to point to, with a summary of all the research on the subject, because it's pretty settled by now.
- happytiger 3y ago
  Any network of sufficiently complex connected nodes develops characteristics of complexity that mirror intelligence. This was always the conjecture of the “ghost in the machine” and mech futurist sci-fi writers. Would you be willing to provide some reading about this? I’d love to see 1-2 things that could help me (and others) know more about what you know.
- pdonis 3y ago
  > LLMs couldn't possibly have a world model I'm not asserting that they "couldn't possibly". I'm asserting that they don't--as in, none of the LLMs we have today have a world model. Perhaps someone might invent a different something in the future that they call an "LLM" that does have a world model, but no such thing exists now. > that confidence is based on intuition, not on any empirical observations No, it's not based on either of those. It's based on the explicit descriptions of how existing LLMs work by the people who made them. Their descriptions make clear that LLMs are just confabulating text based on a given prompt and their training data. The LLM doesn't make any connection at all between the text and anything else. Contrast, for example, with Wolfram Alpha. If you give Wolfram Alpha the prompt "What is the distance from New York to Tokyo", it doesn't confabulate text based on that prompt and some corpus of training data. It first parses the text and infers that the correct response involves a lookup in its geographic database (note that you can skip this step by explicitly selecting the particular tool that does this); then it does the lookup; then it renders the response into text form and outputs it. That is what having a (rudimentary) world model looks like. LLMs do nothing of the sort.
gaganyaan 3y ago
> LLMs are just generating text. I really can't stand these low-effort takes. It's like saying "Human brains are just meat". You can summarize anything badly, but it's not helpful to the conversation. Whatever quibbles you have aside, LLMs are generating useful text, today. At some point you're just pointlessly privileging meat over silicon.
- pdonis 3y ago
  > It's like saying "Human brains are just meat". No, it isn't, because human brains have rich semantic connections to the rest of the world. LLMs do not. So "LLMs are just generating text" is a justified description of their limitations, in a way that "human brains are just meat" is not.
  - gaganyaan 3y ago
    > LLMs do not Prove it
    - pdonis 3y ago
      I don't have to; the people who designed the LLMs have already done so by explicitly describing how they work. See, for example, my contrast of what an LLM does with what Wolfram Alpha does upthread.
      - gaganyaan 3y ago
        Wolfram Alpha has a model of the world manually created by humans. LLMs generate a model of the world from training data. "Explicitly describing" how something works doesn't mean you wave your hands and say "therefore meat is magic". I have no idea why you think Wolfram Alpha is relevant. LLMs can use tools like langchain to do the exact same thing Wolfram Alpha does. What proof do you have for your original claim?