5 ms·

This is correct. I sometimes feel like I'm yelling into a hurricane when I say this, but it bears repeating: it's a language model, not a world knowledge model.

This is correct. I sometimes feel like I'm yelling into a hurricane when I say this, but it bears repeating: it's a language model, not a world knowledge model. LLMs are exceptionally good at transforming many aspects of language - its proficiency in coding is derived from this, not because it "knows" imperative logic. Tasks where you're asking it to transform text from one form to another (make it shorter, make it longer, make it a different language, etc.) are where it excels. It's particularly poor at knowledge retrieval (i.e., hallucinations galore) and very bad at reasoning - but so far all of the breathless hype has been specifically about the use cases it's bad at and rarely about the cases where it's amazing!

teaearlgraycold 3y ago
In some ways I'm glad people are unable to understand how to use them. It gives me a competitive advantage.
marviel 3y ago
(EDIT: I'm understanding you to say "there is no world model in there". This was written as a counterpoint to that.) Counterpoint by example: Imagine someone reads the first half of the original article, and then closes their eyes and (without peeking) writes both: (1) The rest of the article, verbatim (2) All of the hacker news comments on this article's posting here, including yours, and this one. If this person existed, would they not have built a very successful world model?
famouswaffles 3y ago
LLMs are not very bad at reasoning. People say this but struggle to demonstrate how. There are many types of reasoning. For some like abstract analogical reasoning, LLMs are top % performers. https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196 There's many papers at this point demonstrating pretty strong reasoning. From Casual Reasoning, and explanatory reasoning to algorithmic reasoning. https://arxiv.org/abs/2305.00050 https://arxiv.org/abs/2305.00050 https://arxiv.org/abs/2204.02329 https://arxiv.org/abs/2204.02329 https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2211.09066 The only kind of reasoning that's somewhat lacking is logical reasoning. And even then it's not awful, just lagging behind equivalent human performance. So someone really needs to tell where this oft repeated "can't reason" rhetoric is coming from. Because the numbers sure as hell don't support it.
- travisjungroth 3y ago
  Logical reasoning is also the easiest to solve with a program if you can get it into the right format, which LLMs are good at.
cjbprime 3y ago
For what it's worth, I think it feels like you're yelling into a hurricane because you are kind of clearly incorrect. Can you think of a few questions (using unique nouns etc to avoid parroting) for which the answers require a world knowledge model, so that we can ask them to GPT-4? > LLMs are exceptionally good at transforming many aspects of language - its proficiency in coding is derived from this If you paste a Python script and ask GPT-4 what the script will output when run, it is usually correct. How does it achieve this without reasoning about the state of variables over time? I don't know why the "it's a language model" belief is so strong. For one thing, it's untrue technically: GPT-4 has been trained on image input as well (improving its world modeling as a result, even though it can only output text), so it is a multi-modal model, not a language model. For a second point, the fact that the model inputs images+text, and outputs text, doesn't prescribe what happens inside the model. That's just what's happening at the ends of the model. The best way to "predict the next token" from text input inside the model is to construct a world model which examines the causes of that text, and extracts consequences from that model. So that's what it does, with fairly poor reliability but clear capability.
- goncalo-r 3y ago
  It's not good at reasoning. I asked the following (the term substitutions are there to avoid parroting): Let's define the following terms: col means "and", pel means "or" and erf is equivalent to "col" if the first argument evaluates to "true", otherwise it's equivalent to "pel" can you evaluate the following statement? col(true, pel(false, true), erf(col(true, false), true)) And it said this: ... Next, we look at the expression "erf(col(true, false), true)". Since the first argument is true, erf is equivalent to col, which means we evaluate the entire expression as: col(true, false) = false Therefore, erf(col(true, false), true) = col(true, true) = true which is obviously wrong because the first arg is false. The final answer happens to be the same. So I then asked what about col(true, pel(false, true), erf(col(true, false), false)) and this is its full answer: Sure, let's evaluate the expression "col(true, pel(false, true), erf(col(true, false), false))" step by step: First, we look at the expression "pel(false, true)". This is an "or" operation, so we evaluate it as follows: pel(false, true) = true Next, we look at the expression "erf(col(true, false), false)". Since the first argument is true, erf is equivalent to col, which means we evaluate the entire expression as: col(true, false) = false Therefore, erf(col(true, false), false) = pel(false, false) = false Finally, we look at the expression "col(true, pel(false, true), erf(col(true, false), false))". Using the results from steps 1 and 2, we can evaluate this as: col(true, true, false) = true Therefore, the overall value of the statement "col(true, pel(false, true), erf(col(true, false), false))" is true. You can see it's mostly random nonsense masked as "reasoning". It's not even consistent across sentences. I assume if you do a bigger model, it might do better, but as soon as you increase the example size I suspect it will get it wrong again.
pmoriarty 3y ago
You might find this video[1] on why "GPT is smarter than you think" interesting. They're older, but the "Sparks of AGI" paper[2] and video[3] are also well worth your time. [1] - https://m.youtube.com/watch?v=wVzuvf9D9BU https://m.youtube.com/watch?v=wVzuvf9D9BU [2] - https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 [3] - https://m.youtube.com/watch?v=qbIk7-JPB2c https://m.youtube.com/watch?v=qbIk7-JPB2c
stevenhuang 3y ago
You think it's not a world model. You think it doesn't know imperative logic. Rightfully you're being ignored because you're not aware these are in fact open questions; we don't have the answers yet, and those that think they do have not looked deep enough into cognition/philosophy, where they'll find many proposed answers to these same questions also are theorized to underpin human consciousness.
- te_chris 3y ago
  This is a pointlessly religious dimissal of the comment you're replying to. Yes, there's a counterfactual to the comment but this sort of mysticism is ridiculous.
  - stevenhuang 3y ago
    The ability to recursively identify structural patterns is intelligent behavior and theorized to play a crucial component in consciousness known as predictive coding https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Free_energy_principle https://en.m.wikipedia.org/wiki/Free_energy_principle Our brains are theorized to work like this using some form of hierarchical latent structure, and learning via prediction errors. Sounds a lot like model building and LLMs, yes? It's some true tragic comedy that you think this is religious mysticism instead of stopping and wondering if you might not be lacking some knowledge foundation. You really couldn't have proven my point better. Please consider reflecting on your ability to assess material outside your expertise.

teaearlgraycold 3y ago

In some ways I'm glad people are unable to understand how to use them. It gives me a competitive advantage.

marviel 3y ago

(EDIT: I'm understanding you to say "there is no world model in there". This was written as a counterpoint to that.) Counterpoint by example: Imagine someone reads the first half of the original article, and then closes their eyes and (without peeking) writes both: (1) The rest of the article, verbatim (2) All of the hacker news comments on this article's posting here, including yours, and this one. If this person existed, would they not have built a very successful world model?

famouswaffles 3y ago

LLMs are not very bad at reasoning. People say this but struggle to demonstrate how. There are many types of reasoning. For some like abstract analogical reasoning, LLMs are top % performers. https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196 There's many papers at this point demonstrating pretty strong reasoning. From Casual Reasoning, and explanatory reasoning to algorithmic reasoning. https://arxiv.org/abs/2305.00050 https://arxiv.org/abs/2305.00050 https://arxiv.org/abs/2204.02329 https://arxiv.org/abs/2204.02329 https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2211.09066 The only kind of reasoning that's somewhat lacking is logical reasoning. And even then it's not awful, just lagging behind equivalent human performance. So someone really needs to tell where this oft repeated "can't reason" rhetoric is coming from. Because the numbers sure as hell don't support it.

travisjungroth 3y ago
Logical reasoning is also the easiest to solve with a program if you can get it into the right format, which LLMs are good at.

cjbprime 3y ago

For what it's worth, I think it feels like you're yelling into a hurricane because you are kind of clearly incorrect. Can you think of a few questions (using unique nouns etc to avoid parroting) for which the answers require a world knowledge model, so that we can ask them to GPT-4? > LLMs are exceptionally good at transforming many aspects of language - its proficiency in coding is derived from this If you paste a Python script and ask GPT-4 what the script will output when run, it is usually correct. How does it achieve this without reasoning about the state of variables over time? I don't know why the "it's a language model" belief is so strong. For one thing, it's untrue technically: GPT-4 has been trained on image input as well (improving its world modeling as a result, even though it can only output text), so it is a multi-modal model, not a language model. For a second point, the fact that the model inputs images+text, and outputs text, doesn't prescribe what happens inside the model. That's just what's happening at the ends of the model. The best way to "predict the next token" from text input inside the model is to construct a world model which examines the causes of that text, and extracts consequences from that model. So that's what it does, with fairly poor reliability but clear capability.

goncalo-r 3y ago
It's not good at reasoning. I asked the following (the term substitutions are there to avoid parroting): Let's define the following terms: col means "and", pel means "or" and erf is equivalent to "col" if the first argument evaluates to "true", otherwise it's equivalent to "pel" can you evaluate the following statement? col(true, pel(false, true), erf(col(true, false), true)) And it said this: ... Next, we look at the expression "erf(col(true, false), true)". Since the first argument is true, erf is equivalent to col, which means we evaluate the entire expression as: col(true, false) = false Therefore, erf(col(true, false), true) = col(true, true) = true which is obviously wrong because the first arg is false. The final answer happens to be the same. So I then asked what about col(true, pel(false, true), erf(col(true, false), false)) and this is its full answer: Sure, let's evaluate the expression "col(true, pel(false, true), erf(col(true, false), false))" step by step: First, we look at the expression "pel(false, true)". This is an "or" operation, so we evaluate it as follows: pel(false, true) = true Next, we look at the expression "erf(col(true, false), false)". Since the first argument is true, erf is equivalent to col, which means we evaluate the entire expression as: col(true, false) = false Therefore, erf(col(true, false), false) = pel(false, false) = false Finally, we look at the expression "col(true, pel(false, true), erf(col(true, false), false))". Using the results from steps 1 and 2, we can evaluate this as: col(true, true, false) = true Therefore, the overall value of the statement "col(true, pel(false, true), erf(col(true, false), false))" is true. You can see it's mostly random nonsense masked as "reasoning". It's not even consistent across sentences. I assume if you do a bigger model, it might do better, but as soon as you increase the example size I suspect it will get it wrong again.

pmoriarty 3y ago

You might find this video[1] on why "GPT is smarter than you think" interesting. They're older, but the "Sparks of AGI" paper[2] and video[3] are also well worth your time. [1] - https://m.youtube.com/watch?v=wVzuvf9D9BU https://m.youtube.com/watch?v=wVzuvf9D9BU [2] - https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 [3] - https://m.youtube.com/watch?v=qbIk7-JPB2c https://m.youtube.com/watch?v=qbIk7-JPB2c

stevenhuang 3y ago

You think it's not a world model. You think it doesn't know imperative logic. Rightfully you're being ignored because you're not aware these are in fact open questions; we don't have the answers yet, and those that think they do have not looked deep enough into cognition/philosophy, where they'll find many proposed answers to these same questions also are theorized to underpin human consciousness.

te_chris 3y ago
This is a pointlessly religious dimissal of the comment you're replying to. Yes, there's a counterfactual to the comment but this sort of mysticism is ridiculous.
- stevenhuang 3y ago
  The ability to recursively identify structural patterns is intelligent behavior and theorized to play a crucial component in consciousness known as predictive coding https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Free_energy_principle https://en.m.wikipedia.org/wiki/Free_energy_principle Our brains are theorized to work like this using some form of hierarchical latent structure, and learning via prediction errors. Sounds a lot like model building and LLMs, yes? It's some true tragic comedy that you think this is religious mysticism instead of stopping and wondering if you might not be lacking some knowledge foundation. You really couldn't have proven my point better. Please consider reflecting on your ability to assess material outside your expertise.