4 ms·

> The point is not the specific question (while an egregious example), but rather how much you can trust the system. And the answer is "you can trust the syste

> The point is not the specific question (while an egregious example), but rather how much you can trust the system. And the answer is "you can trust the system a little (enough to answer that logic puzzle correctly anyway), and you'll likely be able to trust future versions more". Obviously nobody is expecting LLMs to be able to fix every possible bug, but it's entirely possible they will be able to fix enough to be useful. Saying "LLMs will not achieve this. Full stop." seems premature.

godelski 3y ago
I have to say it because a lot of HN users try to convince me that GPT has understanding. We actually have 0 ML models that demonstrate understanding. Frankly, we don't know if this is even possible yet. Sure, in the future I believe we'll have AGI, but an AGI would likely recognize that my statement is contextualized around the current environment and fitting common human speech patters rather than making an absolute immutable statement.
- silverbax88 3y ago
  They are just stochastic parrots. Impressive and useful, but that's all they are.
  - Timon3 3y ago
    The comment above is a completely original thought. It was not memorized and pattern-matched.
- Ajedi32 3y ago
  Depends on how you define "understanding" I guess, but it certainly seems to me like GPT-4 "understands" the concept of weight well enough to not be tripped up by this particular logic puzzle. You can argue it's just an imitation of "understanding" and not the "real thing", but how good does an imitation have to be before it's functionally identical to the real thing? You can argue different cherry-picked examples seem to demonstrate a lack of understanding, but does that mean it doesn't possess understanding in general or just that it doesn't understand those particular topics? I agree GPT-4 is not a human-like AGI, and that the people expecting it to behave as one or expecting a future version right around the corner that does are likely in for disappointment. But at the same time, LLMs are capable of writing coherent, never-before-seen code from natural language instructions, solving basic never-before-seen natural language logic puzzles, and playing half-decent chess moves from never-before-seen positions[1], all in a single model that was never specifically trained to do any of those things. To say those tasks don't require at least some level of intelligence or "understanding" seems far fetched to me. [1]: https://dkb.blog/p/chatgpts-chess-elo-is-1400 https://dkb.blog/p/chatgpts-chess-elo-is-1400
  - godelski 3y ago
    > You can argue different cherry-picked examples seem to demonstrate a lack of understanding To be clear, an example does not necessitate cherry-picking. Cherry picking specifically requires one to ignore evidence to the contrary. There's a certain irony here. The reason I state this is because the criteria upon which you give me is impossible. The list of examples is non-exhaustive. Nor do I have infinite time or space in which to provide these examples. The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer. The LLM demonstrates that it has the requisite knowledge, that it knows the relation between the two units, but it fails to make the correct conclusions which requires the understanding part: putting the knowledge together. The follow-up question also is an example, and a different one for that matter. I'm not sure your link creates a compelling argument. This is despite the fact that Chess itself is not generally considered a good setting for testing understanding. I mean we've pretty much agreed on that even before Deep Blue. We need to be very careful in how we state things and try to interpret others. I hope I have not mischaracterized your claims. But I'd also encourage you to not be so antagonistic with others, especially while demonstrating the very thing you accuse them of. Internet comments are not academic and that's okay, but we should still try to be friendly. Not that a little poking won't happen.
    - Ajedi32 3y ago
      I'm not sure why you interpreted my comment as "antagonistic" or "unfriendly". Yes I do disagree with you (at least assuming I'm understanding you correctly), but I'm not attacking you personally, just your argument. I guess it's fair to say my examples are cherry-picked too (though I didn't really give specific examples in my comment so much as entire general categories of problems ChatGPT is known to be proficient at solving). But they aren't so cherry-picked as for "random chance" or "that example was in the training data verbatim" to be possible explanations. It's not like I had ChatGPT answer 100 billion questions and am only showing you the top 0.1%. It's very common for ChatGPT to be perfectly correct even when answering logic puzzles or coding problems not in its training set. So if not those then what's your explanation other than "understanding"? On the flip side, I don't think it's possible to prove ChatGPT does not posses understanding with individual examples. (And it seems you agree?) Those are easily dismissable as just "ChatGPT isn't great at that particular task". Particularly given how many other tasks ChatGPT is great at. Scott Alexander figured this out back in 2019 before GPT-3 was even a thing (let alone ChatGPT), I think it's worth a read: https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g... > The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer. Not really though. The "pound of bricks or pound of feathers" question is specifically designed to trip up humans, and the specific formulation you used seems specifically designed to trip up LLMs (by playing off their tenancy to pattern match common sayings), yet despite those disadvantages GPT-4 succeeds. Further, I don't think "I'd expect even a child to be able to answer this" is a good metric. ChatGPT isn't human, so we shouldn't expect it to be good at everything humans are good at just because it's good at some things humans are good at. Again, I'm not trying to argue ChatGPT is a human-like AGI.
- fragmede 3y ago
  https://chat.openai.com/share/e7157554-6534-4b73-98b9-672e474254bf https://chat.openai.com/share/e7157554-6534-4b73-98b9-672e47... I used two words that probably don't exist anywhere in the training data and asked it (ChatGPT-3) to compare the weight of a kilogram of one vs an lb of the other. It initially gets it wrong, but when asked to show its work, it's suddenly able to decide that a kilogram of something is heavier than a pound of something. Using understanding or reasoning, or stochastic parroting; whatever you want to call it. I'm not claiming it's AGI, or sentient, but it's also clearly not just a Markov chain.
  - godelski 3y ago
    Be careful, some of the questions you ask are leading. This can be quite difficult because you can't give it any information. Think about if you were to program this, how you would ask it to automate and find a stopping criteria without hinting to it anything. The way I did it uses the same response because you could at least grab the output and compare (easier said than done). It is really easy to hint to it information. "why would that be the same? show your work" Hints at an error. If someone asked you that while solving a math problem would you think you made a mistake? Granted my question also hints as a mistake, but through repetition a reasoning entity would figure out I'm going to continually ask that and to stop second guessing itself.
jerf 3y ago
I've beaten the drum a few times that it is important not to conflate LLMs with AI as a whole. LLMs are never going to solve this. That's OK. That's not their job. Their job in the long run will be to interface between the linguistic world and some other internal representation in some other AI that is structured differently and is capable of doing things language models aren't. I mean, it's right in the name: Language model. It's a bit weird to expect a language model to do other things. In a weird sort of way LLMs stand to set the industry back a bit as people try to tickle them into being more than a language model, rather than figuring out how to adapt them to feed something else that can model the non-linguistic world better. It's kind of nifty that language models can become so overpowered in some dimensions that they are able to do what they can sort of do today, but we would almost certainly be better off lowering the power of the language model and using that compute on something else that would work more like AI as we want it to... it's just that we don't know how to do that yet.
- godelski 3y ago
  I mostly agree. The only fine point I'd disagree on is that there is a significant (academic) camp that believes that language equates to understanding. That to properly speak requires understanding (fwiw, I think LLMs demonstrate that this isn't true). So I definitely understand the Layman confusion. But we'd have to get into other arguments like embodiment, if cognition is functional, and many other philosophical discussions. I don't think the public is generally acquainted with these ideas and it is certainly clear that a growing number of researchers aren't either. There's just too much to learn that it's easy to push of other things, especially as we chase benchmarks. So I understand how people get confused, which is why I try to respond in detail. But also I think many focus on tiny discrepancies or are willing to argue from a novice perspective as if it is authoritative (which corelates strongly with expertise). But I agree, there is a lot we don't know. But there are some things we can reject, which is all I'm trying to do and I think we agree there. Maybe you agree with my nit picking.

godelski 3y ago

I have to say it because a lot of HN users try to convince me that GPT has understanding. We actually have 0 ML models that demonstrate understanding. Frankly, we don't know if this is even possible yet. Sure, in the future I believe we'll have AGI, but an AGI would likely recognize that my statement is contextualized around the current environment and fitting common human speech patters rather than making an absolute immutable statement.

silverbax88 3y ago
They are just stochastic parrots. Impressive and useful, but that's all they are.
- Timon3 3y ago
  The comment above is a completely original thought. It was not memorized and pattern-matched.
Ajedi32 3y ago
Depends on how you define "understanding" I guess, but it certainly seems to me like GPT-4 "understands" the concept of weight well enough to not be tripped up by this particular logic puzzle. You can argue it's just an imitation of "understanding" and not the "real thing", but how good does an imitation have to be before it's functionally identical to the real thing? You can argue different cherry-picked examples seem to demonstrate a lack of understanding, but does that mean it doesn't possess understanding in general or just that it doesn't understand those particular topics? I agree GPT-4 is not a human-like AGI, and that the people expecting it to behave as one or expecting a future version right around the corner that does are likely in for disappointment. But at the same time, LLMs are capable of writing coherent, never-before-seen code from natural language instructions, solving basic never-before-seen natural language logic puzzles, and playing half-decent chess moves from never-before-seen positions[1], all in a single model that was never specifically trained to do any of those things. To say those tasks don't require at least some level of intelligence or "understanding" seems far fetched to me. [1]: https://dkb.blog/p/chatgpts-chess-elo-is-1400 https://dkb.blog/p/chatgpts-chess-elo-is-1400
- godelski 3y ago
  > You can argue different cherry-picked examples seem to demonstrate a lack of understanding To be clear, an example does not necessitate cherry-picking. Cherry picking specifically requires one to ignore evidence to the contrary. There's a certain irony here. The reason I state this is because the criteria upon which you give me is impossible. The list of examples is non-exhaustive. Nor do I have infinite time or space in which to provide these examples. The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer. The LLM demonstrates that it has the requisite knowledge, that it knows the relation between the two units, but it fails to make the correct conclusions which requires the understanding part: putting the knowledge together. The follow-up question also is an example, and a different one for that matter. I'm not sure your link creates a compelling argument. This is despite the fact that Chess itself is not generally considered a good setting for testing understanding. I mean we've pretty much agreed on that even before Deep Blue. We need to be very careful in how we state things and try to interpret others. I hope I have not mischaracterized your claims. But I'd also encourage you to not be so antagonistic with others, especially while demonstrating the very thing you accuse them of. Internet comments are not academic and that's okay, but we should still try to be friendly. Not that a little poking won't happen.
  - Ajedi32 3y ago
    I'm not sure why you interpreted my comment as "antagonistic" or "unfriendly". Yes I do disagree with you (at least assuming I'm understanding you correctly), but I'm not attacking you personally, just your argument. I guess it's fair to say my examples are cherry-picked too (though I didn't really give specific examples in my comment so much as entire general categories of problems ChatGPT is known to be proficient at solving). But they aren't so cherry-picked as for "random chance" or "that example was in the training data verbatim" to be possible explanations. It's not like I had ChatGPT answer 100 billion questions and am only showing you the top 0.1%. It's very common for ChatGPT to be perfectly correct even when answering logic puzzles or coding problems not in its training set. So if not those then what's your explanation other than "understanding"? On the flip side, I don't think it's possible to prove ChatGPT does not posses understanding with individual examples. (And it seems you agree?) Those are easily dismissable as just "ChatGPT isn't great at that particular task". Particularly given how many other tasks ChatGPT is great at. Scott Alexander figured this out back in 2019 before GPT-3 was even a thing (let alone ChatGPT), I think it's worth a read: https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g... > The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer. Not really though. The "pound of bricks or pound of feathers" question is specifically designed to trip up humans, and the specific formulation you used seems specifically designed to trip up LLMs (by playing off their tenancy to pattern match common sayings), yet despite those disadvantages GPT-4 succeeds. Further, I don't think "I'd expect even a child to be able to answer this" is a good metric. ChatGPT isn't human, so we shouldn't expect it to be good at everything humans are good at just because it's good at some things humans are good at. Again, I'm not trying to argue ChatGPT is a human-like AGI.
fragmede 3y ago
https://chat.openai.com/share/e7157554-6534-4b73-98b9-672e474254bf https://chat.openai.com/share/e7157554-6534-4b73-98b9-672e47... I used two words that probably don't exist anywhere in the training data and asked it (ChatGPT-3) to compare the weight of a kilogram of one vs an lb of the other. It initially gets it wrong, but when asked to show its work, it's suddenly able to decide that a kilogram of something is heavier than a pound of something. Using understanding or reasoning, or stochastic parroting; whatever you want to call it. I'm not claiming it's AGI, or sentient, but it's also clearly not just a Markov chain.
- godelski 3y ago
  Be careful, some of the questions you ask are leading. This can be quite difficult because you can't give it any information. Think about if you were to program this, how you would ask it to automate and find a stopping criteria without hinting to it anything. The way I did it uses the same response because you could at least grab the output and compare (easier said than done). It is really easy to hint to it information. "why would that be the same? show your work" Hints at an error. If someone asked you that while solving a math problem would you think you made a mistake? Granted my question also hints as a mistake, but through repetition a reasoning entity would figure out I'm going to continually ask that and to stop second guessing itself.

jerf 3y ago

I've beaten the drum a few times that it is important not to conflate LLMs with AI as a whole. LLMs are never going to solve this. That's OK. That's not their job. Their job in the long run will be to interface between the linguistic world and some other internal representation in some other AI that is structured differently and is capable of doing things language models aren't. I mean, it's right in the name: Language model. It's a bit weird to expect a language model to do other things. In a weird sort of way LLMs stand to set the industry back a bit as people try to tickle them into being more than a language model, rather than figuring out how to adapt them to feed something else that can model the non-linguistic world better. It's kind of nifty that language models can become so overpowered in some dimensions that they are able to do what they can sort of do today, but we would almost certainly be better off lowering the power of the language model and using that compute on something else that would work more like AI as we want it to... it's just that we don't know how to do that yet.

godelski 3y ago
I mostly agree. The only fine point I'd disagree on is that there is a significant (academic) camp that believes that language equates to understanding. That to properly speak requires understanding (fwiw, I think LLMs demonstrate that this isn't true). So I definitely understand the Layman confusion. But we'd have to get into other arguments like embodiment, if cognition is functional, and many other philosophical discussions. I don't think the public is generally acquainted with these ideas and it is certainly clear that a growing number of researchers aren't either. There's just too much to learn that it's easy to push of other things, especially as we chase benchmarks. So I understand how people get confused, which is why I try to respond in detail. But also I think many focus on tiny discrepancies or are willing to argue from a novice perspective as if it is authoritative (which corelates strongly with expertise). But I agree, there is a lot we don't know. But there are some things we can reject, which is all I'm trying to do and I think we agree there. Maybe you agree with my nit picking.