4 ms·

Yeah, that's kind of a different conception of understanding though. The lines do get a little blurry at a certain point, and a lot of what it does "feels" like

Yeah, that's kind of a different conception of understanding though. The lines do get a little blurry at a certain point, and a lot of what it does "feels" like understanding, especially given how it "communicates". But I think it comes down to whether it can reason about things and whether it can draw new conclusions or create new information as a result. Your snake site is probably a good example. ChatGPT has a bunch of words that it knows are associated with snakes. It's pretty straightforward pattern matching. It doesn't really "understand" what those words mean, except that they have relationships to other words. But, if you were to ask it to reason and draw new conclusions about these things beyond its training corpus, it would be unable to reliably do so. Similarly, it had no idea about the quality (and sometimes legality) of the chess moves it generated.

bongodongobob 3y ago
I mostly agree. I really think chess is just a terrible example though. You're really asking a lot of it but I'm honestly shocked it can do what it can. It seems to know some opening books, but falls down immediately. Which really makes sense because you'll find a lot of reading material on specific openings, but the problem space of the game is just too big to find texts about any given game state. Maybe if you have it reason about the board state and "think" about tactics you could push it farther. But we've already solved this. We have Stockfish et al and they've literally changed the game. Asking an LLM to play chess, while cool, is like trying to train a fish to dance. I think once we have an AI that's built with a bunch of different models that specialize in different things the idea of understanding is going to get even blurrier to the point that we might even say "yes, it doesn't 'understand' things, but it's better at humans at literally everything" so the difference becomes meaningless. I'm also of the opinion that humans are fancy automatons, so I tend to argue both sides. I'll say yeah it's thinking and so do we or ok it's not, but either do we.
- unclebucknasty 3y ago
  >Asking an LLM to play chess, while cool, is like trying to train a fish to dance. 100%. I didn't expect much from it going in. Was more curious to know how good/bad it might be and, really, whether it could play at all. But, I mention it here because it illustrates the disparity between understanding/reasoning (LLMs bad) and semantics (LLMs good), per the subject of the thread. >get even blurrier to the point that we might even say "yes, it doesn't 'understand' things, but it's better at humans at literally everything" so the difference becomes meaningless. I get exactly what you're saying here and I've wondered the same. But, that's the thing I'm not sure about, given its inability to reason. I mean, there's the idea that it doesn't matter whether it "understands" if it correctly answers. But, then I wonder how it can "truly" be better than humans at "literally everything", if only humans truly understand.
  - famouswaffles 3y ago
    >Asking an LLM to play chess, while cool, is like trying to train a fish to dance. 100%. I didn't expect much from it going in. Was more curious to know how good/bad it might be and, really, whether it could play at all. >But, I mention it here because it illustrates the disparity between understanding/reasoning (LLMs bad) and semantics (LLMs good), per the subject of the thread. LLMs can play chess (links above) so i'm curious if you still have this opinion.
    - unclebucknasty 3y ago
      >i'm curious if you still have this opinion Largely. I'm not surprised that another model could be trained on a larger chess corpus or otherwise tuned for improvement. I actually alluded to that in my original comment above. In fact, I was really surprised that gpt4 was so bad, given the total volume of training data it had obviously ingested. So, my initial point was around how that revealed the disparity between LLMs appearing to understand versus actually understanding (i.e. engaging in reasoning).
      - famouswaffles 3y ago
        If i came upon a random stranger who did not know how to play chess (but otherwise had heard of the game) and i put the board in front of him, made my move and told him to proceed, he will almost certainly move a piece to some other position on the board. There is no difference between this exercise and the one you have given GPT-4. In fact, the results are manifested the exact same way. By at most a few moves in, i will notice that both the stranger and GPT-4 are moving pieces to some general area of the board but woefully failing to adhere to the rules of the game. "The stranger can not reason" would be an absurd conclusion. Swapping out The stranger for GPT-4 does not make it any less so. So really by this standard then Humans regularly only "appear to understand" a great number of things. I can get behind this, but I doubt this is your conclusion. It seems to me that you were using "appear to understand" and "understand" as something that separates Humans from LLMs.
        3y ago
- famouswaffles 3y ago
  >I really think chess is just a terrible example though. You're really asking a lot of it but I'm honestly shocked it can do what it can. It seems to know some opening books, but falls down immediately. Which really makes sense because you'll find a lot of reading material on specific openings, but the problem space of the game is just too big to find texts about any given game state. Maybe if you have it reason about the board state and "think" about tactics you could push it farther. But we've already solved this. Just so you know, LLMs can play chess just fine. Like this isn't some inability. GPT-3.5-instruct-turbo plays at about 1800 Elo and makes legal moves consistently well past openings. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
  - unclebucknasty 3y ago
    From the README: >Per move, a model gets 5 illegal moves before forced resignation of the round. >Most of gpt-4's losses were due to illegal moves, so it may be possible to come up with a prompt to have gpt-4 correct illegal moves and improve its score. My experience exactly with gpt4. I first solved by keeping a list of illegal moves and subsequently prompting to move again and explicitly exclude those moves. Idea was to constrain gpt as little as possible, but that became inefficient. So I began to prompt for each move with the complete set of legal moves to choose from. In any case, at a certain point it becomes somewhat random when the goal is just to produce a legal move. But, it's interesting that producing legal moves is a challenge for gpt4, especially given that it's such a discrete problem.
  - bongodongobob 3y ago
    The github you linked shows ~1200, not 1800 which is orders of magnitude difference. The other is fine tuning, which is out of scope. A 1200 has something like a 1% chance of beating an 1800. 1200 is sort of you know the rules and maybe have a general idea of an openings, make weak tactical moves mid game and poor concept of endgame. It's just too "deep" for an LLM and a poor standard regardless. As an avid chess and GPT person, I've tried to get it to play decent chess but can't. Neither of your links support it, but if you've got anything else, send it.
visarga 3y ago
> it comes down to whether it can reason about things and whether it can draw new conclusions or create new information as a result Neither can humans, at least with our bare brains. We can do it by carefully observing the effects of our actions in the environment, but we are really studying the world and it takes time. Everything we know comes from the environment. The brain by itself invents or discovers nothing, it is the data-engine made by action-effect-feedback that teaches us all we know. Without the ability to push and prod, set up our experiments and carefully observe effects we wouldn't be at our current level. Environment is the teacher, but there is another important factor - language. Without it every one of us would have to rediscover from scratch. With it we can build upon other people to learn and cooperate. We encode everything we know in language. It acts like an evolutionary system of ideas. LLMs have what is necessary, they can learn language pretty well, but until now have not been exposed much to the world. There are millions of chats but very little in other kinds of environments - computers, simulators, games, robots. LLMs can create their own experiences and learn from each other, and from us. Open ended discovery is a grand project, a social process, it doesn't work well in one agent. Language is the linking element, and the world is the teacher. Some things are not written in any books, only the external world can teach us. Reasoning about things and drawing new conclusions depends on having access to an environment.
- unclebucknasty 3y ago
  I'm grappling too much with whether this argument has any meaning to give a cogent response. It kind of has a, "yeah, but if reality didn't exist" feel to it. Could humans reason if we somehow lived in a void? I don't know. But, then I guess it really wouldn't matter.
  - visarga 3y ago
    You should think about how we get our training data. That is important, we don't have in our DNA that much information, most of it comes by learning. If you don't include the provenance of the information we use to train ourselves you don't really look at the whole process. The brain by itself does nothing just like computers without software and input data. On the other hand scientists need experimentation to come up with anything new. It never comes from pure thinking, there is always the experimental validation. The place we get all our knowledge from is the environment. Without environment as grounding we are just language hallucinators, remember the string theory - that was a nice ungrounded hallucination that kept physicists busy for decades. Without validation our theories are worthless. So a LLM trained purely on human text can't make radical discoveries but when it learns from the environment it can. AlphaGo discovered move 37 from its self-play experience.
famouswaffles 3y ago
>Similarly, it had no idea about the quality (and sometimes legality) of the chess moves it generated. LLMs can play chess just fine. This isn't some inability. You just tested a model that couldn't. GPT-3.5-instruct-turbo plays at about 1800 Elo and makes legal moves consistently well past openings. https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... >Your snake site is probably a good example. ChatGPT has a bunch of words that it knows are associated with snakes. It's pretty straightforward pattern matching. It doesn't really "understand" what those words mean, except that they have relationships to other words. Lol there is absolutely nothing straightforward about that. It's just so funny, people so ready to relegate anything to "pattern matching" that any supposed difference becomes meaningless.

bongodongobob 3y ago

I mostly agree. I really think chess is just a terrible example though. You're really asking a lot of it but I'm honestly shocked it can do what it can. It seems to know some opening books, but falls down immediately. Which really makes sense because you'll find a lot of reading material on specific openings, but the problem space of the game is just too big to find texts about any given game state. Maybe if you have it reason about the board state and "think" about tactics you could push it farther. But we've already solved this. We have Stockfish et al and they've literally changed the game. Asking an LLM to play chess, while cool, is like trying to train a fish to dance. I think once we have an AI that's built with a bunch of different models that specialize in different things the idea of understanding is going to get even blurrier to the point that we might even say "yes, it doesn't 'understand' things, but it's better at humans at literally everything" so the difference becomes meaningless. I'm also of the opinion that humans are fancy automatons, so I tend to argue both sides. I'll say yeah it's thinking and so do we or ok it's not, but either do we.

unclebucknasty 3y ago
>Asking an LLM to play chess, while cool, is like trying to train a fish to dance. 100%. I didn't expect much from it going in. Was more curious to know how good/bad it might be and, really, whether it could play at all. But, I mention it here because it illustrates the disparity between understanding/reasoning (LLMs bad) and semantics (LLMs good), per the subject of the thread. >get even blurrier to the point that we might even say "yes, it doesn't 'understand' things, but it's better at humans at literally everything" so the difference becomes meaningless. I get exactly what you're saying here and I've wondered the same. But, that's the thing I'm not sure about, given its inability to reason. I mean, there's the idea that it doesn't matter whether it "understands" if it correctly answers. But, then I wonder how it can "truly" be better than humans at "literally everything", if only humans truly understand.
- famouswaffles 3y ago
  >Asking an LLM to play chess, while cool, is like trying to train a fish to dance. 100%. I didn't expect much from it going in. Was more curious to know how good/bad it might be and, really, whether it could play at all. >But, I mention it here because it illustrates the disparity between understanding/reasoning (LLMs bad) and semantics (LLMs good), per the subject of the thread. LLMs can play chess (links above) so i'm curious if you still have this opinion.
  - unclebucknasty 3y ago
    >i'm curious if you still have this opinion Largely. I'm not surprised that another model could be trained on a larger chess corpus or otherwise tuned for improvement. I actually alluded to that in my original comment above. In fact, I was really surprised that gpt4 was so bad, given the total volume of training data it had obviously ingested. So, my initial point was around how that revealed the disparity between LLMs appearing to understand versus actually understanding (i.e. engaging in reasoning).
    - famouswaffles 3y ago
      If i came upon a random stranger who did not know how to play chess (but otherwise had heard of the game) and i put the board in front of him, made my move and told him to proceed, he will almost certainly move a piece to some other position on the board. There is no difference between this exercise and the one you have given GPT-4. In fact, the results are manifested the exact same way. By at most a few moves in, i will notice that both the stranger and GPT-4 are moving pieces to some general area of the board but woefully failing to adhere to the rules of the game. "The stranger can not reason" would be an absurd conclusion. Swapping out The stranger for GPT-4 does not make it any less so. So really by this standard then Humans regularly only "appear to understand" a great number of things. I can get behind this, but I doubt this is your conclusion. It seems to me that you were using "appear to understand" and "understand" as something that separates Humans from LLMs.
      - 3y ago
famouswaffles 3y ago
>I really think chess is just a terrible example though. You're really asking a lot of it but I'm honestly shocked it can do what it can. It seems to know some opening books, but falls down immediately. Which really makes sense because you'll find a lot of reading material on specific openings, but the problem space of the game is just too big to find texts about any given game state. Maybe if you have it reason about the board state and "think" about tactics you could push it farther. But we've already solved this. Just so you know, LLMs can play chess just fine. Like this isn't some inability. GPT-3.5-instruct-turbo plays at about 1800 Elo and makes legal moves consistently well past openings. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
- unclebucknasty 3y ago
  From the README: >Per move, a model gets 5 illegal moves before forced resignation of the round. >Most of gpt-4's losses were due to illegal moves, so it may be possible to come up with a prompt to have gpt-4 correct illegal moves and improve its score. My experience exactly with gpt4. I first solved by keeping a list of illegal moves and subsequently prompting to move again and explicitly exclude those moves. Idea was to constrain gpt as little as possible, but that became inefficient. So I began to prompt for each move with the complete set of legal moves to choose from. In any case, at a certain point it becomes somewhat random when the goal is just to produce a legal move. But, it's interesting that producing legal moves is a challenge for gpt4, especially given that it's such a discrete problem.
- bongodongobob 3y ago
  The github you linked shows ~1200, not 1800 which is orders of magnitude difference. The other is fine tuning, which is out of scope. A 1200 has something like a 1% chance of beating an 1800. 1200 is sort of you know the rules and maybe have a general idea of an openings, make weak tactical moves mid game and poor concept of endgame. It's just too "deep" for an LLM and a poor standard regardless. As an avid chess and GPT person, I've tried to get it to play decent chess but can't. Neither of your links support it, but if you've got anything else, send it.

visarga 3y ago

> it comes down to whether it can reason about things and whether it can draw new conclusions or create new information as a result Neither can humans, at least with our bare brains. We can do it by carefully observing the effects of our actions in the environment, but we are really studying the world and it takes time. Everything we know comes from the environment. The brain by itself invents or discovers nothing, it is the data-engine made by action-effect-feedback that teaches us all we know. Without the ability to push and prod, set up our experiments and carefully observe effects we wouldn't be at our current level. Environment is the teacher, but there is another important factor - language. Without it every one of us would have to rediscover from scratch. With it we can build upon other people to learn and cooperate. We encode everything we know in language. It acts like an evolutionary system of ideas. LLMs have what is necessary, they can learn language pretty well, but until now have not been exposed much to the world. There are millions of chats but very little in other kinds of environments - computers, simulators, games, robots. LLMs can create their own experiences and learn from each other, and from us. Open ended discovery is a grand project, a social process, it doesn't work well in one agent. Language is the linking element, and the world is the teacher. Some things are not written in any books, only the external world can teach us. Reasoning about things and drawing new conclusions depends on having access to an environment.

unclebucknasty 3y ago
I'm grappling too much with whether this argument has any meaning to give a cogent response. It kind of has a, "yeah, but if reality didn't exist" feel to it. Could humans reason if we somehow lived in a void? I don't know. But, then I guess it really wouldn't matter.
- visarga 3y ago
  You should think about how we get our training data. That is important, we don't have in our DNA that much information, most of it comes by learning. If you don't include the provenance of the information we use to train ourselves you don't really look at the whole process. The brain by itself does nothing just like computers without software and input data. On the other hand scientists need experimentation to come up with anything new. It never comes from pure thinking, there is always the experimental validation. The place we get all our knowledge from is the environment. Without environment as grounding we are just language hallucinators, remember the string theory - that was a nice ungrounded hallucination that kept physicists busy for decades. Without validation our theories are worthless. So a LLM trained purely on human text can't make radical discoveries but when it learns from the environment it can. AlphaGo discovered move 37 from its self-play experience.

famouswaffles 3y ago

>Similarly, it had no idea about the quality (and sometimes legality) of the chess moves it generated. LLMs can play chess just fine. This isn't some inability. You just tested a model that couldn't. GPT-3.5-instruct-turbo plays at about 1800 Elo and makes legal moves consistently well past openings. https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... >Your snake site is probably a good example. ChatGPT has a bunch of words that it knows are associated with snakes. It's pretty straightforward pattern matching. It doesn't really "understand" what those words mean, except that they have relationships to other words. Lol there is absolutely nothing straightforward about that. It's just so funny, people so ready to relegate anything to "pattern matching" that any supposed difference becomes meaningless.