4 ms·

As Stallman said, "... it is important to realize that ChatGPT is not artificial intelligence. It has no intelligence; it doesn't know anything and doesn't unde

As Stallman said, "... it is important to realize that ChatGPT is not artificial intelligence. It has no intelligence; it doesn't know anything and doesn't understand anything. It plays games with words to make plausible-sounding English text, but any statements made in it are liable to be false. It can't avoid that because it doesn't know what the words mean."

K0balt 3y ago
This gets into a very fascinating semantic corner. While LLMs are “merely” token predictors, it seems that perhaps the “knowing” is actually imbedded in the large corpus of memetic data. In the same way that an infinitely detailed “choose your own adventure” book shows that data and computation are interchangeable by degree, (calculation vs lookup), my conjecture is that the often surprising effectiveness of LLMs, as well as there very human like flaws, are the result of this kind of computation-in-the-data scenario. I posit that the overt connections in textual human knowledge also carry the implied knowledge that provides the unwritten context necessary to understand, if enough vector relationships can be teased out of a sufficiently large corpus of data. It could easily be that our own “understanding” is also derived from the vast n-dimensional matrix that represents all human cultural knowledge.
- wongarsu 3y ago
  I would argue that "LLMs are merely token predictors" is even a misguided argument. They are token predictors because that's how we designed the only viable output, but that doesn't necessarily limit the amount of intelligence inside the model. You can have dumb token predictors, and you can have AGI that is a token predictor. For a though experiment, you could put a human in a room, and only communicate with them via a terminal. You can write them messages, but the only way they can communicate to you is by giving you a probability distribution for the next token of their answer, then you pick one with a method of your choosing, tell them what you picked, and they choose the next probability distribution. This would severely hamper their ability to communicate effectively, and somebody who only sees the interface and doesn't know the probabilities might conclude it's "only a token predictor". But is it any less of a general intelligence because of that? After all it's still a human, just with a "dumb" input/output protocol that happens to be a token predictor.
  - Jensson 3y ago
    The problem is that they are trained solely to predict tokens. The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton. If it was trained to solve logical problems or to fact check etc, then it is no longer just a token predictor. But as is all the model is trained to do is to predict tokens, and you notice that immediately when you use it because it leads to obvious issues with what the model can do. It is really easy to target sparse parts of its data and get it to say nonsense because it is just trained to predict tokens.
    - hackinthebochs 3y ago
      >The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton. You're being loose with semantics and drawing bad conclusions. The training describes the nature of the feedback given to the system. Being trained to predict tokens means it gets feedback on the quality of its predictions. This does not constrain or determine the structure of the learned model. The learned model is a function of the training time, quality and amount of data, as well as the architecture that is biased towards capturing certain features or entails limits on what can be captured. The point is, the space and properties of solutions to the problem of "predicting the next token" is not constrained by the nature of the feedback.
      - Jensson 3y ago
        > This does not constrain or determine the structure of the learned model It absolutely does! A human can learn to perform an action by reading a manual. But an LLM is only trained to predict tokens, if you feed it a manual it will learn to recite the manual, it wont learn how to perform the actions in the manual. This makes the training inherently flawed from a learning perspective, and that flaw is impossible to fix without moving past the token prediction way of training models.
        hackinthebochs 3y ago
        >if you feed it a manual it will learn to recite the manual, it wont learn how to perform the actions in the manual. This claim doesn't track with LLM-based agents. For example: https://towardsai.net/p/machine-learning/meet-webagent-deepminds-new-llm-that-follows-instructions-and-complete-tasks-on-websites https://towardsai.net/p/machine-learning/meet-webagent-deepm...
doctor_eval 3y ago
> It plays games with words to make plausible-sounding English text, but any statements made in it are liable to be false Unfortunately, I’ve met several humans to which this statement would also apply.
wongarsu 3y ago
> It plays games with words to make plausible-sounding English text, but any statements made in it are liable to be false So like a marketing department? Or most humans, most of the time, for that matter.
_lvbh 3y ago
Sounds like me

K0balt 3y ago

This gets into a very fascinating semantic corner. While LLMs are “merely” token predictors, it seems that perhaps the “knowing” is actually imbedded in the large corpus of memetic data. In the same way that an infinitely detailed “choose your own adventure” book shows that data and computation are interchangeable by degree, (calculation vs lookup), my conjecture is that the often surprising effectiveness of LLMs, as well as there very human like flaws, are the result of this kind of computation-in-the-data scenario. I posit that the overt connections in textual human knowledge also carry the implied knowledge that provides the unwritten context necessary to understand, if enough vector relationships can be teased out of a sufficiently large corpus of data. It could easily be that our own “understanding” is also derived from the vast n-dimensional matrix that represents all human cultural knowledge.

wongarsu 3y ago
I would argue that "LLMs are merely token predictors" is even a misguided argument. They are token predictors because that's how we designed the only viable output, but that doesn't necessarily limit the amount of intelligence inside the model. You can have dumb token predictors, and you can have AGI that is a token predictor. For a though experiment, you could put a human in a room, and only communicate with them via a terminal. You can write them messages, but the only way they can communicate to you is by giving you a probability distribution for the next token of their answer, then you pick one with a method of your choosing, tell them what you picked, and they choose the next probability distribution. This would severely hamper their ability to communicate effectively, and somebody who only sees the interface and doesn't know the probabilities might conclude it's "only a token predictor". But is it any less of a general intelligence because of that? After all it's still a human, just with a "dumb" input/output protocol that happens to be a token predictor.
- Jensson 3y ago
  The problem is that they are trained solely to predict tokens. The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton. If it was trained to solve logical problems or to fact check etc, then it is no longer just a token predictor. But as is all the model is trained to do is to predict tokens, and you notice that immediately when you use it because it leads to obvious issues with what the model can do. It is really easy to target sparse parts of its data and get it to say nonsense because it is just trained to predict tokens.
  - hackinthebochs 3y ago
    >The method signature at the end doesn't matter you are right there but what the model is trained to do matters a ton. You're being loose with semantics and drawing bad conclusions. The training describes the nature of the feedback given to the system. Being trained to predict tokens means it gets feedback on the quality of its predictions. This does not constrain or determine the structure of the learned model. The learned model is a function of the training time, quality and amount of data, as well as the architecture that is biased towards capturing certain features or entails limits on what can be captured. The point is, the space and properties of solutions to the problem of "predicting the next token" is not constrained by the nature of the feedback.
    - Jensson 3y ago
      > This does not constrain or determine the structure of the learned model It absolutely does! A human can learn to perform an action by reading a manual. But an LLM is only trained to predict tokens, if you feed it a manual it will learn to recite the manual, it wont learn how to perform the actions in the manual. This makes the training inherently flawed from a learning perspective, and that flaw is impossible to fix without moving past the token prediction way of training models.
      - hackinthebochs 3y ago
        >if you feed it a manual it will learn to recite the manual, it wont learn how to perform the actions in the manual. This claim doesn't track with LLM-based agents. For example: https://towardsai.net/p/machine-learning/meet-webagent-deepminds-new-llm-that-follows-instructions-and-complete-tasks-on-websites https://towardsai.net/p/machine-learning/meet-webagent-deepm...

doctor_eval 3y ago

> It plays games with words to make plausible-sounding English text, but any statements made in it are liable to be false Unfortunately, I’ve met several humans to which this statement would also apply.

wongarsu 3y ago

> It plays games with words to make plausible-sounding English text, but any statements made in it are liable to be false So like a marketing department? Or most humans, most of the time, for that matter.

_lvbh 3y ago

Sounds like me