6 ms·

Yes. The fact that they are so good at these things is why we might reasonably ask how good they are at making predictions about the world. The fact that their

Yes. The fact that they are so good at these things is why we might reasonably ask how good they are at making predictions about the world. The fact that their predictions are not particularly good is therefore of interest.

NoMoreNicksLeft 3y ago
There must be oodles of good "chess via text" in the training data. So they're pretty good at chess. But, without even checking, I know that all the real-world predictions in the training data (written by humans) is utter garbage. And that's without even assuming that 4chan (or 4chan-quality) text slipped into it. Humans are bad at predicting the real world, and convinced thoroughly that they're good and it's just other people who are bad. If one could prepare a training dataset large enough, with good predictions (which aren't necessarily correct predictions), would an LLM trained on that become good at it? [edit] This is a (bad?) assumption that humans could be good predictors, and that there's not something that just rules it out entirely. If they could, and if you could prepare the dataset of that. If they can't because physics disallows it or something, then LLMs which crudely mimic how humans predict these things will also find it impossible. I don't think it's impossible to be good at prediction (should be possible, at least if we narrow the categories a bit), I just think we all suck at it.
- dragonwriter 3y ago
  > There must be oodles of good "chess via text" in the training data. So they're pretty good at chess. LLMs, including ChatGPT (including GPT-4) without tools, are notoriously bad at chess, actually. Papers have been written about it.
  - famouswaffles 3y ago
    There's a community project in the Eleuther ai discord that has trained several pythia models on chess games and come out playing it just fine. and they play better with increased scale too. Then recently Open ai released GPT-3.5 turbo instruct and that plays chess reliably at around 1800 Elo. So i don't know about "notoriously bad". People probably just overestimate how much chess game data most language models are actually getting. That or chat RLHF tanks chess abilities.
  - Der_Einzige 3y ago
    A lot of the issues are related to the model trying to generate bad/illegal moves. Far better is to do RAG and give it a list of the valid moves given the current board state. Might be able to exploit current computer chess formats to do this more compactly in the prompt. I expect above 2000 elo if you do what I am describing with GPT4.
    - layman51 3y ago
      What does RAG mean? I’m interested because I have tried to play a chess game purely through chat and my experience is that GPT 3 always forgets what is happening in the game and wants to make illegal moves. I guess if there was a format (maybe PGN?) then maybe it could draw on some of its experience.
      - esafak 3y ago
        Retrieval Augmented Generation
    - janalsncm 3y ago
      The fact that it’s making illegal moves at all should tell you something. The model may be able to suggest reasonable seeming moves, but it can’t even understand the rules of the game or the current board position. Maybe ChatGPT plays at an 1800 elo but how many 1800 players regularly have a hard time making legal moves? Limiting ChatGPT to legal moves only might help, but it’s ultimately a band aid. I think this is what this article is about.
      - famouswaffles 3y ago
        3.5-turbo instruct does not have a hard time making legal moves. You many get one illegal move every 300 moves it makes. >The fact that it’s making illegal moves at all should tell you something It tells you nothing because even grandmasters make illegal moves occasionally.
        danShumway 3y ago
janalsncm 3y ago
There’s a difference between predictions which can be reasonably ascertained from the inputs and those which cannot. No language model can tell you the outcome of a fair coin flip. If you want a reasonable test, you can test autoregressive models on out of domain tasks like MNIST classification after pasting raw pixel values into the chat. This is a meaningful test because there’s enough information in the input to determine the outputs (any CS undergrad can train a model to do this). For the cases tested in the article, we can’t tell if there’s just not enough information to make a decision, or if the task is too far out of domain. Given GPT’s performance in other out of domain tasks, it’s likely the issue is both in this case.
- pk-protect-ai 3y ago
  That was exactly my point ... What you put is what you get.
pk-protect-ai 3y ago
You should not expect intelligence from LLMs. They merely predict the next token based on your input. You should not anticipate their predictions to be anything other than a normal distribution, due to the data they have been trained on.
- famouswaffles 3y ago
  They "just" predict tokens ? What does that actually means ? Do you know what they do in order to predict tokens accurately?
  - pk-protect-ai 3y ago
    This is a good one. The answer to this question is both very simple and very complex. Here is the simple (and very simplified) answer: Text is converted into tokens (indices of the tokenizer's vocabulary), so the model does not work with text directly but with the indices of the tokenizer's dictionary. The tokens are then organized into a tensor for processing. This tensor is typically a 2D tensor, where each row represents a sequence of tokens. The number of rows corresponds to the number of sequences being processed simultaneously (the batch size), and the number of columns corresponds to the length of the longest sequence in the batch (the sequence length). Each element in the tensor is an integer that represents a specific token. The tensor is then passed through an embedding layer, which converts each token (integer) into a dense vector of fixed size. The embedding layer is essentially a lookup table, where each row corresponds to a token in the vocabulary. It is expected that this embedding vector captures the semantic meaning of the input text. These embeddings undergo a series of arithmetic operations in each hidden layer of the model. Each element of the hidden layer matrix represents a real number, known as a weight. Weights are "learned" - approximated during training from an enormous amount of data. The set of weights learned from the data represents the probabilistic behavior of the data processed during training (please note that the weights do not represent probabilities directly). The embeddings are multiplied by these weights. This is followed by the addition of a bias term and the application of a non-linear activation function (like swish). This process transforms the embeddings, allowing the model to learn complex patterns in the data. This process continues for all the hidden layers, and this is where the "magic" happens. I can't explain this magic because I'm under-qualified for this. However, what I can tell is that probabilities of tokens appearing close to each other or far from each other are stored in the network of weights. The final layer of the model produces a tensor of logits, which represent the model’s predictions for the next token in the sequence for each input token. Depending on the architecture, a softmax function may be applied to these logits to produce a probability distribution over the vocabulary for each predicted token. The method for selecting the predicted token can vary, including choosing the token with the highest probability, using beam search, nucleus sampling, or other methods specified in the architecture. This is also part of the "magic". These predicted tokens are then converted back into human-readable text, forming the model’s generated response. This is how the model processes text and generates responses. You might find clues to the complex part in the simple answer tagged as "magic". I'm under-qualified for the magic part. You need to ask those with extensive knowledge of mathematics and probability theory to reveal the actual math behind the magic for you. But there is a part that no one can answer at the moment: How do the relations between tokens caught from the training data cause the spatial comprehension and basic arithmetic capabilities (yet very limited) in LLMs? We can't answer this question for living creatures as well. How can a goldfish distinguish "less" from "more", and how can chickens count to 10? In any case, LLMs are very spectacular predictors for what they have learned, and for what they haven't, they are even more spectacular.

NoMoreNicksLeft 3y ago

There must be oodles of good "chess via text" in the training data. So they're pretty good at chess. But, without even checking, I know that all the real-world predictions in the training data (written by humans) is utter garbage. And that's without even assuming that 4chan (or 4chan-quality) text slipped into it. Humans are bad at predicting the real world, and convinced thoroughly that they're good and it's just other people who are bad. If one could prepare a training dataset large enough, with good predictions (which aren't necessarily correct predictions), would an LLM trained on that become good at it? [edit] This is a (bad?) assumption that humans could be good predictors, and that there's not something that just rules it out entirely. If they could, and if you could prepare the dataset of that. If they can't because physics disallows it or something, then LLMs which crudely mimic how humans predict these things will also find it impossible. I don't think it's impossible to be good at prediction (should be possible, at least if we narrow the categories a bit), I just think we all suck at it.

dragonwriter 3y ago
> There must be oodles of good "chess via text" in the training data. So they're pretty good at chess. LLMs, including ChatGPT (including GPT-4) without tools, are notoriously bad at chess, actually. Papers have been written about it.
- famouswaffles 3y ago
  There's a community project in the Eleuther ai discord that has trained several pythia models on chess games and come out playing it just fine. and they play better with increased scale too. Then recently Open ai released GPT-3.5 turbo instruct and that plays chess reliably at around 1800 Elo. So i don't know about "notoriously bad". People probably just overestimate how much chess game data most language models are actually getting. That or chat RLHF tanks chess abilities.
- Der_Einzige 3y ago
  A lot of the issues are related to the model trying to generate bad/illegal moves. Far better is to do RAG and give it a list of the valid moves given the current board state. Might be able to exploit current computer chess formats to do this more compactly in the prompt. I expect above 2000 elo if you do what I am describing with GPT4.
  - layman51 3y ago
    What does RAG mean? I’m interested because I have tried to play a chess game purely through chat and my experience is that GPT 3 always forgets what is happening in the game and wants to make illegal moves. I guess if there was a format (maybe PGN?) then maybe it could draw on some of its experience.
    - esafak 3y ago
      Retrieval Augmented Generation
  - janalsncm 3y ago
    The fact that it’s making illegal moves at all should tell you something. The model may be able to suggest reasonable seeming moves, but it can’t even understand the rules of the game or the current board position. Maybe ChatGPT plays at an 1800 elo but how many 1800 players regularly have a hard time making legal moves? Limiting ChatGPT to legal moves only might help, but it’s ultimately a band aid. I think this is what this article is about.
    - famouswaffles 3y ago
      3.5-turbo instruct does not have a hard time making legal moves. You many get one illegal move every 300 moves it makes. >The fact that it’s making illegal moves at all should tell you something It tells you nothing because even grandmasters make illegal moves occasionally.
      - danShumway 3y ago

janalsncm 3y ago

There’s a difference between predictions which can be reasonably ascertained from the inputs and those which cannot. No language model can tell you the outcome of a fair coin flip. If you want a reasonable test, you can test autoregressive models on out of domain tasks like MNIST classification after pasting raw pixel values into the chat. This is a meaningful test because there’s enough information in the input to determine the outputs (any CS undergrad can train a model to do this). For the cases tested in the article, we can’t tell if there’s just not enough information to make a decision, or if the task is too far out of domain. Given GPT’s performance in other out of domain tasks, it’s likely the issue is both in this case.

pk-protect-ai 3y ago
That was exactly my point ... What you put is what you get.

pk-protect-ai 3y ago

You should not expect intelligence from LLMs. They merely predict the next token based on your input. You should not anticipate their predictions to be anything other than a normal distribution, due to the data they have been trained on.

famouswaffles 3y ago
They "just" predict tokens ? What does that actually means ? Do you know what they do in order to predict tokens accurately?
- pk-protect-ai 3y ago
  This is a good one. The answer to this question is both very simple and very complex. Here is the simple (and very simplified) answer: Text is converted into tokens (indices of the tokenizer's vocabulary), so the model does not work with text directly but with the indices of the tokenizer's dictionary. The tokens are then organized into a tensor for processing. This tensor is typically a 2D tensor, where each row represents a sequence of tokens. The number of rows corresponds to the number of sequences being processed simultaneously (the batch size), and the number of columns corresponds to the length of the longest sequence in the batch (the sequence length). Each element in the tensor is an integer that represents a specific token. The tensor is then passed through an embedding layer, which converts each token (integer) into a dense vector of fixed size. The embedding layer is essentially a lookup table, where each row corresponds to a token in the vocabulary. It is expected that this embedding vector captures the semantic meaning of the input text. These embeddings undergo a series of arithmetic operations in each hidden layer of the model. Each element of the hidden layer matrix represents a real number, known as a weight. Weights are "learned" - approximated during training from an enormous amount of data. The set of weights learned from the data represents the probabilistic behavior of the data processed during training (please note that the weights do not represent probabilities directly). The embeddings are multiplied by these weights. This is followed by the addition of a bias term and the application of a non-linear activation function (like swish). This process transforms the embeddings, allowing the model to learn complex patterns in the data. This process continues for all the hidden layers, and this is where the "magic" happens. I can't explain this magic because I'm under-qualified for this. However, what I can tell is that probabilities of tokens appearing close to each other or far from each other are stored in the network of weights. The final layer of the model produces a tensor of logits, which represent the model’s predictions for the next token in the sequence for each input token. Depending on the architecture, a softmax function may be applied to these logits to produce a probability distribution over the vocabulary for each predicted token. The method for selecting the predicted token can vary, including choosing the token with the highest probability, using beam search, nucleus sampling, or other methods specified in the architecture. This is also part of the "magic". These predicted tokens are then converted back into human-readable text, forming the model’s generated response. This is how the model processes text and generates responses. You might find clues to the complex part in the simple answer tagged as "magic". I'm under-qualified for the magic part. You need to ask those with extensive knowledge of mathematics and probability theory to reveal the actual math behind the magic for you. But there is a part that no one can answer at the moment: How do the relations between tokens caught from the training data cause the spatial comprehension and basic arithmetic capabilities (yet very limited) in LLMs? We can't answer this question for living creatures as well. How can a goldfish distinguish "less" from "more", and how can chickens count to 10? In any case, LLMs are very spectacular predictors for what they have learned, and for what they haven't, they are even more spectacular.