4 ms·

> A LLM (...) merely produces syntactically likely combinations of the training data While doesn't matter much in the rest of the policy, this is a common misc

> A LLM (...) merely produces syntactically likely combinations of the training data While doesn't matter much in the rest of the policy, this is a common misconception among AI skeptics. It is not the case for a long time (since RL is used heavily in the training) and a LLM may go beyond its training data.

aabhay 2mo ago
I think the questionable term is not “syntactically likely” but “merely”. Syntactical likeness is a vast solution space that encompasses the work of a terrible developer and a genius developer. In fact this solution space is a gap wide enough to encompass all coding knowledge and expertise.
- rcxdude 2mo ago
  "Combinations of the training data" is also hiding a pretty common and serious misconception about modern generative AI: this seems to come from a mental model that LLMs and image generators are merely producing a collage of sorts, especially that this is some way linked to specific parts of the training data as opposed to a generalized result of building up an understanding of how to produce a sentence, piece of code, or plausible image.
  - inigyou 2mo ago
    That is how they feel when using them for code. It feels like they generate templates filled in with the details of a specific case.
runarberg 2mo ago
I assume RL stands for reinforcement learning, which just means that the weights are adjusted by evaluating some loss function. If the loss function uses the training data (i.e. supervised learning) then the original assumption still holds. So I fail to see how this is a misconception.
- atherton94027 2mo ago
  I think OP might be referring to synthetic data, which is now used to provide more data for training?
  - runarberg 2mo ago
    That would makes sense. This is how AlphaGo became better then the training data, and was the basis for Alpha Zero which used no human data and became even better. However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.
- tulio_ribeiro 2mo ago
  Supervised learning and reinforcement learning are mathematically distinct. Supervised pre-training as you said maximizes the likelihood of a token sequence. RL optimizes a policy against an external reward signal or execution environment. For example, RL evaluates code against runtime execution/unit tests/proof checkers (e.g lean). Models learn new strategies and are able to produce novel code if trained in an RL environment.
  - runarberg 2mo ago
    I don‘t understand what you mean. Supervised learning is reinforcement learning, while not all reinforcement learning is supervised learning (e.g unsupervised learning is also reinforcement learning; i.e. reinforcement learning with unlabelled data). Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
    - xdavidliu 2mo ago
      that's not how the terminology is typical used, even if it makes literal sense
e12e 2mo ago
Doesn't the LLM still output the next most probable token (modulus heat/desired variation) according to its model? Training shifts the probability - but doesn't change the fact that the output is a sampling based on input and the model?
- josu 2mo ago
  The combination of the most probable tokens doesn't necessarily have to be in the training set, thus creating something completely novel.
  - DiabloD3 2mo ago
    That is called a hallucination.
    - Gander5739 2mo ago
      Only if it's incorrect.
      - deleted 2mo ago
        [deleted]
    - josu 2mo ago
      Yes, like the Jacobian counterexample.
      - inigyou 2mo ago
        That LLM was attached to a Python and MATLAB evaluator. It generated plausible things to feed into a real computer.
        pixl97 2mo ago
        Weird take as a lot of things humans invent and test are done in similar ways.
LiamPowell 2mo ago
The natural next argument that I see a lot is "well it's still all probabilistic", which is technically true. However, don't the atoms that make up the cells that make up a human move around and interact based on probabilities? Saying a LLM is all just based on probabilities is pretty meaningless, even if it is true, since everything else is just based on probabilities too. What matters is the massive amount of machinery that generates those probabilities. Anything from rolling a dice to an accurate model of an entire human, or even a model of the entire universe, is all "just probabilities".
- runarberg 2mo ago
  That argument is mostly used against the misconception of AI-stans that LLMs are compilers, English is a programming language, and that prompting is a programing. There are many reasons for why that is false, and LLMs being stochastic is one of the more obvious reasons. But there are more probabilistic elements about LLMs. Inference is by default deterministic for any given input and weights, however it is made stochastic algorithmically, when the model chooses the second or third likeliest outcome with some probability. Training is stochastic, the adjustment of the weights is probabilistic, etc. etc. You seem to make the fact that our brains are also probabilistic as some sort of a gotcha. However that has never been in dispute. Neurons fire with certain probability, this has been known since the advent of neuroscience. However this is why computers are preferred for certain tasks over human brains. When you run a program you can be certain it behaves in a certain way given a particular input and parameters.
  - LiamPowell 2mo ago
    The "gotcha" here, if you want to call it that, is that the picking the next token based on probabilities (either in training or in the sampler) is not an argument that LLMs are inherently flawed, especially when one reduces a LLM to "it just picks a token based on a probability" as you could apply that same description to a human.
    - runarberg 2mo ago
      Like I said, usually this is used to debunk false claims about LLMs being compilers. I personally think there are better arguments to be made to debunk that, but the probabilistic nature of LLMs is an obvious one, so it is not unsurprising that it is also popular. I don‘t think I have ever seen anybody make a claim that LLMs are a bad (or otherwise limited) technology because of the probabilistic nature of it. Plenty of excellent algorithms (including other machine learning algorithms) are probabilistic and work excellently for what they are meant to do. Problems arise when the algorithm is used for something more, and that is the case against LLMs. It is a next token prediction algorithm that people are using to write software. If they treat it as a compiler it will be lacking, and LLMs being non-deterministic is one of many reasons for why LLMs are bad compilers (or more accurately; are in fact not compilers). And for that matter, neither are humans compilers. If you find an AI-hater who says “humans are good compilers” then I will agree with you that that person is wrong.
probably_wrong 2mo ago
I would like to offer this comment from three days ago as counterpoint: https://news.ycombinator.com/item?id=49013694 https://news.ycombinator.com/item?id=49013694
- lifthrasiir 2mo ago
  Common sense testing only tells about the presence of common sense, that's it. It is clear from recent events that LLMs do definitely possess some sort of intelligence even though it might have been polarized to certain domains.
- scotty79 2mo ago
  What does that prove? That AIs are not RLd for counting letters in texts? Do you doubt that they could excel at it with only synthetic data?

aabhay 2mo ago

I think the questionable term is not “syntactically likely” but “merely”. Syntactical likeness is a vast solution space that encompasses the work of a terrible developer and a genius developer. In fact this solution space is a gap wide enough to encompass all coding knowledge and expertise.

rcxdude 2mo ago
"Combinations of the training data" is also hiding a pretty common and serious misconception about modern generative AI: this seems to come from a mental model that LLMs and image generators are merely producing a collage of sorts, especially that this is some way linked to specific parts of the training data as opposed to a generalized result of building up an understanding of how to produce a sentence, piece of code, or plausible image.
- inigyou 2mo ago
  That is how they feel when using them for code. It feels like they generate templates filled in with the details of a specific case.

runarberg 2mo ago

I assume RL stands for reinforcement learning, which just means that the weights are adjusted by evaluating some loss function. If the loss function uses the training data (i.e. supervised learning) then the original assumption still holds. So I fail to see how this is a misconception.

atherton94027 2mo ago
I think OP might be referring to synthetic data, which is now used to provide more data for training?
- runarberg 2mo ago
  That would makes sense. This is how AlphaGo became better then the training data, and was the basis for Alpha Zero which used no human data and became even better. However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.
tulio_ribeiro 2mo ago
Supervised learning and reinforcement learning are mathematically distinct. Supervised pre-training as you said maximizes the likelihood of a token sequence. RL optimizes a policy against an external reward signal or execution environment. For example, RL evaluates code against runtime execution/unit tests/proof checkers (e.g lean). Models learn new strategies and are able to produce novel code if trained in an RL environment.
- runarberg 2mo ago
  I don‘t understand what you mean. Supervised learning is reinforcement learning, while not all reinforcement learning is supervised learning (e.g unsupervised learning is also reinforcement learning; i.e. reinforcement learning with unlabelled data). Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
  - xdavidliu 2mo ago
    that's not how the terminology is typical used, even if it makes literal sense

e12e 2mo ago

Doesn't the LLM still output the next most probable token (modulus heat/desired variation) according to its model? Training shifts the probability - but doesn't change the fact that the output is a sampling based on input and the model?

josu 2mo ago
The combination of the most probable tokens doesn't necessarily have to be in the training set, thus creating something completely novel.
- DiabloD3 2mo ago
  That is called a hallucination.
  - Gander5739 2mo ago
    Only if it's incorrect.
    - deleted 2mo ago
      [deleted]
  - josu 2mo ago
    Yes, like the Jacobian counterexample.
    - inigyou 2mo ago
      That LLM was attached to a Python and MATLAB evaluator. It generated plausible things to feed into a real computer.
      - pixl97 2mo ago
        Weird take as a lot of things humans invent and test are done in similar ways.

LiamPowell 2mo ago

The natural next argument that I see a lot is "well it's still all probabilistic", which is technically true. However, don't the atoms that make up the cells that make up a human move around and interact based on probabilities? Saying a LLM is all just based on probabilities is pretty meaningless, even if it is true, since everything else is just based on probabilities too. What matters is the massive amount of machinery that generates those probabilities. Anything from rolling a dice to an accurate model of an entire human, or even a model of the entire universe, is all "just probabilities".

runarberg 2mo ago
That argument is mostly used against the misconception of AI-stans that LLMs are compilers, English is a programming language, and that prompting is a programing. There are many reasons for why that is false, and LLMs being stochastic is one of the more obvious reasons. But there are more probabilistic elements about LLMs. Inference is by default deterministic for any given input and weights, however it is made stochastic algorithmically, when the model chooses the second or third likeliest outcome with some probability. Training is stochastic, the adjustment of the weights is probabilistic, etc. etc. You seem to make the fact that our brains are also probabilistic as some sort of a gotcha. However that has never been in dispute. Neurons fire with certain probability, this has been known since the advent of neuroscience. However this is why computers are preferred for certain tasks over human brains. When you run a program you can be certain it behaves in a certain way given a particular input and parameters.
- LiamPowell 2mo ago
  The "gotcha" here, if you want to call it that, is that the picking the next token based on probabilities (either in training or in the sampler) is not an argument that LLMs are inherently flawed, especially when one reduces a LLM to "it just picks a token based on a probability" as you could apply that same description to a human.
  - runarberg 2mo ago
    Like I said, usually this is used to debunk false claims about LLMs being compilers. I personally think there are better arguments to be made to debunk that, but the probabilistic nature of LLMs is an obvious one, so it is not unsurprising that it is also popular. I don‘t think I have ever seen anybody make a claim that LLMs are a bad (or otherwise limited) technology because of the probabilistic nature of it. Plenty of excellent algorithms (including other machine learning algorithms) are probabilistic and work excellently for what they are meant to do. Problems arise when the algorithm is used for something more, and that is the case against LLMs. It is a next token prediction algorithm that people are using to write software. If they treat it as a compiler it will be lacking, and LLMs being non-deterministic is one of many reasons for why LLMs are bad compilers (or more accurately; are in fact not compilers). And for that matter, neither are humans compilers. If you find an AI-hater who says “humans are good compilers” then I will agree with you that that person is wrong.

probably_wrong 2mo ago

I would like to offer this comment from three days ago as counterpoint: https://news.ycombinator.com/item?id=49013694 https://news.ycombinator.com/item?id=49013694

lifthrasiir 2mo ago
Common sense testing only tells about the presence of common sense, that's it. It is clear from recent events that LLMs do definitely possess some sort of intelligence even though it might have been polarized to certain domains.
scotty79 2mo ago
What does that prove? That AIs are not RLd for counting letters in texts? Do you doubt that they could excel at it with only synthetic data?