3 ms·

> A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse pro

> A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data" I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an approximation of the unknown source that generated the source data in the first place. In the case of LLMs an approximation not of a human but of the sum of humanity that produced the training copora?

jbay808 2mo ago
Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem. And yeah, that means the base model training process is solving the inverse problem of finding an approximation for the processes that generate all human records and also anything else in the dataset with a compressible pattern to discover (weather data, etc). So it's not surprisng that some kind of world model emerges out of this.
- vrighter 2mo ago
  except that it's really not. It's about finding a function to fit the data. And "generalizing" is only when the question is jsut barely outside their training data. Like a truncated taylor series, is still somewhat accurate just outside the region it was designed for, But I couldn't say a truncated taylor series generalizes over the function it's approximating.
  - versteegen 2mo ago
    I urge you to reconsider your beliefs. You are missing something important because you are thinking in terms of low-dimensional statistics. Deep learning doesn't just fit data, it finds features (abstractions) of the data.
    - vrighter 2mo ago
      this is not a belief. it is a fact. Doing the same thing in a higher dimensional space is still doing the same thing.
      - SgtBastard 2mo ago
        I asked the original commenter to confirm my understanding of what they were saying. I find it a fascinating alternative view to what is largely well understood (your counter-point). The thing that stood out is the comment that it’s an approximation of the original data generator (humanity). Early approximations were poor (GPT 2-3, to an extent GPT-4). I’m not so sure I can reject the hypothesis that such an approximation can be found.

jbay808 2mo ago

Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem. And yeah, that means the base model training process is solving the inverse problem of finding an approximation for the processes that generate all human records and also anything else in the dataset with a compressible pattern to discover (weather data, etc). So it's not surprisng that some kind of world model emerges out of this.

vrighter 2mo ago
except that it's really not. It's about finding a function to fit the data. And "generalizing" is only when the question is jsut barely outside their training data. Like a truncated taylor series, is still somewhat accurate just outside the region it was designed for, But I couldn't say a truncated taylor series generalizes over the function it's approximating.
- versteegen 2mo ago
  I urge you to reconsider your beliefs. You are missing something important because you are thinking in terms of low-dimensional statistics. Deep learning doesn't just fit data, it finds features (abstractions) of the data.
  - vrighter 2mo ago
    this is not a belief. it is a fact. Doing the same thing in a higher dimensional space is still doing the same thing.
    - SgtBastard 2mo ago
      I asked the original commenter to confirm my understanding of what they were saying. I find it a fascinating alternative view to what is largely well understood (your counter-point). The thing that stood out is the comment that it’s an approximation of the original data generator (humanity). Early approximations were poor (GPT 2-3, to an extent GPT-4). I’m not so sure I can reject the hypothesis that such an approximation can be found.