5 ms·

Is this an artifact of floating point precision or a fundamental mathematical truth.

slashdave 2y ago
Floating point precision is not involved (most LLM models still function after floating-point quantization). I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.
- anon291 2y ago
  I'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.
  - slashdave 2y ago
    No, not even close. Gradients don't come to zero in the first place. Training is never perfect.
    - anon291 2y ago
      Let's restate. I'd imagine you end up in local minima that are difficult to escape using model generated data. So sure, non-zero gradients, but if you plot the gradients, I would expect them to orbit at that point. But it seems like they diverge.
      - slashdave 2y ago
        Mini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.
        anon291 2y ago
        Sure, but one might think that training on self-generated data would keep you in a constrained subset of minima, but that is not the case.
skybrian 2y ago
It’s a lossy transformation, so you’re losing information each time. It’s never going to add information. However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much. Papers like this one show what not to do.
- visarga 2y ago
  > there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much Like, take for example search. Instead of training on a bunch of scraped texts, you take one prompt, select 10 references, and use it to synthesize an answer. Referencing multiple texts gives you more than training on them directly. The LLM could catch contradictions, observe the distribution of human opinions, note if the topic is controversial. And then output a wikipedia-like article. Do this billions of times, and you got a refined dataset. You can iterate on top, using the articles as source and writing meta articles. Or just silly studies like writing a paper about "Characters named Charlie in literature". You can slice and dice the data in any way, and analyze the cross section.
- anon291 2y ago
  It'll never add information, but one may think it would be useful to refine information as you feed 'good' model outputs into itself for training.

Floating point precision is not involved (most LLM models still function after floating-point quantization). I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.

anon291 2y ago
I'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.
- slashdave 2y ago
  No, not even close. Gradients don't come to zero in the first place. Training is never perfect.
  - anon291 2y ago
    Let's restate. I'd imagine you end up in local minima that are difficult to escape using model generated data. So sure, non-zero gradients, but if you plot the gradients, I would expect them to orbit at that point. But it seems like they diverge.
    - slashdave 2y ago
      Mini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.
      - anon291 2y ago
        Sure, but one might think that training on self-generated data would keep you in a constrained subset of minima, but that is not the case.

skybrian 2y ago

It’s a lossy transformation, so you’re losing information each time. It’s never going to add information. However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much. Papers like this one show what not to do.

visarga 2y ago
> there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much Like, take for example search. Instead of training on a bunch of scraped texts, you take one prompt, select 10 references, and use it to synthesize an answer. Referencing multiple texts gives you more than training on them directly. The LLM could catch contradictions, observe the distribution of human opinions, note if the topic is controversial. And then output a wikipedia-like article. Do this billions of times, and you got a refined dataset. You can iterate on top, using the articles as source and writing meta articles. Or just silly studies like writing a paper about "Characters named Charlie in literature". You can slice and dice the data in any way, and analyze the cross section.
anon291 2y ago
It'll never add information, but one may think it would be useful to refine information as you feed 'good' model outputs into itself for training.