4 ms·
Floating point precision is not involved (most LLM models still function after floating-point quantization). I am puzzled that some find this result at all sur
by slashdave 2y ago
Floating point precision is not involved (most LLM models still function after floating-point quantization).
I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.
- anon291 2y agoI'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.
- slashdave 2y agoNo, not even close. Gradients don't come to zero in the first place. Training is never perfect.
- anon291 2y agoLet's restate. I'd imagine you end up in local minima that are difficult to escape using model generated data. So sure, non-zero gradients, but if you plot the gradients, I would expect them to orbit at that point. But it seems like they diverge.
- slashdave 2y agoMini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.
- anon291 2y agoSure, but one might think that training on self-generated data would keep you in a constrained subset of minima, but that is not the case.