4 ms·
Mini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an im
by slashdave 2y ago
Mini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.
- anon291 2y agoSure, but one might think that training on self-generated data would keep you in a constrained subset of minima, but that is not the case.