4 ms·
Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example: https://dl.acm.org/doi/10.5555/3524938.3525
by fxtentacle 2y ago
Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example:
https://dl.acm.org/doi/10.5555/3524938.3525354 https://dl.acm.org/doi/10.5555/3524938.3525354
"Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence.“
"we then propose a new weight initialization scheme with theoretical justification, that enables training without warmup"
Or "Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes"
https://arxiv.org/html/2410.05052v1 https://arxiv.org/html/2410.05052v1
If all local minima were similar, then these papers would represent a lot of wasted effort. Or said in a different way: If labs like DeepMind are willing to pay for large teams that are trying to optimize initialization, then surely this represents a real existing problem.
- noosphr 2y agoThere are basically no people who have done pre-training of LLMs. What you're describing is absolutely the case, but everyone not in a top 100 AI lab would have never had the displeasure of having to deal with that. That said I strongly suggest everyone follow along with a GPT2 tutorial like: https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT to get a feel for what the big boys have to deal with. Any $100,000 training run can end in failure for no reason what so ever because you just got unlucky with your random seed. If you want something that is both SOTA and cheap: https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert you can get from pre-training to fine tuning for a couple hundred dollars with a model that is better than anything available to the public. Bert models are so under researched that you legitimately push the state of the art for them on a work station. Also feel free to drop a line at my profile's address.
- kristjansson 2y agonit: a divergent training run is somewhat separate from the question of local minima. The latter observes the loss curve stabilize and asks 'is the _best_ (loss-minimizing) set of parameters for this loss/data/model/hypers/seed/etc?'. The former observes the loss curve explode, and asks for another 10k GPU hours to try again with a different choice of loss/data/model/... . They're definitely related in some instances e.g. 'flat' vs 'steep' local minima with comparable losses.