17 ms·
It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down e
by minimaxir 3y ago
It was fun to follow the public TinyLlama loss curves in near real-time, although it showed that it can be frustrating since the loss curves barely moved down even after an extra trillion tokens: https://wandb.ai/lance777/lightning_logs/reports/metric-train_loss-23-09-04-23-38-15---Vmlldzo1MzA4MzIw?accessToken=5eu2sndit2mo6eqls8h38sklcgfwt660ek1f2czlgtqjv2c6tida47qm1oty8ik9 https://wandb.ai/lance777/lightning_logs/reports/metric-trai... (note the log-scaled X-axis)
But they did move down and that's what's important.
There should probably be more aggressive learning rate annealing for models trying to be Chinchilla-optimal instead of just cosine-with-warmup like every other model nowadays.
- ssivark 3y agoHow crucial is it to freeze the learning rate schedule a priori, instead of tweaking it on the fly?
- grandma_tea 3y agoAdaptive learning rate is a thing. For example, one scheme I've used before is to decrease the learning rate if the validation loss stops decreasing. It's not clear to me if this is applicable to LLMs though.
- minimaxir 3y agoConstant learning rates were the default in older ML implementations, but linear decay became an obvious optimization, and now we have both warmup and cosine decay to handle common training patterns, especially with the AdamW optimizer. If the learning rate is too high at a given point in training, it can result in either a) the model stopping learning or b) exploding gradients, which is very bad.
- WhitneyLand 3y agoMore aggressive learning rate? Wouldn’t we need more information on why they decided to stop training at this point to conclude that? My current understanding of the story is, to recap: - First the game was increase model size massively - For example GPT3 had 175B parameters, but less than 0.5T tokens of training data - Then Chinchilla showed for a given compute budget we can scale better by increasing training data - Now we have models like this, and Phi, that have over 1T trained tokens For any model, the loss curve going down could mean it’s learning, or could mean it’s overfitting, we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before. So getting back to your comment, I thought they were actually a multitude of indicators that should be used, not just validation loss, to determine what would be gained with more training.
- minimaxir 3y ago> Wouldn’t we need more information on why they decided to stop training at this point to conclude that? The experiment was fixed at 3 epochs on 1T tokens, they didn't decide to "stop" at a given criterion. > we don’t know which without looking at validation loss, which is like a second set of test data the model hasn’t seen before. The data I linked shows the validation loss, which has the same behavior as the training loss.
- viraptor 3y agoI'd love to see someone go for another few epochs in the future. Two of the benchmarks got a significant jump almost at the end of training. I wonder if there's a chance for more of that - looks like an interesting effect on its own.
- smpanaro 3y agoThe jump was due to them fixing a bug. There’s a footnote about it on the bottom of page 5. In the Discord, they mentioned a TinyLLaMa v2, presumably that would have this bug (and another bug, footnote page 4) fixed.
- zozbot234 3y agoOverfitting is quite unlikely with a smaller model though. Model parsimony provides a kind of regularization "for free", in fact with the extra benefit of saving on compute costs.