3 ms·
From anecdotal testing, using the 774M/1.5B GPT-2 models for anything less than hundreds of megabytes of input data will result in worse generation quality than
by minimaxir 7y ago
From anecdotal testing, using the 774M/1.5B GPT-2 models for anything less than hundreds of megabytes of input data will result in worse generation quality than using the smaller 124M/355M models.
The addiction to the larger GPT-2 models is IMO a trap.
- Tenoke 7y agoIt's definitely not the case for me. I have models trained on the same dataset which is 14mb (though I needed to tweak more for the 1.5b). 1.5b outperforms it here if trained long enough - in this case 1-2 months as I was doing it all for free on Colab. One of the big things was batching - it seems like nobody really tries todo larger batches the biggest models, and without batching but while having little data the model was getting stuck.
- MasterScrat 7y agoYou trained (finetuned) GPT2 for 1-2 months on 14mb of data? I don't understand how this doesn't massively overfit. How long of these 1-2 months was the model actually training?
- Tenoke 7y agoI train for maybe ~12 hours a day, some days, especially around Christmas I didn't. I also lost a lot of days when trying out different stuff or when the weights didn't save to drive before the Colab timed out. Having said that, I was training the full model with an accumulated batch size for a while so it was taking > 10min per step. I've also been using pretty low learning rates for most of the latter stages. Overall the model is currently at ~11k steps and the loss can actually go down further but after playing with different checkpoints last week, the best one didnt seem to be the newest one so I left it at that one.
- sillysaurusx 7y agoAI dungeon was trained on 50mb. You might be overfitting. Don't train for too long. You want to transfer knowledge, not replace knowledge.
- brokensegue 7y agoAre we talking about training from scratch or fine tuning?
- sillysaurusx 7y agoFine tuning. For training from scratch, you want a dataset of at least 20GB gathered from all corners of the internet. I think OpenAI used around 160GB. Even if you can't process that much data, merely having it available forces the model to learn a diverse variety of knowledge. The difficulty of training from scratch (and generating a quality model) vs the difficulty of fine tuning is like the difficulty of becoming fluent in emacs vs using notepad. It's doable, but quality results take focused effort. It's fun! Definitely within reach of lots of people who wouldn't normally consider themselves data scientists / ML engineers. (I'm one of 'em.)
- marmuel 7y agoYep, I can fully confirm. Best results are by far with smaller models (such as openai-gpt).
- gwern 7y agoI disagree. 1.5b is strictly superior to 117M or 345M on everything we've trained it on, from 20MB contemporary poetry on up. Assuming of course you don't screw it up or train too long. The only times we've concluded the smaller models were worth using is when the transfer learning was basically useless (eg for our ABC/MIDI models, there's hardly anything English-like in it, so no transfer, and no point in using 1.5b since it'll just overfit like 117M does, so might as well stick with the small model, since that lets us do things like use >25k context windows).
- gwern 7y agoFor example, look at OpenAI's latest paper on scaling Transformers "Scaling Laws for Neural Language Models" https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361 , Kaplan et al 2020: larger models are better, in the entire range they test up to billion-parameter models, in pretty much every way - they need hardly any additional data, achieve lower losses, they train faster, they can be parallelized better, and they're even more compute-efficient and sample-efficient (!).