4 ms·
I've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather th
by gwern 7y ago
I've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.
- p1esk 7y agoAre you using FP16?
- gwern 7y agoNo. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though. (Another option is model parallelism since I have 2 GPUs and that might be enough, perhaps freezing more layers and training incrementally, or reducing the 1024 token window to smaller ones like 700.)
- slashcom 7y agofp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.
- sdan 7y agoHave you fine tuned 774 successfully using a single GPU?
- p1esk 7y agoI recommend Nvidia Apex, it offers several ways to mix precision.
- The_rationalist 7y agoYou maybe should try tensorflow automatic mixed precision! https://github.com/zihangdai/xlnet/pull/200 https://github.com/zihangdai/xlnet/pull/200
- JonathanFly 7y agoOn the NVIDIA GPT- 2 implementation: >What would be the largest model one could train across 2x 2080Ti? >~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations https://twitter.com/TheRealRPuri/status/1161322580126470145 https://twitter.com/TheRealRPuri/status/1161322580126470145
- gwern 7y agoThey haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).
- JonathanFly 7y agoI haven't look into at all myself, but he also said: >We do provide training code that should work out of the box for gpt2 117M/345M https://twitter.com/TheRealRPuri/status/1161319745259393024 https://twitter.com/TheRealRPuri/status/1161319745259393024
- p1esk 7y agoIt would take forever (or $$$) to train even 117M model from scratch.
- JonathanFly 7y agoI read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.
- minimaxir 7y agoFigured. I'll make changes to allow sampling from the default model more easily.
- newhaus1994 7y agoI'm the lead researcher on the Middlebury Institute project looking at fine-tuning the bigger models, and I got OOM on 745M and 1.5B originally. I had to get an Azure instance with 24GB VRAM to handle it (using nshepperd's codebase). It works, but takes a while (~500 epochs takes 12 hours on a 100k word training dataset).
- gwern 7y agoOuch! So 11GB is nowhere close to being enough, then. I wonder if even switching to FP16 will be adequate?
- newhaus1994 7y agoMight be able to get 745M down to work on a single GPU. I'm definitely not using all 24GB, so fp16 might be able to get it down enough.
- sdan 7y agoHow would you use fp16 to get it to work on a single GPU? And if you did, what GPU should you use?