4 ms·
For finetuning GPT-2 on custom text, my gpt-2-simple package (https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/gpt-2-simple) gets close to
by minimaxir 7y ago
For finetuning GPT-2 on custom text, my gpt-2-simple package (https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/gpt-2-simple) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test.
Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text compared to the 345M which is slow even on a GPU.
How exactly the large GPT-2 models are deployed is a mystery I really wish was open-sourced more.
- gwern 7y agoI've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.
- p1esk 7y agoAre you using FP16?
- gwern 7y agoNo. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though. (Another option is model parallelism since I have 2 GPUs and that might be enough, perhaps freezing more layers and training incrementally, or reducing the 1024 token window to smaller ones like 700.)
- slashcom 7y agofp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.
- sdan 7y agoHave you fine tuned 774 successfully using a single GPU?
- p1esk 7y agoI recommend Nvidia Apex, it offers several ways to mix precision.
- The_rationalist 7y agoYou maybe should try tensorflow automatic mixed precision! https://github.com/zihangdai/xlnet/pull/200 https://github.com/zihangdai/xlnet/pull/200
- JonathanFly 7y agoOn the NVIDIA GPT- 2 implementation: >What would be the largest model one could train across 2x 2080Ti? >~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations https://twitter.com/TheRealRPuri/status/1161322580126470145 https://twitter.com/TheRealRPuri/status/1161322580126470145
- gwern 7y agoThey haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).
- JonathanFly 7y agoI haven't look into at all myself, but he also said: >We do provide training code that should work out of the box for gpt2 117M/345M https://twitter.com/TheRealRPuri/status/1161319745259393024 https://twitter.com/TheRealRPuri/status/1161319745259393024
- p1esk 7y agoIt would take forever (or $$$) to train even 117M model from scratch.
- JonathanFly 7y agoI read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.
- minimaxir 7y agoFigured. I'll make changes to allow sampling from the default model more easily.
- newhaus1994 7y agoI'm the lead researcher on the Middlebury Institute project looking at fine-tuning the bigger models, and I got OOM on 745M and 1.5B originally. I had to get an Azure instance with 24GB VRAM to handle it (using nshepperd's codebase). It works, but takes a while (~500 epochs takes 12 hours on a 100k word training dataset).
- gwern 7y agoOuch! So 11GB is nowhere close to being enough, then. I wonder if even switching to FP16 will be adequate?
- newhaus1994 7y agoMight be able to get 745M down to work on a single GPU. I'm definitely not using all 24GB, so fp16 might be able to get it down enough.
- sdan 7y agoHow would you use fp16 to get it to work on a single GPU? And if you did, what GPU should you use?
- the8472 7y agoPossibly a stupid question, but does AMD lift such restrictions on models with its unified memory, by allowing the GPU to "page out" chunks of vram to system ram?
- minimaxir 7y agoNo idea how modeling works on AMD. (most discussions are about NVidia/CUDA)
- p1esk 7y agoMy guess is it would be much slower, because GPU processor would wait for data. Compare bandwidth - system RAM to GPU memory (PCIe): 16GBps vs GPU memory to GPU processor: 900GBps.
- vimy 7y agoIsn't that a macOS specific feature?
- AdamDKing 7y ago>How exactly the large GPT-2 models are deployed is a mystery I really wish was open-sourced more. TalkToTransformer.com uses preemptible P4 GPUs on Google Kubernetes Engine. Changing the number of workers and automatically restarting them when they're preempted is easy with Kubernetes. To provide outputs incrementally rather than waiting for the entire sequence to be generated, I open a websocket to a a worker and have it do a few tokens at a time, sending the output back as it goes. GPT-2 tokens can end partway through a multi-byte character, so to make this work you need to send the raw UTF-8 bytes to the browser and then have it concatenate them _before_ decoding the string. While my workers can batch requests from multiple users, the modest increase in performance is probably not worth the complexity in most cases.
- jcims 7y agoAny thoughts on the larger model? Doesn't seem materially better than the last one. Maybe the fine tuning exercises will show the benefit?
- deleted 7y ago[deleted]