5 ms·
> a lot of new models require the higher-end 80GB A100 variants For training, yeah. But at least in some cases, inference can be optimized to use less memory.
by Const-me 4y ago
> a lot of new models require the higher-end 80GB A100 variants
For training, yeah. But at least in some cases, inference can be optimized to use less memory.
For their large Whisper model, OpenAI says they need 10GB of VRAM: https://github.com/openai/whisper#available-models-and-languages https://github.com/openai/whisper#available-models-and-langu...
My DirectCompute port of that uses slightly over 4GB of VRAM, see the last column in that table: https://github.com/Const-me/Whisper/blob/master/SampleClips/summary.tsv https://github.com/Const-me/Whisper/blob/master/SampleClips/... I haven’t actually optimized memory usage I only did the DirectCompute port, the memory savings were achieved by Georgi Gerganov in whisper.cpp project, which I ported to Windows.
- anentropic 4y agoapparently GPT-J needs ~25GB RAM min (for inference only), i.e. a 32GB card ...and GPT-3 is an order of magnitude bigger again https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-gpt-neo.html https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-gp... though TBH just making cards with more RAM to fit these kind of models seems easier than being at absolute cutting edge of performance I also wonder how this metric translates to something like Apple Silicon 'unified memory'... latest MBP have 32GB RAM, I wonder if anyone managed to run GPT-J there yet?
- anentropic 4y agoThere's a float16 version of GPT-J rewritten in C++ here that apparently fits into 16GB... https://github.com/ggerganov/ggml/tree/master/examples/gpt-j https://github.com/ggerganov/ggml/tree/master/examples/gpt-j it's CPU-only and runs inference at "about ~6 words per second" on an M1 MBP
- smoldesu 4y agoThat's for the max-size model (6B parameters). GPT-Neo has pruned models 125m and 1.3b that fit into 1.5gb and 6gb of memory respectively. These models are obviously nerfed, but you can run them on budget ARM hardware and get legible, paragraph-length responses in less than 10 seconds.
- flangola7 4y agoThere's a 20B model now
- smoldesu 4y agoThe 20B model was the original I believe, and the other models were all pruned from it.
- atylerrice 4y agoThis is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.
- mark_l_watson 4y agoThey have a link to the bottom for something similar, but with GPT-2. The Hugging Face docs have hints for using FP16, and avoiding double memory requirements for model and checkpoint data.
- atylerrice 4y agoYou can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)
- trifurcate 4y agoI am running GPT-J-6B inference on 13GB of VRAM. FP16 is a must at this scale. I've also loaded GPT-NeoX-20B using int8 quantization onto a single RTX 3090 before (24GB VRAM) however inference latency was quite high.
- andruby 4y agoActually, the MacBook Pro with M2 Max CPU can be configured with 96GB of ram. The Mac Studio with M1 Ultra can have up to 128GB of ram.
- anentropic 4y agoAh great, I hadn't seen that... seems like that should be plenty for some of these models!
- TOMDM 4y agoI've been running the stock implementation that OpenAI put out, I didn't know people had made so many improvements! Thank you for your work.