4 ms·
If you're opinion, what is the best model I can run on my M1 MBP with 64gb memory and 32 GPU cores?
by doctoboggan 4y ago
If you're opinion, what is the best model I can run on my M1 MBP with 64gb memory and 32 GPU cores?
- Vetch 4y agoFor practical tasks, I would like to say FlanT5 11B which is 45GB but my experience is if you're using huggingface the usual way, it can initially take up to 2x the memory of the model to load. GPT-JT was released recently and seems interesting but I haven't tried it. If you're focused on scientific domain and want to do Open book Q/A, summarization, keyword extraction etc. Galactica 6B parameter version might be worth checking out. If our main language is not English one of the mt0 models might be worth a try https://huggingface.co/bigscience/mt0-xl https://huggingface.co/bigscience/mt0-xl These models are distinguished by being able to follow relatively complex natural language instructions and examples without needing to be finetuned.
- zaptrem 4y agoI'm able to run a 22b parameter GPT-Neo model on my 24gb 3090 and can fit a 30b parameter OPT model when combining my 3090 and 12gb 3080
- mdda 4y agoCould you point to any resources online about how to do this? e.g. is this using 8-bit quantisation?