4 ms·
Yes, but I also care about "can I load this onto my home GPU?" where, if I need all experts for this to run, the answer is "no".
by stavros 3y ago
Yes, but I also care about "can I load this onto my home GPU?" where, if I need all experts for this to run, the answer is "no".
- MacsHeadroom 3y agoThe answer is yes if you have a 24GB GPU. Just wait for 4bit quantization. Or watch Tim Dettmers, who is releasing code to run Mixtral 8x7b in just 4GB of RAM.