4 ms·
To this day 8x7b Mixtral remains the best model you can run on a single 48GB GPU. This has the potential to become the best model you can run on two such GPUs,
by ein0p 3y ago
To this day 8x7b Mixtral remains the best model you can run on a single 48GB GPU. This has the potential to become the best model you can run on two such GPUs, or on an MBP with maxed out RAM, when 4-bit quantized.
- noman-land 3y agoMy first thought was how much RAM? Will it work on 64GB M1?
- ein0p 3y agoNope. Just the weights would take 88GB at 4 bit. 128GB MBP ought to be able to run it. If I were to guess, a version for Apple MLX should be available within a few days, for those of us fortunate enough to own such a thing.
- Art9681 3y agoIt’s already available. I had it running yesterday morning in an M3 MAX 128GB. I get about 6tps. https://www.reddit.com/r/LocalLLaMA/s/MSsrqWHYga https://www.reddit.com/r/LocalLLaMA/s/MSsrqWHYga
- jwitthuhn 3y agoIt is ~260GB with presumably fp16 weights. Should fit into 64GB at 3-bit quantization (~49GB). Edit: To add to this, I've had good luck getting solid output out of mixtral 8x7b at 3-bit, so that isn't small enough to completely kill the model's quality.
- deleted 3y ago[deleted]
- rspoerri 3y agoI hope i get it to run on my 96gb m2 in q4.
- ryao 3y agoI am looking forward to the pricing of those dropping. It is a shame that high memory graphics cards are not mainstream.