3 ms·
Somewhat odd sizing. 7B -> 57B is quite a jump...and crucially misses basically the entire spectrum of consumer GPUs wholesole.
by Havoc 2y ago
Somewhat odd sizing. 7B -> 57B is quite a jump...and crucially misses basically the entire spectrum of consumer GPUs wholesole.
- Terretta 2y agoIn general, 70B models run well on the 128GB MacBook Pros. LM Studio makes it easy to see which fit.
- anon373839 2y agoThe 57B is a mixture of experts, with only 14B parameters “active” at one time. So it has the memory footprint of a 57B but the compute footprint of a 14B. It won’t fit on consumer GPUs but it will fit on Apple Silicon if there’s enough system RAM, and will be pretty fast (like Mixtral 8x7B).