3 ms·
This reads like an unfairly shallow dismissal. 4.4 tok/s is more than par for the course (even acknowledging that the model was heavily quantized and stripped d
by zozbot234 8mo ago
This reads like an unfairly shallow dismissal. 4.4 tok/s is more than par for the course (even acknowledging that the model was heavily quantized and stripped down by limiting active experts) for something that's streaming weights from storage in order to run on vastly more limited hardware than the model was designed for. It's even more impressive since the OP writeup mentions Apple Silicon hardware cannot overlap GPU compute with SSD access.
- ActorNightly 8mo agoThe problem is that the hardware is still like $3000. Making anything run on Macs is an exercise in futility. And its a shame that people get duped into buying Macs for LLM inference.
- zozbot234 8mo ago$3000 for running a 397B total parameters model is quite a bargain. The Mac is being used for its access to fast internal storage here since that's the key bottleneck, you could probably achieve similar outcomes with conventional (even fairly low-end) iGPU/APU hardware plus a fast PCIe x4 5.0 SSD (which would also allow you to overlap SSD transfers with iGPU/APU compute), but the cost would also be in a similar range. (Unless you carefully chose low-end e.g. Intel hardware with proper PCIe x4 5.0 NVMe support - which is still quite uncommon, especially for laptops.)
- ActorNightly 7mo agoIf you want to flex on being able to run 397b parameter models at unusably slow tokens/second sure. You can buy a 3090 for $2k, and run QWEN3.5 at 50+ token a second, and it will do everything you need, especially if you give it enough context.