4 ms·
lmao 4.4 tokens per second is hilariously and utterly bad. anyone suggesting that it's a reasonable speed should find another career
by NamlchakKhandro 7mo ago
lmao 4.4 tokens per second is hilariously and utterly bad.
anyone suggesting that it's a reasonable speed should find another career
- zozbot234 7mo agoThis reads like an unfairly shallow dismissal. 4.4 tok/s is more than par for the course (even acknowledging that the model was heavily quantized and stripped down by limiting active experts) for something that's streaming weights from storage in order to run on vastly more limited hardware than the model was designed for. It's even more impressive since the OP writeup mentions Apple Silicon hardware cannot overlap GPU compute with SSD access.
- ActorNightly 7mo agoThe problem is that the hardware is still like $3000. Making anything run on Macs is an exercise in futility. And its a shame that people get duped into buying Macs for LLM inference.
- zozbot234 7mo ago$3000 for running a 397B total parameters model is quite a bargain. The Mac is being used for its access to fast internal storage here since that's the key bottleneck, you could probably achieve similar outcomes with conventional (even fairly low-end) iGPU/APU hardware plus a fast PCIe x4 5.0 SSD (which would also allow you to overlap SSD transfers with iGPU/APU compute), but the cost would also be in a similar range. (Unless you carefully chose low-end e.g. Intel hardware with proper PCIe x4 5.0 NVMe support - which is still quite uncommon, especially for laptops.)
- ActorNightly 7mo agoIf you want to flex on being able to run 397b parameter models at unusably slow tokens/second sure. You can buy a 3090 for $2k, and run QWEN3.5 at 50+ token a second, and it will do everything you need, especially if you give it enough context.