4 ms·

Macs can't run CUDA but everyone is buying those for AI anyway.

• jszymborski 1y ago
  GP was likely referring to training, not inference.
• wolfgangK 1y ago
  Only those who don't care/know about prompt processing speed are buying Macs for LLM inference.
  • esseph 1y ago
    Don't know and don't care are definitely things that I could be, but it also makes sense if they want to keep lookups private.
  • com2kid 1y ago
    Even 40 tokens per second is plenty enough for real time usage. The average person reads at ~4 words per second, 40 tokens per second is going to be 15-20 words per second. Even useful models like gemma3 27b are hitting 22 t/s on 4bit quants. You aren't going to be reformatting gigabytes of PDFs or anything, but for a lot of common use cases, those speeds are fine.