3 ms·
> discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference At the point of purchase of the lowest cost configurat
by washadjeffmad 3y ago
> discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference
At the point of purchase of the lowest cost configuration with 24GB Unified Memory, you've already paid the an equivalent of over 2200 hours of GPU compute time on an RTX 4090 24GB, with a performance that exceeds the MacBook by around 1200% (it/s).
If you buy that MacBook for AI, you would have to run continuous generative inference on it for over a decade to match the return of just having used runpod instead. Even Apple doesn't use Macs to do AI - they use GCP TPUs. Buy the Mac if you like it, by all means, but be realistic.
LLM performance on Apple Silicon is decent for small models but does not scale. If any Mac architecture were cost effective at any scale for AI, we would be putting them in data centers, and we're not.
- adastra22 3y agoThis sub thread isn’t about hosted inference though. I interpreted the question as “is there a reasonable laptop out there that can run large models locally in real time?” And a MacBook Air does fit the bill for that. It’s an entry-level Mac, and it is capable of doing inference jobs. If you have around $1k available to spend on a daily driver laptop, but you also want to do some AI inference experiments, the MacBook Air is what I’d recommend. The AMD Framework laptop might be a good alternative. But I don’t know how well its integrated GPU is supported by llama.cpp.