4 ms·
I am seriously considering switching some workloads from amd/linux to mac mini and osx at least until there is a better platform for cost-efficient ml inference
by tarasglek 3y ago
I am seriously considering switching some workloads from amd/linux to mac mini and osx at least until there is a better platform for cost-efficient ml inference for whisper and other large ml models where common cpu/gpu ram is cheap and convenient.
Amd should've had this market sealed with their integrated graphics platforms, maybe they will still get there with the new crosscompilers to Vulcan.
Don't need massive perf just 2x faster than cpu and better energy efficiency.
- smoldesu 3y agoCurrently the only option for cost-efficient ML inference is Linux. MacOS doesn't have CUDA drivers, which makes it pretty much useless for production. The Pytorch support on M1 is nice, but there's a reason VPS providers aren't throwing away their A100s for an M1 Ultra any time soon. The inferencing performance is different by an order of magnitude on the latest chips, and still a sizable lead when adjusting for power consumption. Not sure why you'd switch to an OS with weaker driver support to try and eke out better ML performance.