3 ms·
Not really? Inference on Mac GPUs is barely in its infancy. I have no idea about training... I have never even seen a metal llm training repo, but you can bet
by brucethemoose2 3y ago
Not really? Inference on Mac GPUs is barely in its infancy.
I have no idea about training... I have never even seen a metal llm training repo, but you can bet the new innovations/papers will not work ootb.
But setting the software (and OSX quirks) aside, the M2s would be fantastic... If they didn't cost a bazillion dollars for the higher memory configs.
- ftxbro 3y agoOK but I asked "Is this now the best hardware (or at least pareto frontier) for local LLMs?" and it sounds like you're mostly talking about software issues.
- brucethemoose2 3y agoHmmm... theoretically maybe? You can't argue with 192GB hooked up to a single GPU, even if the die is split. I'm not sure if it behaves more like a single GPU or 2 NVLink GPUs when running or training llms. But again, at that price point and with theoretical software support, you are starting to look at AMD MI300s, Intel GPU max/gaudi boxes, 4x 4090 boxes and such, so I am not sure.