3 ms·
Is this now the best hardware (or at least pareto frontier) for local LLMs?
by ftxbro 3y ago
Is this now the best hardware (or at least pareto frontier) for local LLMs?
- brucethemoose2 3y agoNot really? Inference on Mac GPUs is barely in its infancy. I have no idea about training... I have never even seen a metal llm training repo, but you can bet the new innovations/papers will not work ootb. But setting the software (and OSX quirks) aside, the M2s would be fantastic... If they didn't cost a bazillion dollars for the higher memory configs.
- ftxbro 3y agoOK but I asked "Is this now the best hardware (or at least pareto frontier) for local LLMs?" and it sounds like you're mostly talking about software issues.
- brucethemoose2 3y agoHmmm... theoretically maybe? You can't argue with 192GB hooked up to a single GPU, even if the die is split. I'm not sure if it behaves more like a single GPU or 2 NVLink GPUs when running or training llms. But again, at that price point and with theoretical software support, you are starting to look at AMD MI300s, Intel GPU max/gaudi boxes, 4x 4090 boxes and such, so I am not sure.
- kllrnohj 3y agoIf you need RAM more than you need compute maybe? The M1 Ultra was much slower than the 3090, and by all accounts the 4090 made a significantly bigger step forward in performance over the 3090 than the M2 Ultra does over the M1 Ultra.
- ls612 3y agoIf the M2 Ultra can double the performance of an M2 Max it will be on par with the 4070ti in compute, or slightly ahead of a 3090. But we'll have to wait for benchmarks to see.