6 ms·
This is great, but one thing I really hoped would come sooner is fast training on Metal. As things are, you can get an M1/M2 Ultra (~800 Gb/s memory bandwidth;
by int_19h 3y ago
This is great, but one thing I really hoped would come sooner is fast training on Metal. As things are, you can get an M1/M2 Ultra (~800 Gb/s memory bandwidth; for comparison, RTX 4090 is ~1050 Gb/s) Mac Studio with 128Gb RAM for ~$3500. For large model inference, this is already way more affordable than stacking GPUs while being "fast enough", but training solutions are basically non-existent. I do wonder why; it feels like a low-hanging fruit.
- sqreept 3y agoM1, M2, M3 still have very low number of GPU cores. Apple should release some better hardware to take advantage of their recently released MLX library.
- sbinnee 3y agoAt this moment it looks clear to me that Apple won’t go that way. It’s enough for them to focus on inference and actual application not the heavy training part. They have been probably training models on a cluster with non Apple silicon and make them available for their chips only for inference.
- ttul 3y agoNot to mention entirely outsourcing training workloads to specialist firms. Apple does a lot of secretive outsourcing of things you might think they would or should do in-house. This contrasts with Google and Meta who seem to like keeping everything in-house.
- kergonath 3y agoIt’s true that their GPUs are slower than Nvidia’s. But keep in mind that cores are really different and cannot be compared across architectures. You want more Gflops, not necessarily more cores.
- int_19h 3y agoThey do, but for inference at least, it's memory bandwidth that is the primary limiting factor for home LLMs right now, not raw compute.
- sroussey 3y agoWonder if the apple silicon ultra series will start using HBM3(e) on desktop in the future.
- buildbot 3y agoCompute limited - an m2 ultra has 27 tflops, a 4090 80+
- erichocean 3y agoMemory limited - an m2 ultra has >150GiB, a 4090 24GiB
- lawlessone 3y agoSo why is nobody doing this?
- ttul 3y agoMy personal experience with Apple Silicon and machine learning in comparison with Nvidia is that the libraries are often missing various features on Apple, leading to headaches when trying to use various libraries and tools. Apple is working to bridge the gap and I am excited for the gap to be closed because the memory bandwidth on big M2 and M3 machines is monstrous.
- lawlessone 3y agosounds similar to how people have described game dev for Mac. The hardware is there. It just isn't supported.
- imhoguy 3y agoApple could single-handedly kill consumer dGPU market if they have released proper low-level APIs for their M1/2/3. I feel they have some huge coming out in the pipe to tumble the "AI" market.
- deleted 3y ago[deleted]
- 3y ago