5 ms·
Compute limited - an m2 ultra has 27 tflops, a 4090 80+
by buildbot 3y ago
Compute limited - an m2 ultra has 27 tflops, a 4090 80+
- erichocean 3y agoMemory limited - an m2 ultra has >150GiB, a 4090 24GiB
- lawlessone 3y agoSo why is nobody doing this?
- ttul 3y agoMy personal experience with Apple Silicon and machine learning in comparison with Nvidia is that the libraries are often missing various features on Apple, leading to headaches when trying to use various libraries and tools. Apple is working to bridge the gap and I am excited for the gap to be closed because the memory bandwidth on big M2 and M3 machines is monstrous.
- lawlessone 3y agosounds similar to how people have described game dev for Mac. The hardware is there. It just isn't supported.
- imhoguy 3y agoApple could single-handedly kill consumer dGPU market if they have released proper low-level APIs for their M1/2/3. I feel they have some huge coming out in the pipe to tumble the "AI" market.
- deleted 3y ago[deleted]
- yumraj 3y agoSo it should just take longer..
- AnthonyMouse 3y agoIf you don't care how long it takes you can get an old server with 128GB of RAM for a lot less than $3500.
- ErneX 3y agoBut that isn't GPU memory right? On the Mac it is.
- rthnbgrredf 3y agoThe issue here isn't specifically about the classification of memory, be it "unified memory," RAM, or VRAM. The primary concern is ensuring there's enough memory capacity for the models required for inference. The real question at hand is the Mac's value proposition in terms of inference speed, particularly for models as large as 70 billion parameters. Utilizing a 4090 GPU can facilitate real-time inference, which is the desired outcome for most users. In contrast, a Mac Studio offers close to real-time inference speeds, which might be disappointing for users expecting a real-time experience. Then, there's the option of CPU + RAM-based inference, which suits scenarios where immediate responses aren't crucial, allowing for batch processing of prompts and subsequent retrieval of responses. Considering the price points of both the Mac Studio and high-end GPUs are relatively comparable, it begs the question of the practicality and value of near real-time inference in specific use cases.
- abtinf 3y ago[flagged]
- rthnbgrredf 3y agoI'm not a gpt. But now you could say that this is exactly how a gpt would answer and we get stuck in a loop and there's no obvious way to prove that I'm not a gpt.
- lostmsu 3y ago4090 is actually 300+, so > 10 times faster. What would take 1 month on 2x4090 on M2 will take 2 years.