4 ms·
I agree, GPUs can still generate more token per second per dollar, but what is new and great about the high-end M1 and M2 is Apple offering this much memory ban
by mrb 3y ago
I agree, GPUs can still generate more token per second per dollar, but what is new and great about the high-end M1 and M2 is Apple offering this much memory bandwidth on a general purpose CPU, thus immediately available to all software running on the CPU.
- boesboes 3y agoUnless you calculate power usage I’d bet
- datameta 3y agoI would wager the M2 is more energy efficient than a 4090.
- smoldesu 3y agoIt fully depends on the workload. The 4090 itself draws around 450w at load, and the M2 Ultra peaks around 300w. If your workload is >1.5x faster on Nvidia hardware, then it's per-prompt efficiency probably beats out the M2 Ultra.
- frontierkodiak 3y agoNot to mention that you can trivially set a 300w power cap on the 4090 and still get 75-80% of peak FLOPS.
- datameta 3y agoAll valid points, I think I looked at it too trivially and fired an opinion from the hip.
- kristianp 3y agoLooking at the prices, The 4090 is about 3/4 of the price of a base model M2 Ultra Mac Studio, which has 64GB RAM. With the rest of the PC to go with the graphics card, it's about 7/8s of the price of an Ultra. Then you have the compatibility. Do you want your software to be CUDA compatible or Metal compatible? If you're writing the software, maybe you want both! The budget option is to go with a used 3090, which still has greater memory bandwidth than the M2 Ultra. In terms of FP16 Flops on the GPU, you have: M2 Ultra, 27 TFLOPS RTX 3090, 35 TFLOPS RTX 4090, 82 TFLOPS https://www.cpu-monkey.com/en/igpu-apple_m2_ultra_76_core https://www.cpu-monkey.com/en/igpu-apple_m2_ultra_76_core https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622 https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622 https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889 https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889