3 ms·

Anyone have comparisons to how many tokens/s a laptop with a mobile Nvidia gpu gets? Considering the cheapest M2 Max laptop in the US costs almost $3k, you can

by itg 3y ago

Anyone have comparisons to how many tokens/s a laptop with a mobile Nvidia gpu gets? Considering the cheapest M2 Max laptop in the US costs almost $3k, you can easily get a laptop with a 3090/4090 Nvidia mobile GPU.

smoldesu 3y ago
I like to use the OpenCL benchmarks as a rough point of comparison: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks M2 Max lands just around the 3060 mobile performance profile in this instance, it would be curious to see how the tokens/s reflect that.
- senttoschool 3y ago
  It's a little bit more nuanced than that. First, OpenCL is basically deprecated at this point and only Metal is up to date. Second, M2 Max might have a lot more RAM available than a 4090 because of its unified memory architecture. It can have up to 96GB vs 24GB for the 4090. So we need to look at different RAM configurations.
  - smoldesu 3y ago
    > OpenCL is basically deprecated at this point By Apple. > unified memory architecture Not really a bottleneck when layering over PCI exists. You're only constrained by PCI bandwidth and memory speeds, neither of which are really slow enough to meaningfully impact AI inferencing performance. Honestly, disk speeds are the #1 AI bottleneck I've seen on older systems. > It can have up to 96GB vs 24GB for the 4090 Good, at the M2 Max's price point I could almost afford 4x 3090s anyways. I'd only need one to beat it in inferencing performance though.
satysin 3y ago
Yes I would like another laptop to compare this against as by itself it is a bit meaningless to me. Also total power draw would be interesting to know as well. However I suspect the unified memory architecture of the M2 is a big benefit here and this M2 Max system is either a 64 or 96GB model? Even the very highest end Nvidia dGPU powered laptops won't have anywhere near that amount of GPU memory available and I don't know what the performance impact is with using system memory vs GPU memory on such systems?

by itg 3y ago

smoldesu 3y ago

I like to use the OpenCL benchmarks as a rough point of comparison: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks M2 Max lands just around the 3060 mobile performance profile in this instance, it would be curious to see how the tokens/s reflect that.

senttoschool 3y ago
It's a little bit more nuanced than that. First, OpenCL is basically deprecated at this point and only Metal is up to date. Second, M2 Max might have a lot more RAM available than a 4090 because of its unified memory architecture. It can have up to 96GB vs 24GB for the 4090. So we need to look at different RAM configurations.
- smoldesu 3y ago
  > OpenCL is basically deprecated at this point By Apple. > unified memory architecture Not really a bottleneck when layering over PCI exists. You're only constrained by PCI bandwidth and memory speeds, neither of which are really slow enough to meaningfully impact AI inferencing performance. Honestly, disk speeds are the #1 AI bottleneck I've seen on older systems. > It can have up to 96GB vs 24GB for the 4090 Good, at the M2 Max's price point I could almost afford 4x 3090s anyways. I'd only need one to beat it in inferencing performance though.

satysin 3y ago

Yes I would like another laptop to compare this against as by itself it is a bit meaningless to me. Also total power draw would be interesting to know as well. However I suspect the unified memory architecture of the M2 is a big benefit here and this M2 Max system is either a 64 or 96GB model? Even the very highest end Nvidia dGPU powered laptops won't have anywhere near that amount of GPU memory available and I don't know what the performance impact is with using system memory vs GPU memory on such systems?