4 ms·

there is also some hint about performance/watt vs an NVIDIA A100 using BERT models: > The A100 draws 131W peak during the training run, the M1MAX GPU draws a m

by yayr 5y ago

there is also some hint about performance/watt vs an NVIDIA A100 using BERT models: > The A100 draws 131W peak during the training run, the M1MAX GPU draws a maximum of 15.4 W. The A100 runs an iteration at 7ms vs 104ms (with SHARK) and 196ms (with TF-Metal)

sudosysgen 5y ago
So the A100 is 75% more efficient despite being in a high power variation of a larger node. Seems Apple still haw a lot of work to do in the GPU/ML side of things.
- olliej 5y ago
  My reading implied that they weren't able to use the apple hardware acceleration for AI, which given the massive perf delta I suspect the A100 is using. If they weren't using the A100's ML hardware I'd be both amazed by Nvidia's engineers, and also unhappy as it would be unfair for the test to not use the features of the hardware (this is a common refrain in some benchmarking circles where you disable features to get "fairness" in comparisons). Honestly though I'm more concerned about the other aspects of apple's software: "The OSX Window Server crashes when all GPUs are used to the maximum"
  - noxa 5y ago
    Correct! We're currently compiling to SPIR-V and then going through MoltenVK/spirv-cross to get to Metal - and our SPIR-V needs quite a bit of work (currently tuned for Mali, which is a very different architecture). The biggest limiter though is as you note Apple doesn't have a way to use the fancy hardware instructions whereas our CUDA version is using the Tensor Cores. The good news (for us) is that this is all effectively -O1 today; there's still potential for 2-8x speedups over these numbers and that's not even factoring in the inaccessible HW features that maybe one day Apple will expose :crossed-fingers: :) // IREE dev
    - AceJohnny2 5y ago
      Any chance of using the Neural Engine (otherwise exposed by CoreML) ?
      - magic_at_nodai 5y ago
        unlikely since the interface from ANE is not public and it may change between hardware versions.
- yayr 5y ago
  yeah, you could also say the performance is only 57% in this particular test for the M1 vs A100. Still not bad for a laptop though with a first gen chip.
  - smoldesu 5y ago
    The M1 is (supposedly) specialized for ML calculations and on a decidedly denser node, though. I guess it's a testament to either CUDA's power or machine learning GPGPU optimization.
    - yayr 5y ago
      I like this blog to understand NVidia perf stats (search for "Normalized GPU deep learning performance"): https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/ https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni... It seems that their consumer GPUs also have a gap
    - simondotau 5y ago
      > The M1 is (supposedly) specialized for ML calculations It is, but they’re not using any of that. Only the M1 GPU at this time.
      - magic_at_nodai 5y ago
        Yeah the ANE and AMX on cpu are wrapped behind Accelerate Framework and CoreML. So you will have to use CoreML (which wasn't able to compile the latest TF BERT). ANE is also inference only. So if you want training you will have to use the GPU with Apple's Tensorflow-Metal. //part of nod.ai / SHARK team.

by yayr 5y ago

sudosysgen 5y ago

So the A100 is 75% more efficient despite being in a high power variation of a larger node. Seems Apple still haw a lot of work to do in the GPU/ML side of things.

olliej 5y ago
My reading implied that they weren't able to use the apple hardware acceleration for AI, which given the massive perf delta I suspect the A100 is using. If they weren't using the A100's ML hardware I'd be both amazed by Nvidia's engineers, and also unhappy as it would be unfair for the test to not use the features of the hardware (this is a common refrain in some benchmarking circles where you disable features to get "fairness" in comparisons). Honestly though I'm more concerned about the other aspects of apple's software: "The OSX Window Server crashes when all GPUs are used to the maximum"
- noxa 5y ago
  Correct! We're currently compiling to SPIR-V and then going through MoltenVK/spirv-cross to get to Metal - and our SPIR-V needs quite a bit of work (currently tuned for Mali, which is a very different architecture). The biggest limiter though is as you note Apple doesn't have a way to use the fancy hardware instructions whereas our CUDA version is using the Tensor Cores. The good news (for us) is that this is all effectively -O1 today; there's still potential for 2-8x speedups over these numbers and that's not even factoring in the inaccessible HW features that maybe one day Apple will expose :crossed-fingers: :) // IREE dev
  - AceJohnny2 5y ago
    Any chance of using the Neural Engine (otherwise exposed by CoreML) ?
    - magic_at_nodai 5y ago
      unlikely since the interface from ANE is not public and it may change between hardware versions.
yayr 5y ago
yeah, you could also say the performance is only 57% in this particular test for the M1 vs A100. Still not bad for a laptop though with a first gen chip.
- smoldesu 5y ago
  The M1 is (supposedly) specialized for ML calculations and on a decidedly denser node, though. I guess it's a testament to either CUDA's power or machine learning GPGPU optimization.
  - yayr 5y ago
    I like this blog to understand NVidia perf stats (search for "Normalized GPU deep learning performance"): https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/ https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni... It seems that their consumer GPUs also have a gap
  - simondotau 5y ago
    > The M1 is (supposedly) specialized for ML calculations It is, but they’re not using any of that. Only the M1 GPU at this time.
    - magic_at_nodai 5y ago
      Yeah the ANE and AMX on cpu are wrapped behind Accelerate Framework and CoreML. So you will have to use CoreML (which wasn't able to compile the latest TF BERT). ANE is also inference only. So if you want training you will have to use the GPU with Apple's Tensorflow-Metal. //part of nod.ai / SHARK team.