6 ms·

Running PyTorch on the M1 GPU

92 points by my123 4y ago

machinekob 4y ago
MPS (Metal backend for PyTorch) is pretty poor integrated right now most of my custom models are crashing and pure Swift + MPSGraph versions is working 3-10x faster then PyTorch. So I'm pretty sure there is A LOT of optimizing and bug fixing before we can even consider PyTorch on apple devices (and this is ofc. 0.1 or smth like that so maybe we'll get something working fine in next few years). Also it'll be cool to compare Tensorflow vs PyTorch on metal device :)
- microtonal 4y ago
  MPS (Metal backend for PyTorch) is pretty poor integrated right now most of my custom models are crashing and pure Swift + MPSGraph versions is working 3-10x faster then PyTorch. So I'm pretty sure there is A LOT of optimizing and bug fixing before we can even consider PyTorch on apple devices Also, even for ops that do not crash, it often returns garbage. I built PyTorch from git this morning: >>> torch.arange(10, device="mps") tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='mps:0') >>> torch.ones(10, device="mps").type(torch.int32) tensor([1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216], device='mps:0', dtype=torch.int32) Many bugs will probably be squashed over the coming weeks, so I am still very exited about MPS support. I have done some preliminary benchmarks with a spaCy transformer model and the speedup was 2.55x on an M1 Pro. Which is quite nice because transformers were already very fast on M1 Macs, thanks to the AMX units (PyTorch links against Accelerate, so the AMX units are used for matrix multiplication). If they can squeeze more performance out of M1 GPUs in the future, it will be a very nice speedup.
  - machinekob 4y ago
    Weeks :? Looking at every other project from Apple that try to integrate with some opensource lib it'll be months/years if they didn't drop support for it before that :P Did you get only 2.55x on BERT/other transformer vs CPU version?
    - microtonal 4y ago
      I think in this case the PyTorch team also involved. They have already fixed some annoying bugs in the last two days (like matrix multiplication often failing because of buffer size mismatches). As for BERT inference performance, I am not sure what kind of speedups you are expecting. The M1 Pro gives me 2.6 TFLOPs in single precision matrix multiplication of 768x768 matrices. M1 Pro GPU performance is supposed to be 5.3 TFLOPS (not sure, I haven’t benchmarked it).
      - machinekob 4y ago
        Ahh nvm I was thinking about m1 max (my brain is damaged by the m1 naming and i didnt saw information that you are using m1 pro gpu)
        microtonal 4y ago
        Right, the Max should make a much bigger difference, since it has the same number of AMX units as the Pro, but double the GPU cores.
londons_explore 4y ago
What ML are people actually wanting to do on M1 laptops? I was under the impression that outside toy problems, you can't really get decent results unless you have a TPU or fleet of GPU's for training. If you just train on a laptop you're shooting yourself in the foot compared to someone who gets the same job done 10x faster in the cloud (usually with free credits). Obviously inference is still possible on a laptop for many models... but not many 'hacker' tasks involve only inference.
- shimmeringleaf 4y ago
  Totally, performance for actual useful training etc. is at least a couple of hardware generations away. This is super useful however for people like me who like to use an m1 Mac as their daily driver and don’t always want to have to fire up a remote rig just to sketch a little. Also being able to do some local inference can be great for when building a web GUI, etc. So absolutely not mature, but having PyTorch working better also on macs is only positive imho.
- andybak 4y ago
  The whole Diffusion text to image thing which is hugely popular right now doesn't involve any training. Same for text generation where you download pre-trained models. I'm using a laptop with a 3080 and getting much faster results than I typically get from a paid Colab account (not Pro mind you).
  - natly 4y ago
    Which repo/tool are you using for local text to image diffusion if you don't mind sharing? :)
    - andybak 4y ago
      Not at all. I discovered that Visions of Chaos (originally developed for fractals etc) has extensive support for installing and running ML models https://softology.pro/tutorials/tensorflow/tensorflow.htm https://softology.pro/tutorials/tensorflow/tensorflow.htm It handles (most) of the installation issues and gives you a GUI. You can however bypass the GUI. Every time you run a command the console shows you what the command line looks like so copy/paste into a terminal and away you go.
SemanticStrengh 4y ago
Does the M1 support openCL/SYCL on macos?
- my123 4y ago
  OpenCL 1.2* with very few extensions, pretty much stuck in the past. Don’t even think about using unified memory from there. SYCL: due to the above, not on the GPU. Just on the CPU. * x86_64 apps also get to see the OpenCL CPU backend in addition to the GPU one. However, arm64 apps only see the GPU.
  - SemanticStrengh 4y ago
    Apple is an irrelevant company at this point
    - pjmlp 4y ago
      Maybe for those that buy Apple as shinny UNIX instead of supporting Linux OEMs, for others there are plenty of market opportunities.
    - atty 4y ago
      On the contrary, I believe MacBooks have continued to eat general market share since they switched to Apple Silicon, and they are by far the best built UNIX-based laptop you can get, and have by far the best battery life. There’s a reason so many devs (and scientists, like myself) use them.
    - olliej 4y ago
      They're the fourth largest laptop vendor, making 9% market share, up from 7% the year prior [1] Apple is #2 world wide for phones at 18% [2] Apple is #1 for phones in the US with 51% share [3] Apple is #2 for phones in the EU at 23% share [4] Apple's laptop seem to wipe the floor in power usage, while not getting anyway more in gaming land than any prior generation [5] Apple's revenue is the 3rd highest in the world which also doesn't seem particularly irrelevant. You can hate Apple all you want, but going "I don't like apple, therefore the company is irrelevant" is a dumbass approach. Hell what will android get without finding out what's in iOS? [1] https://9to5mac.com/2022/04/11/mac-market-bucks-trend-with-continued-growth-while-pc-shipments-slow/ https://9to5mac.com/2022/04/11/mac-market-bucks-trend-with-c... [2] https://www.canalys.com/newsroom/global-smartphone-market-Q1-2022 https://www.canalys.com/newsroom/global-smartphone-market-Q1... [3] https://www.imore.com/apple-dominates-us-smartphone-market-51-q1-share-thanks-iphone-13 https://www.imore.com/apple-dominates-us-smartphone-market-5... (This is iMore so take with a grain of salt) [4] https://www.appleworld.today/2022/02/28/apples-iphone-has-23-of-the-smartphone-market-in-europe/ https://www.appleworld.today/2022/02/28/apples-iphone-has-23... [5] https://www.pcmag.com/news/intel-core-i9-vs-apple-m1-max-which-laptop-cpu-is-better https://www.pcmag.com/news/intel-core-i9-vs-apple-m1-max-whi...
fartcannon 4y ago
Too bad there's a requirement for metal. I had tensorflow running on a raspberry pi 3 a few years ago doing inference for a chat bot.
- Raymonf 4y ago
  Why wouldn't GPU support for the M1 GPU require Metal? You should be able to run PyTorch on the CPU of a Pi?
  - fartcannon 4y ago
    Metal is an unnecessary complication.
    - Raymonf 4y ago
      Then what API are you going to use on macOS?
      - fartcannon 4y ago
        You're not permitted to use anything else, thus: too bad it has a requirement for metal.
    - olliej 4y ago
      How so? NVIDIA GPUs have their own APIs, AMD does, Direct3D is its own. The Apple platform acceleration APIs are Metal. The MS acceleration APIs are all generally Direct3D, etc. The fact the you want Apple to support other platforms doesn't mean that they're wrong for not doing so. They same arguments to require Apple to support your preferred API also applies to MS, NVIDIA, AMD, etc and the same arguments to not do that apply them all as well.
      - fartcannon 4y ago
        That other companies use shitty proprietary code to lock out competition doesn't make Apples attempt to do the same any better. It compounds the error and wasted effort. In fact Apple's is the worst of the lot you mention because at least you can use alternatives with the others.
neogodless 4y ago
Related post: https://news.ycombinator.com/item?id=31424048 https://news.ycombinator.com/item?id=31424048 Accelerated PyTorch Training on M1 Mac (pytorch.org) 436 points by tgymnich 2 days ago 143 comments
deleted 4y ago
[deleted]
lvl102 4y ago
Can someone explain to me why M1 Ultra doesn’t scale ratably compared to RTX? It seems it should perform at about 1/2 of 3090. Is it because Apple locked down what you can do with the GPU?
- brrrrrm 4y ago
  If the tensor cores are used, the 3090 can hit 300Tflops. I don’t think the M1 can even come close to that (21Tflops for the ultra)
  - lvl102 4y ago
    Makes sense so it’s basically 7x more just in terms of raw power. I was wrongly under the impression that Apple was limiting ML capacity.
- Iwan-Zotow 4y ago
  1/2? 1/2?!? make it at least 1/10
krstn 4y ago
Nice, does not look super fast, but a great start and looks very usable for development or smaller tasks (a great start especially considering the missing nvidia GPUs on intel macbooks)

92 points by my123 4y ago

MPS (Metal backend for PyTorch) is pretty poor integrated right now most of my custom models are crashing and pure Swift + MPSGraph versions is working 3-10x faster then PyTorch. So I'm pretty sure there is A LOT of optimizing and bug fixing before we can even consider PyTorch on apple devices (and this is ofc. 0.1 or smth like that so maybe we'll get something working fine in next few years). Also it'll be cool to compare Tensorflow vs PyTorch on metal device :)

microtonal 4y ago
MPS (Metal backend for PyTorch) is pretty poor integrated right now most of my custom models are crashing and pure Swift + MPSGraph versions is working 3-10x faster then PyTorch. So I'm pretty sure there is A LOT of optimizing and bug fixing before we can even consider PyTorch on apple devices Also, even for ops that do not crash, it often returns garbage. I built PyTorch from git this morning: >>> torch.arange(10, device="mps") tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='mps:0') >>> torch.ones(10, device="mps").type(torch.int32) tensor([1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216], device='mps:0', dtype=torch.int32) Many bugs will probably be squashed over the coming weeks, so I am still very exited about MPS support. I have done some preliminary benchmarks with a spaCy transformer model and the speedup was 2.55x on an M1 Pro. Which is quite nice because transformers were already very fast on M1 Macs, thanks to the AMX units (PyTorch links against Accelerate, so the AMX units are used for matrix multiplication). If they can squeeze more performance out of M1 GPUs in the future, it will be a very nice speedup.
- machinekob 4y ago
  Weeks :? Looking at every other project from Apple that try to integrate with some opensource lib it'll be months/years if they didn't drop support for it before that :P Did you get only 2.55x on BERT/other transformer vs CPU version?
  - microtonal 4y ago
    I think in this case the PyTorch team also involved. They have already fixed some annoying bugs in the last two days (like matrix multiplication often failing because of buffer size mismatches). As for BERT inference performance, I am not sure what kind of speedups you are expecting. The M1 Pro gives me 2.6 TFLOPs in single precision matrix multiplication of 768x768 matrices. M1 Pro GPU performance is supposed to be 5.3 TFLOPS (not sure, I haven’t benchmarked it).
    - machinekob 4y ago
      Ahh nvm I was thinking about m1 max (my brain is damaged by the m1 naming and i didnt saw information that you are using m1 pro gpu)
      - microtonal 4y ago
        Right, the Max should make a much bigger difference, since it has the same number of AMX units as the Pro, but double the GPU cores.

londons_explore 4y ago

What ML are people actually wanting to do on M1 laptops? I was under the impression that outside toy problems, you can't really get decent results unless you have a TPU or fleet of GPU's for training. If you just train on a laptop you're shooting yourself in the foot compared to someone who gets the same job done 10x faster in the cloud (usually with free credits). Obviously inference is still possible on a laptop for many models... but not many 'hacker' tasks involve only inference.

shimmeringleaf 4y ago
Totally, performance for actual useful training etc. is at least a couple of hardware generations away. This is super useful however for people like me who like to use an m1 Mac as their daily driver and don’t always want to have to fire up a remote rig just to sketch a little. Also being able to do some local inference can be great for when building a web GUI, etc. So absolutely not mature, but having PyTorch working better also on macs is only positive imho.
andybak 4y ago
The whole Diffusion text to image thing which is hugely popular right now doesn't involve any training. Same for text generation where you download pre-trained models. I'm using a laptop with a 3080 and getting much faster results than I typically get from a paid Colab account (not Pro mind you).
- natly 4y ago
  Which repo/tool are you using for local text to image diffusion if you don't mind sharing? :)
  - andybak 4y ago
    Not at all. I discovered that Visions of Chaos (originally developed for fractals etc) has extensive support for installing and running ML models https://softology.pro/tutorials/tensorflow/tensorflow.htm https://softology.pro/tutorials/tensorflow/tensorflow.htm It handles (most) of the installation issues and gives you a GUI. You can however bypass the GUI. Every time you run a command the console shows you what the command line looks like so copy/paste into a terminal and away you go.

SemanticStrengh 4y ago

Does the M1 support openCL/SYCL on macos?

my123 4y ago
OpenCL 1.2* with very few extensions, pretty much stuck in the past. Don’t even think about using unified memory from there. SYCL: due to the above, not on the GPU. Just on the CPU. * x86_64 apps also get to see the OpenCL CPU backend in addition to the GPU one. However, arm64 apps only see the GPU.
- SemanticStrengh 4y ago
  Apple is an irrelevant company at this point
  - pjmlp 4y ago
    Maybe for those that buy Apple as shinny UNIX instead of supporting Linux OEMs, for others there are plenty of market opportunities.
  - atty 4y ago
    On the contrary, I believe MacBooks have continued to eat general market share since they switched to Apple Silicon, and they are by far the best built UNIX-based laptop you can get, and have by far the best battery life. There’s a reason so many devs (and scientists, like myself) use them.
  - olliej 4y ago
    They're the fourth largest laptop vendor, making 9% market share, up from 7% the year prior [1] Apple is #2 world wide for phones at 18% [2] Apple is #1 for phones in the US with 51% share [3] Apple is #2 for phones in the EU at 23% share [4] Apple's laptop seem to wipe the floor in power usage, while not getting anyway more in gaming land than any prior generation [5] Apple's revenue is the 3rd highest in the world which also doesn't seem particularly irrelevant. You can hate Apple all you want, but going "I don't like apple, therefore the company is irrelevant" is a dumbass approach. Hell what will android get without finding out what's in iOS? [1] https://9to5mac.com/2022/04/11/mac-market-bucks-trend-with-continued-growth-while-pc-shipments-slow/ https://9to5mac.com/2022/04/11/mac-market-bucks-trend-with-c... [2] https://www.canalys.com/newsroom/global-smartphone-market-Q1-2022 https://www.canalys.com/newsroom/global-smartphone-market-Q1... [3] https://www.imore.com/apple-dominates-us-smartphone-market-51-q1-share-thanks-iphone-13 https://www.imore.com/apple-dominates-us-smartphone-market-5... (This is iMore so take with a grain of salt) [4] https://www.appleworld.today/2022/02/28/apples-iphone-has-23-of-the-smartphone-market-in-europe/ https://www.appleworld.today/2022/02/28/apples-iphone-has-23... [5] https://www.pcmag.com/news/intel-core-i9-vs-apple-m1-max-which-laptop-cpu-is-better https://www.pcmag.com/news/intel-core-i9-vs-apple-m1-max-whi...

fartcannon 4y ago

Too bad there's a requirement for metal. I had tensorflow running on a raspberry pi 3 a few years ago doing inference for a chat bot.

Raymonf 4y ago
Why wouldn't GPU support for the M1 GPU require Metal? You should be able to run PyTorch on the CPU of a Pi?
- fartcannon 4y ago
  Metal is an unnecessary complication.
  - Raymonf 4y ago
    Then what API are you going to use on macOS?
    - fartcannon 4y ago
      You're not permitted to use anything else, thus: too bad it has a requirement for metal.
  - olliej 4y ago
    How so? NVIDIA GPUs have their own APIs, AMD does, Direct3D is its own. The Apple platform acceleration APIs are Metal. The MS acceleration APIs are all generally Direct3D, etc. The fact the you want Apple to support other platforms doesn't mean that they're wrong for not doing so. They same arguments to require Apple to support your preferred API also applies to MS, NVIDIA, AMD, etc and the same arguments to not do that apply them all as well.
    - fartcannon 4y ago
      That other companies use shitty proprietary code to lock out competition doesn't make Apples attempt to do the same any better. It compounds the error and wasted effort. In fact Apple's is the worst of the lot you mention because at least you can use alternatives with the others.

neogodless 4y ago

Related post: https://news.ycombinator.com/item?id=31424048 https://news.ycombinator.com/item?id=31424048 Accelerated PyTorch Training on M1 Mac (pytorch.org) 436 points by tgymnich 2 days ago 143 comments

deleted 4y ago

[deleted]

lvl102 4y ago

Can someone explain to me why M1 Ultra doesn’t scale ratably compared to RTX? It seems it should perform at about 1/2 of 3090. Is it because Apple locked down what you can do with the GPU?

brrrrrm 4y ago
If the tensor cores are used, the 3090 can hit 300Tflops. I don’t think the M1 can even come close to that (21Tflops for the ultra)
- lvl102 4y ago
  Makes sense so it’s basically 7x more just in terms of raw power. I was wrongly under the impression that Apple was limiting ML capacity.
Iwan-Zotow 4y ago
1/2? 1/2?!? make it at least 1/10

krstn 4y ago

Nice, does not look super fast, but a great start and looks very usable for development or smaller tasks (a great start especially considering the missing nvidia GPUs on intel macbooks)