6 ms·

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way

by mkaic 4y ago

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete directly with Nvidia GPUs in the near future in terms of raw compute power in future generations of products like the Mac Studio and Mac Pro, which is very exciting. Competition in this space is incredibly good for consumers. 3.) At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory. With proper PyTorch support, we'll actually be able to use this memory for training big models or using big batch sizes. For the kind of DL work I do where dataloading is much more of a bottleneck than actual raw compute power, Mac Studio is now looking very enticing.

smoldesu 4y ago
There's definitely competition, and it's going to be really interesting to watch Nvidia and Apple duke it out over the next few years: - Apple undoubtedly owns the densest nodes, and will fight TSMC tooth-and-nail over first dibs on whatever silicon they have coming next. - Apple's current GPU design philosophy relies on horizontally scaling the tech they already use, whereas Nvidia has been scaling vertically, albeit slowly. - Nvidia has insane engineers. Despite the fact they're using silicon that's more than twice as large by-area when compared to Apple, they're still doubling their numbers across the board. And that's their last-gen tech too, the comparison once they're on 5nm later this summer is going to be insane. I expect things to be very heated by the end of this year, with new Nvidia, Intel and potentially new Apple GPUs.
my123 4y ago
> but they're already way ahead on energy efficiency 1) Nope. For neural network training not the case: https://tlkh.dev/benchmarking-the-apple-m1-max https://tlkh.dev/benchmarking-the-apple-m1-max And that's with the 3090 set at a very high 400W power limit, can get far more efficient when clocked lower. (which is normal, because no dedicated matrix math accelerators on the GPU notably) 2) We'll see, hopefully Apple thinks that the market is worth bothering with... (which would be great) 3) Indeed, if you need a giant pool of VRAM above everything else at a relatively low price tag, Apple is indeed a quite enticing option. If you can stand Metal for your use case of course.
- highfrequency 4y ago
  What do you mean by: "if you can stand Metal for your use case?" What is Metal?
  - rrss 4y ago
    Metal is apple’s api for writing software that uses their GPUs. https://en.m.wikipedia.org/wiki/Metal_(API) https://en.m.wikipedia.org/wiki/Metal_(API)
  - deleted 4y ago
    [deleted]
sudosysgen 4y ago
Apple Silicon is not ahead at all on energy efficiency for desktop workloads. If they were ahead on energy efficiency, they would simply be ahead on power. Indeed, GPUs are massively parallel architectures, and they are generally limited by the transistor and power budget (and memory, of course). Apple is simply behind in the GPU space. > At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory. With proper PyTorch support, we'll actually be able to use this memory for training big models or using big batch sizes. For the kind of DL work I do where dataloading is much more of a bottleneck than actual raw compute power, Mac Studio is now looking very enticing. The reason why it's cheaper is that its memory is at a fraction (around 20-35%) of the memory bandwidth of a 128GB equivalent GPU set up, which also has to be split with the CPU. This is an unavoidable bottleneck of shared memory systems, and for a great many applications this is a terminal performance bottleneck. That's the reason you don't have a GPU with 128GB of normal DDR5. It would just be quite limited. Perhaps for some cases it can be useful.
- p1esk 4y ago
  its memory is at a fraction (around 30-40%) of the memory bandwidth of a 128GB equivalent GPU setup Here's some info about M1 memory bandwidth: https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc...
  - sudosysgen 4y ago
    Yes. And the M1 Ultra has even more memory bandwidth than the M1 Max. But a 128 GB system made of 3 NVidia A6000 has 3x768Gb/s of memory bandwidth, a more common AI-grade card has 2x2Tb/s of memory bandwidth, which simply dwarfs the M1 Ultra.
    - matthew-wegner 4y ago
      For researchers, sure, but it's still quite an apples-to-oranges comparison. A6000 is ~$5k per card. I guess you're referring to something like an A100 on that other spec, which is $10k/card (for 40GB of memory). I do a fair bit of neural/AI art experimentation, where memory on the execution side is sometimes a limiting factor for me. I'm not training models, I'm not a hardcore researcher--those folks will absolutely be using NVIDIA's high-end stuff or TPU pods. 128GB in a Studio is super compelling if it means I can up-res some of my pieces without needing to use high-memory-but-super-slow CPU cloud VMs, or hope I get lucky with an A100 on Colab (or just pay for a GPU VM). I have a 128GB/Ultra Studio in my office now. It's a great piece of kit, and a big reason I splurged on it--okay, maybe "excuse"--was that I expect it'll be useful for a lot of my side project workloads over the next couple of years...
ActorNightly 4y ago
> but they're already way ahead on energy efficiency. For raw compute like you need for ML training, the M1s efficiency doesn't matter. Under the hood at hardware level, you have a direct mapping of power consumption to compute circuit activation that you really can't get around. The general efficiency of M1 is due its architecture and how it fits together with normal consumer use. Less stuff on the instruction decode, more efficient reordering, less energy wasted moving around data due to shared memory architecture, e.t.c
- ribit 4y ago
  And yet somehow Apples GPU ALUs are more efficient at 3.8 watts per TFLOP. Mind, I am not talking about specialized matrix multiplication units that have a different internal organization and can do things like matrix multiplication much more efficiently, but about basic general-purpose GPU ALUs. The comparison of efficiency between Apple and Nvidia here is a bit misleading because one compares Apples general-purpose ALUs to Nvidia’s specialized ALUs. For a more direct efficiency comparison, one would need to compare the Tensor Cores against the AMX or ANE coprocessors. As to how Apple achieves such high efficiency, nobody knows. The fact that they are on 5nm node might help, but there must be something special about the ALU design as well. My speculation is that they are wider and much more simpler than in other GPUs, which directly translates to efficiency wins.
  - blihp 4y ago
    How they do it at a conceptual level isn't a big secret: they don't need to minimize die area the way other companies do. For Apple, the die is just part of a chip that is part of the larger system they sell that they can amortize the cost over. nVidia doesn't have a system to do that with so their natural inclination is to lean towards keeping the die size as small as possible and just overclock the hell out of it. (right there is the 'trick': Apple can afford to do things that chew up die space that nVidia and others can't while maintaining their profit margins) Being a process generation ahead is also a rather huge thing too. (which is another thing they can amortize the cost of over large numbers of complete systems and mobile devices which their competitors can't) Also related: Apple designs their hardware to do just what they want it to while everyone else is designing for a more general use case. This also costs die area, IP licensing fees etc.
hedgehog 4y ago
To me the cool thing is working through a PyTorch-based course like FastAI on a local Mac may now be above the tolerably fast threshold.
mhh__ 4y ago
The thing is with the efficiency (which I'm not sure of) and the competition (probably possible) is that the current nvidia lineup is pretty old and on an even older process. They have a big moat.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
dekhn 4y ago
I remain skeptical that Apple's best GPU silicon will match nvidia's premiere products (either the top-end desktop card, or a server monster) for training. It seems like this is ideal as an accelerator for already trained models; one can imagine Photoshop utilizing it for deep-learning based infill-painting. I was doing training on battery with a laptop that had a 1080 and could do training; I have trained models on the airplane while totalyl unplugged and still had enough power to websurf afterwards.
fulafel 4y ago
> an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory Interesting observation. I wonder is the biggest memory iGPU configuration you can get on the x86 side?

by mkaic 4y ago

smoldesu 4y ago

There's definitely competition, and it's going to be really interesting to watch Nvidia and Apple duke it out over the next few years: - Apple undoubtedly owns the densest nodes, and will fight TSMC tooth-and-nail over first dibs on whatever silicon they have coming next. - Apple's current GPU design philosophy relies on horizontally scaling the tech they already use, whereas Nvidia has been scaling vertically, albeit slowly. - Nvidia has insane engineers. Despite the fact they're using silicon that's more than twice as large by-area when compared to Apple, they're still doubling their numbers across the board. And that's their last-gen tech too, the comparison once they're on 5nm later this summer is going to be insane. I expect things to be very heated by the end of this year, with new Nvidia, Intel and potentially new Apple GPUs.

my123 4y ago

> but they're already way ahead on energy efficiency 1) Nope. For neural network training not the case: https://tlkh.dev/benchmarking-the-apple-m1-max https://tlkh.dev/benchmarking-the-apple-m1-max And that's with the 3090 set at a very high 400W power limit, can get far more efficient when clocked lower. (which is normal, because no dedicated matrix math accelerators on the GPU notably) 2) We'll see, hopefully Apple thinks that the market is worth bothering with... (which would be great) 3) Indeed, if you need a giant pool of VRAM above everything else at a relatively low price tag, Apple is indeed a quite enticing option. If you can stand Metal for your use case of course.

highfrequency 4y ago
What do you mean by: "if you can stand Metal for your use case?" What is Metal?
- rrss 4y ago
  Metal is apple’s api for writing software that uses their GPUs. https://en.m.wikipedia.org/wiki/Metal_(API) https://en.m.wikipedia.org/wiki/Metal_(API)
- deleted 4y ago
  [deleted]

sudosysgen 4y ago

Apple Silicon is not ahead at all on energy efficiency for desktop workloads. If they were ahead on energy efficiency, they would simply be ahead on power. Indeed, GPUs are massively parallel architectures, and they are generally limited by the transistor and power budget (and memory, of course). Apple is simply behind in the GPU space. > At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory. With proper PyTorch support, we'll actually be able to use this memory for training big models or using big batch sizes. For the kind of DL work I do where dataloading is much more of a bottleneck than actual raw compute power, Mac Studio is now looking very enticing. The reason why it's cheaper is that its memory is at a fraction (around 20-35%) of the memory bandwidth of a 128GB equivalent GPU set up, which also has to be split with the CPU. This is an unavoidable bottleneck of shared memory systems, and for a great many applications this is a terminal performance bottleneck. That's the reason you don't have a GPU with 128GB of normal DDR5. It would just be quite limited. Perhaps for some cases it can be useful.

p1esk 4y ago
its memory is at a fraction (around 30-40%) of the memory bandwidth of a 128GB equivalent GPU setup Here's some info about M1 memory bandwidth: https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 https://www.anandtech.com/show/17024/apple-m1-max-performanc...
- sudosysgen 4y ago
  Yes. And the M1 Ultra has even more memory bandwidth than the M1 Max. But a 128 GB system made of 3 NVidia A6000 has 3x768Gb/s of memory bandwidth, a more common AI-grade card has 2x2Tb/s of memory bandwidth, which simply dwarfs the M1 Ultra.
  - matthew-wegner 4y ago
    For researchers, sure, but it's still quite an apples-to-oranges comparison. A6000 is ~$5k per card. I guess you're referring to something like an A100 on that other spec, which is $10k/card (for 40GB of memory). I do a fair bit of neural/AI art experimentation, where memory on the execution side is sometimes a limiting factor for me. I'm not training models, I'm not a hardcore researcher--those folks will absolutely be using NVIDIA's high-end stuff or TPU pods. 128GB in a Studio is super compelling if it means I can up-res some of my pieces without needing to use high-memory-but-super-slow CPU cloud VMs, or hope I get lucky with an A100 on Colab (or just pay for a GPU VM). I have a 128GB/Ultra Studio in my office now. It's a great piece of kit, and a big reason I splurged on it--okay, maybe "excuse"--was that I expect it'll be useful for a lot of my side project workloads over the next couple of years...

ActorNightly 4y ago

> but they're already way ahead on energy efficiency. For raw compute like you need for ML training, the M1s efficiency doesn't matter. Under the hood at hardware level, you have a direct mapping of power consumption to compute circuit activation that you really can't get around. The general efficiency of M1 is due its architecture and how it fits together with normal consumer use. Less stuff on the instruction decode, more efficient reordering, less energy wasted moving around data due to shared memory architecture, e.t.c

ribit 4y ago
And yet somehow Apples GPU ALUs are more efficient at 3.8 watts per TFLOP. Mind, I am not talking about specialized matrix multiplication units that have a different internal organization and can do things like matrix multiplication much more efficiently, but about basic general-purpose GPU ALUs. The comparison of efficiency between Apple and Nvidia here is a bit misleading because one compares Apples general-purpose ALUs to Nvidia’s specialized ALUs. For a more direct efficiency comparison, one would need to compare the Tensor Cores against the AMX or ANE coprocessors. As to how Apple achieves such high efficiency, nobody knows. The fact that they are on 5nm node might help, but there must be something special about the ALU design as well. My speculation is that they are wider and much more simpler than in other GPUs, which directly translates to efficiency wins.
- blihp 4y ago
  How they do it at a conceptual level isn't a big secret: they don't need to minimize die area the way other companies do. For Apple, the die is just part of a chip that is part of the larger system they sell that they can amortize the cost over. nVidia doesn't have a system to do that with so their natural inclination is to lean towards keeping the die size as small as possible and just overclock the hell out of it. (right there is the 'trick': Apple can afford to do things that chew up die space that nVidia and others can't while maintaining their profit margins) Being a process generation ahead is also a rather huge thing too. (which is another thing they can amortize the cost of over large numbers of complete systems and mobile devices which their competitors can't) Also related: Apple designs their hardware to do just what they want it to while everyone else is designing for a more general use case. This also costs die area, IP licensing fees etc.

hedgehog 4y ago

To me the cool thing is working through a PyTorch-based course like FastAI on a local Mac may now be above the tolerably fast threshold.

mhh__ 4y ago

The thing is with the efficiency (which I'm not sure of) and the competition (probably possible) is that the current nvidia lineup is pretty old and on an even older process. They have a big moat.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

dekhn 4y ago

I remain skeptical that Apple's best GPU silicon will match nvidia's premiere products (either the top-end desktop card, or a server monster) for training. It seems like this is ideal as an accelerator for already trained models; one can imagine Photoshop utilizing it for deep-learning based infill-painting. I was doing training on battery with a laptop that had a 1080 and could do training; I have trained models on the airplane while totalyl unplugged and still had enough power to websurf afterwards.

fulafel 4y ago

> an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory Interesting observation. I wonder is the biggest memory iGPU configuration you can get on the x86 side?