3 ms·

> an nvidia gpu will still run circles arounnd this thing Not for loading up models larger than 32GB it wouldn't. (They exist! That's what the "full-detail mod

by derefr 5y ago

> an nvidia gpu will still run circles arounnd this thing Not for loading up models larger than 32GB it wouldn't. (They exist! That's what the "full-detail model of the starship Enterprise" thing in the keynote was about.) Remember that on any computer without unified memory, you can only load a scene the size of the GPU's VRAM. No matter how much main memory you have to swap against, no matter how many GPUs you throw at the problem, no magic wand is going to let you render a single tile of a single frame if it has more texture-memory as inputs than one of your GPUs has VRAM. Right now, consumer GPUs top out at 32GB of VRAM. The M1 Max has, in a sense, 64GB (minus OS baseline overhead) of VRAM for its GPU to use. Of course, there is "an nvidia gpu" that can bench more than the M1 Max: the Nvidia A100 Tensor Core GPU, with 80GB of VRAM... which costs $149,000. (And even then, I should point out that the leaked Mac Pro M1 variant is apparently 4x larger again — i.e. it's probably available in a configuration with 256GB of unified memory. That's getting close to "doing the training for GPT-3 — a 350GB model before optimization — on a single computer" territory.)

localhost 5y ago
No way you could train, but if they could squeeze a bit more RAM onto that machine, you could actually do inferencing using the full 175B parameter GPT-3 model (vs. one of its smaller, e.g., 13B parameter versions [1] - if I could get my hands on the parameters for that one I could run it on my MBP 14 in a couple of weeks!). The ML folks are finding ways to consume everything the HW folks can make and then some. [1] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165
shaklee3 5y ago
Terrible, terrible take. GPU ram is about 10x faster than ddr in practical usage, so your worloads can finish and a new batch can be transferred over pcie faster than the apple would finish the first pass.
ClumsyPilot 5y ago
"Right now, consumer GPUs top out at 32GB of VRAM. The M1 Max has, in a sense, 64GB (minus OS baseline overhead) of VRAM for its GPU to use." We've had AMD APU's for years, you can shove 256GB of RAM in there. But noone cares because a huge chunk of memory attached to a slow GPU is useless.
jjcon 5y ago
Memory != Speed You could throw a TB of memory in something and it won't get any faster or be of any use for 99.99% of use cases. Large ML architectures don't need more memory, they need distributed processing. Ignoring memory requirements, GPT-3 would take hundreds of years to train on a single high end GPU (on say a desktop 3090 which is >10x faster than m1) which is why they aren't trained that way (and why NVidia has the offerings set up the way they do). >That's getting close to "doing the training for GPT-3 — a 350GB model before optimization — on a single computer" territory. Not even close... not by a mile. That isn't how it works. The unified memory is cool but its utility is massively bottlenecked by the single cpu/gpu it is attached to.
- camillomiller 5y ago
  I think we’re losing the perspective here that Apple is not in the business of selling chips, but rather in the business of selling laptop to professional who would never even need what you describe
- derefr 5y ago
  I don't disagree that there are many use cases for which more memory has diminishing returns. But I would disagree that those encompass 99.99% of use cases. Not all problems are embarrassingly-parallel. In fact, most problems aren't embarrassingly parallel. It's just that we mostly use GPUs for embarrassingly-parallel problems, because that's mostly what they're good at, and humans aren't clever enough by half to come up with every possible way to map MIMD problems (e.g. graph search) into their SIMD equivalents (e.g. matrix multiplication, ala PageRank's eigenvector calculation.) The M1 Max isn't the absolute best GPU for doing the things GPUs already do well. But its GPU is a much better "connection machine" than e.g. the Xeon Phi ever was. It's a (weak) TPU in a laptop. (And likely the Mac Pro variant will be a true TPU.) Having a cheap, fast-ish GPU with that much memory, opens up use-cases for which current GPUs aren't suited. In those use-cases, this chip will "run circles around" current GPUs. (Mostly because current GPUs wouldn't be able to run those workloads at any speed.) Just one fun example of a use-case that has been obvious for years, yet has been mostly moot until now: there are database engines that run on GPUs. For parallelizable table-scan queries, they're ~100x faster still than even memory databases like memSQL. But guess where all the data needs to be loaded into, for those GPU DB engines to do their work? You'd never waste $150k on an A100 just to host an 80GB database. For that price, you could rent 100 regular servers and set them up as memSQL shards. But if you could get a GPU-parallel-scannable 64GB DB [without a memory-bandwidth bottleneck] for $4000? Now we're talking. For the cost of one A100, you get a cluster of ~37 64GB M1 Max MBPs — that's 2.3TB of addressable VRAM. That's enough to start doing real-time OLAP aggregations on some Big-Ish Data. (And that's with the ridiculous price overhead of paying for a whole laptop just to use its SoC. If integrators could buy these chips standalone, that'd probably knock the pricing down by another order of magnitude.)
deleted 5y ago
[deleted]

by derefr 5y ago

localhost 5y ago

No way you could train, but if they could squeeze a bit more RAM onto that machine, you could actually do inferencing using the full 175B parameter GPT-3 model (vs. one of its smaller, e.g., 13B parameter versions [1] - if I could get my hands on the parameters for that one I could run it on my MBP 14 in a couple of weeks!). The ML folks are finding ways to consume everything the HW folks can make and then some. [1] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165

shaklee3 5y ago

Terrible, terrible take. GPU ram is about 10x faster than ddr in practical usage, so your worloads can finish and a new batch can be transferred over pcie faster than the apple would finish the first pass.

ClumsyPilot 5y ago

"Right now, consumer GPUs top out at 32GB of VRAM. The M1 Max has, in a sense, 64GB (minus OS baseline overhead) of VRAM for its GPU to use." We've had AMD APU's for years, you can shove 256GB of RAM in there. But noone cares because a huge chunk of memory attached to a slow GPU is useless.

jjcon 5y ago

Memory != Speed You could throw a TB of memory in something and it won't get any faster or be of any use for 99.99% of use cases. Large ML architectures don't need more memory, they need distributed processing. Ignoring memory requirements, GPT-3 would take hundreds of years to train on a single high end GPU (on say a desktop 3090 which is >10x faster than m1) which is why they aren't trained that way (and why NVidia has the offerings set up the way they do). >That's getting close to "doing the training for GPT-3 — a 350GB model before optimization — on a single computer" territory. Not even close... not by a mile. That isn't how it works. The unified memory is cool but its utility is massively bottlenecked by the single cpu/gpu it is attached to.

camillomiller 5y ago
I think we’re losing the perspective here that Apple is not in the business of selling chips, but rather in the business of selling laptop to professional who would never even need what you describe
derefr 5y ago
I don't disagree that there are many use cases for which more memory has diminishing returns. But I would disagree that those encompass 99.99% of use cases. Not all problems are embarrassingly-parallel. In fact, most problems aren't embarrassingly parallel. It's just that we mostly use GPUs for embarrassingly-parallel problems, because that's mostly what they're good at, and humans aren't clever enough by half to come up with every possible way to map MIMD problems (e.g. graph search) into their SIMD equivalents (e.g. matrix multiplication, ala PageRank's eigenvector calculation.) The M1 Max isn't the absolute best GPU for doing the things GPUs already do well. But its GPU is a much better "connection machine" than e.g. the Xeon Phi ever was. It's a (weak) TPU in a laptop. (And likely the Mac Pro variant will be a true TPU.) Having a cheap, fast-ish GPU with that much memory, opens up use-cases for which current GPUs aren't suited. In those use-cases, this chip will "run circles around" current GPUs. (Mostly because current GPUs wouldn't be able to run those workloads at any speed.) Just one fun example of a use-case that has been obvious for years, yet has been mostly moot until now: there are database engines that run on GPUs. For parallelizable table-scan queries, they're ~100x faster still than even memory databases like memSQL. But guess where all the data needs to be loaded into, for those GPU DB engines to do their work? You'd never waste $150k on an A100 just to host an 80GB database. For that price, you could rent 100 regular servers and set them up as memSQL shards. But if you could get a GPU-parallel-scannable 64GB DB [without a memory-bandwidth bottleneck] for $4000? Now we're talking. For the cost of one A100, you get a cluster of ~37 64GB M1 Max MBPs — that's 2.3TB of addressable VRAM. That's enough to start doing real-time OLAP aggregations on some Big-Ish Data. (And that's with the ridiculous price overhead of paying for a whole laptop just to use its SoC. If integrators could buy these chips standalone, that'd probably knock the pricing down by another order of magnitude.)

deleted 5y ago

[deleted]