4 ms·

> For example, M2 Ultra can train massive machine learning workloads in a single system that the most powerful discrete GPU can’t even process. It's interestin

> For example, M2 Ultra can train massive machine learning workloads in a single system that the most powerful discrete GPU can’t even process. It's interesting to see that Apple is explicitly targeting the M2 Ultra as an alternative to GPUs for machine learning. I think it is great for consumers for Nvidia to have a competitor in this space. If you wanted to do ML, you really didn't have much of an option other than Nvidia, which allowed Nvidia to do things like disable certain features in consumer GPUs even though the hardware was capable and limit RAM. Having Apple as a competitor in the consumer space will force them to up their game.

senko 3y ago
Yup! > It's interesting to see that Apple is explicitly targeting the M2 Ultra as an alternative to GPUs for machine learning. For many purposes the limiting factor on local LLMs is the amount of VRAM. nVidias with 64GB+ are insanely expensive (unless you're a funded startup). The ability of Apple Sillicon GPUs to use system RAM for the GPUs is a game-changer.
- nordsieck 3y ago
  > The ability of Apple Sillicon GPUs to use system RAM for the GPUs is a game-changer. It'd be a lot more of a game change if Apple didn't continue its tradition of charging extortionate prices for ram.
  - minhazm 3y ago
    While I agree it's expensive it's not quite apples to apples. GPU memory is very very expensive. A100 80GB costs $10k msrp (> $15k on ebay). An RTX 4090 only has 24gb of vram at $1600. Even if you do dual 3090's + nvlink for $1600 you only get up to 48GB, no where near the up to 192GB of vram you can access on apple silicon. The upgrade from 64GB to 192GB of ram is $1600. That's still expensive of course, but 128GB of 800GB/s memory that your GPU has access to for $1600 is actually not bad value. It's not like just sticking an extra 128gb of ram into your computer. DDR5-6400 dual channel will give up up to 102 GB/s, which is no where near M2 Ultra or GDDR6X.
giobox 3y ago
That unified memory architecture makes them well suited in many cases - that's the one bit current x86 desktop platforms don't have a good answer for, where the system design generally keeps GPU and system memory as entirely separate technologies. This is why fanless M2 MacBook Airs are regularly loading models 5k dollar PCs might choke on - you can get 24gb of unified memory, while almost all high end consumer GPUs top out at 16gb.
- nightski 3y ago
  High end consumer GPUs would be your 3090/4090 cards which both have 24GB.
  - giobox 3y ago
    4080 is still a 16gb part, and still the 2nd fastest part Nvidia sell at consumer level (its a 1200 dollar part). The 3090 (for however long it remains on sale) and the 4090 are outliers. The obvious counter point is an M2 MacBook in 14/16 inch size, where wallet permitting you can have up to 96gb of unified memory too... I simply chose the Air in my example earlier because its incredible you can have that level of performance in ostensibly an entry level passively cooled machine.
    - deleted 3y ago
      [deleted]
    - nightski 3y ago
      You don't have that level of performance though. Not even close really. RAM amount is only one piece of the puzzle.
      - deleted 3y ago
        [deleted]
      - deleted 3y ago
        [deleted]
    - 3y ago

senko 3y ago

Yup! > It's interesting to see that Apple is explicitly targeting the M2 Ultra as an alternative to GPUs for machine learning. For many purposes the limiting factor on local LLMs is the amount of VRAM. nVidias with 64GB+ are insanely expensive (unless you're a funded startup). The ability of Apple Sillicon GPUs to use system RAM for the GPUs is a game-changer.

nordsieck 3y ago
> The ability of Apple Sillicon GPUs to use system RAM for the GPUs is a game-changer. It'd be a lot more of a game change if Apple didn't continue its tradition of charging extortionate prices for ram.
- minhazm 3y ago
  While I agree it's expensive it's not quite apples to apples. GPU memory is very very expensive. A100 80GB costs $10k msrp (> $15k on ebay). An RTX 4090 only has 24gb of vram at $1600. Even if you do dual 3090's + nvlink for $1600 you only get up to 48GB, no where near the up to 192GB of vram you can access on apple silicon. The upgrade from 64GB to 192GB of ram is $1600. That's still expensive of course, but 128GB of 800GB/s memory that your GPU has access to for $1600 is actually not bad value. It's not like just sticking an extra 128gb of ram into your computer. DDR5-6400 dual channel will give up up to 102 GB/s, which is no where near M2 Ultra or GDDR6X.

giobox 3y ago

That unified memory architecture makes them well suited in many cases - that's the one bit current x86 desktop platforms don't have a good answer for, where the system design generally keeps GPU and system memory as entirely separate technologies. This is why fanless M2 MacBook Airs are regularly loading models 5k dollar PCs might choke on - you can get 24gb of unified memory, while almost all high end consumer GPUs top out at 16gb.

nightski 3y ago
High end consumer GPUs would be your 3090/4090 cards which both have 24GB.
- giobox 3y ago
  4080 is still a 16gb part, and still the 2nd fastest part Nvidia sell at consumer level (its a 1200 dollar part). The 3090 (for however long it remains on sale) and the 4090 are outliers. The obvious counter point is an M2 MacBook in 14/16 inch size, where wallet permitting you can have up to 96gb of unified memory too... I simply chose the Air in my example earlier because its incredible you can have that level of performance in ostensibly an entry level passively cooled machine.
  - deleted 3y ago
    [deleted]
  - nightski 3y ago
    You don't have that level of performance though. Not even close really. RAM amount is only one piece of the puzzle.
    - deleted 3y ago
      [deleted]
    - deleted 3y ago
      [deleted]
  - 3y ago