3 ms·

I think the exciting thing with Apple silicone and "AI" is inference, not training. Due to their unified memory, you can potentially have enormous local model i

I think the exciting thing with Apple silicone and "AI" is inference, not training. Due to their unified memory, you can potentially have enormous local model inference. That's potentially much more expensive on a PC with a GPU having its own memory. Apple have an opportunity, if they 2-4x the memory on the entry level devices (not beyond the realms of possibility), to make local inference a thing available to all.

microtonal 3y ago
Sure, but for inference it’s much easier to work in lower precision. Training with very low precision is also possible, but is often more tricky due to numerical stability. A lot of work is going on in 8-bit inference and even 4 bit inference. So, models that need 64 GB in FP32 can do with 16GB VRAM in FP8 or INT8, which is well within the realm of consumer NVIDIA cards. And the latest NVIDIA tensor cores will absolutely destroy Apple Silicon GPUs or the Neural Engine in 8 bit. So, I don’t think it’s really a strong argument. And as someone who is a Mac user and a ML practitioner, I’d be very happy if they started supporting eGPUs again. Apple Silicon has many strengths and the GPU core are fine for many ends, from games to graphics apps. But let’s not pretend that Apple is beating NVIDIA at their own game (yet). That day might come, but currently it only leads to disappointed users in ML forums who were hyped into thinking that their vanilla M2 MacBook Airs can almost compete with a 4090 in training a deep transformer model. (Yes, that happens.)
incrudible 3y ago
Multiple NVIDIA GPUs also have unified memory from an application perspective, one GPU has up to 80GBs right now. They also have a proper software stack. If NVIDIA saw any credible threat there, they could ramp that up, DRAM is not beholden to Apple.

3 ms·

I think the exciting thing with Apple silicone and "AI" is inference, not training. Due to their unified memory, you can potentially have enormous local model i

by samwillis 3y ago

microtonal 3y ago
Sure, but for inference it’s much easier to work in lower precision. Training with very low precision is also possible, but is often more tricky due to numerical stability. A lot of work is going on in 8-bit inference and even 4 bit inference. So, models that need 64 GB in FP32 can do with 16GB VRAM in FP8 or INT8, which is well within the realm of consumer NVIDIA cards. And the latest NVIDIA tensor cores will absolutely destroy Apple Silicon GPUs or the Neural Engine in 8 bit. So, I don’t think it’s really a strong argument. And as someone who is a Mac user and a ML practitioner, I’d be very happy if they started supporting eGPUs again. Apple Silicon has many strengths and the GPU core are fine for many ends, from games to graphics apps. But let’s not pretend that Apple is beating NVIDIA at their own game (yet). That day might come, but currently it only leads to disappointed users in ML forums who were hyped into thinking that their vanilla M2 MacBook Airs can almost compete with a 4090 in training a deep transformer model. (Yes, that happens.)
incrudible 3y ago
Multiple NVIDIA GPUs also have unified memory from an application perspective, one GPU has up to 80GBs right now. They also have a proper software stack. If NVIDIA saw any credible threat there, they could ramp that up, DRAM is not beholden to Apple.