2 ms·

>One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system me

>One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. Should Nvidia be afraid of Apple? Is OpenAI going to ditch their A100 for Macbook Airs?

michaelt 3y ago
> Should Nvidia be afraid of Apple? Depends on what you think about how technologies become successful. Some people would say the foundation of nvidia's current dominance is their mass-market gaming GPUs. While every other High Performance Computing technology wanted you to call their enterprise sales team and pay a big markup, any academic or open source developer could get their hands on an nvidia GPU - and that lead to a huge academic and open source ecosystem. In recent years, nvidia has decided if you want more than 24GB of vram, you've got to call their enterprise sales team and pay a big markup. And a lot of people working on LLMs want more than 24GB of vram. On the other hand, nvidia do have a lot of momentum. And of course cloud options are available these days.
SkyMarshal 3y ago
Not Apple since they’re not a player in datacenter AI, only local laptop/desktop AI. But AMD yes, since AMD is developing similar architecture APU systems that will roll out in 2024. Intel is also developing them, but unclear if their NPU and GPU performance will be comparable. https://www.amd.com/en/products/accelerators/instinct/mi300.html https://www.amd.com/en/products/accelerators/instinct/mi300.... https://www.techradar.com/computing/cpu/intel-launches-core-ultra-processors-to-power-up-your-next-laptop-with-ai https://www.techradar.com/computing/cpu/intel-launches-core-... There is also some tech that lets you put some model layers on the Nvidia GPU’s local NVRAM and the rest in system RAM. It’s slower though. Some discussion of that here: https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_model_to_locally_run_in_a_low_end_gpu_with_4/ https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_mo... https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are_you_people_running_70b_parameter_models/ https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are...

michaelt 3y ago

> Should Nvidia be afraid of Apple? Depends on what you think about how technologies become successful. Some people would say the foundation of nvidia's current dominance is their mass-market gaming GPUs. While every other High Performance Computing technology wanted you to call their enterprise sales team and pay a big markup, any academic or open source developer could get their hands on an nvidia GPU - and that lead to a huge academic and open source ecosystem. In recent years, nvidia has decided if you want more than 24GB of vram, you've got to call their enterprise sales team and pay a big markup. And a lot of people working on LLMs want more than 24GB of vram. On the other hand, nvidia do have a lot of momentum. And of course cloud options are available these days.

SkyMarshal 3y ago

Not Apple since they’re not a player in datacenter AI, only local laptop/desktop AI. But AMD yes, since AMD is developing similar architecture APU systems that will roll out in 2024. Intel is also developing them, but unclear if their NPU and GPU performance will be comparable. https://www.amd.com/en/products/accelerators/instinct/mi300.html https://www.amd.com/en/products/accelerators/instinct/mi300.... https://www.techradar.com/computing/cpu/intel-launches-core-ultra-processors-to-power-up-your-next-laptop-with-ai https://www.techradar.com/computing/cpu/intel-launches-core-... There is also some tech that lets you put some model layers on the Nvidia GPU’s local NVRAM and the rest in system RAM. It’s slower though. Some discussion of that here: https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_model_to_locally_run_in_a_low_end_gpu_with_4/ https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_mo... https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are_you_people_running_70b_parameter_models/ https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are...