3 ms·

One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system me

One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. For example, currently most GPUs have a max of 16Gb of NVRAM, but usually 8GB or 12GB (except for more expensive workstation cards and similar [1]). But if you can spec a system with 32GB, 64GB, 96GB, or 128GB or more system memory with sufficient bandwidth to the CPU, then the accelerator can run a broader range of bigger models locally. Afaik Apple M2 Pro and M3 Pro/Max systems are the only ones with that capability right now. [1]:https://www.nvidia.com/en-us/design-visualization/desktop-graphics/ https://www.nvidia.com/en-us/design-visualization/desktop-gr...

DeathArrow 3y ago
>One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. Should Nvidia be afraid of Apple? Is OpenAI going to ditch their A100 for Macbook Airs?
- michaelt 3y ago
  > Should Nvidia be afraid of Apple? Depends on what you think about how technologies become successful. Some people would say the foundation of nvidia's current dominance is their mass-market gaming GPUs. While every other High Performance Computing technology wanted you to call their enterprise sales team and pay a big markup, any academic or open source developer could get their hands on an nvidia GPU - and that lead to a huge academic and open source ecosystem. In recent years, nvidia has decided if you want more than 24GB of vram, you've got to call their enterprise sales team and pay a big markup. And a lot of people working on LLMs want more than 24GB of vram. On the other hand, nvidia do have a lot of momentum. And of course cloud options are available these days.
- SkyMarshal 3y ago
  Not Apple since they’re not a player in datacenter AI, only local laptop/desktop AI. But AMD yes, since AMD is developing similar architecture APU systems that will roll out in 2024. Intel is also developing them, but unclear if their NPU and GPU performance will be comparable. https://www.amd.com/en/products/accelerators/instinct/mi300.html https://www.amd.com/en/products/accelerators/instinct/mi300.... https://www.techradar.com/computing/cpu/intel-launches-core-ultra-processors-to-power-up-your-next-laptop-with-ai https://www.techradar.com/computing/cpu/intel-launches-core-... There is also some tech that lets you put some model layers on the Nvidia GPU’s local NVRAM and the rest in system RAM. It’s slower though. Some discussion of that here: https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_model_to_locally_run_in_a_low_end_gpu_with_4/ https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_mo... https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are_you_people_running_70b_parameter_models/ https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are...

DeathArrow 3y ago

>One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. Should Nvidia be afraid of Apple? Is OpenAI going to ditch their A100 for Macbook Airs?

michaelt 3y ago
> Should Nvidia be afraid of Apple? Depends on what you think about how technologies become successful. Some people would say the foundation of nvidia's current dominance is their mass-market gaming GPUs. While every other High Performance Computing technology wanted you to call their enterprise sales team and pay a big markup, any academic or open source developer could get their hands on an nvidia GPU - and that lead to a huge academic and open source ecosystem. In recent years, nvidia has decided if you want more than 24GB of vram, you've got to call their enterprise sales team and pay a big markup. And a lot of people working on LLMs want more than 24GB of vram. On the other hand, nvidia do have a lot of momentum. And of course cloud options are available these days.
SkyMarshal 3y ago
Not Apple since they’re not a player in datacenter AI, only local laptop/desktop AI. But AMD yes, since AMD is developing similar architecture APU systems that will roll out in 2024. Intel is also developing them, but unclear if their NPU and GPU performance will be comparable. https://www.amd.com/en/products/accelerators/instinct/mi300.html https://www.amd.com/en/products/accelerators/instinct/mi300.... https://www.techradar.com/computing/cpu/intel-launches-core-ultra-processors-to-power-up-your-next-laptop-with-ai https://www.techradar.com/computing/cpu/intel-launches-core-... There is also some tech that lets you put some model layers on the Nvidia GPU’s local NVRAM and the rest in system RAM. It’s slower though. Some discussion of that here: https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_model_to_locally_run_in_a_low_end_gpu_with_4/ https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_mo... https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are_you_people_running_70b_parameter_models/ https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are...