4 ms·
Reducing memory bandwidth is particularly bad given that most AI work loads are memory bound. With the memory bandwidth Apple's fully integrated GPU with share
by api 3y ago
Reducing memory bandwidth is particularly bad given that most AI work loads are memory bound.
With the memory bandwidth Apple's fully integrated GPU with shared memory is actually pretty good for smaller AI stuff like running and fine tuning llama2-sized models. It gives you a decently fast GPU option with a lot of RAM. NVIDIA cards with similar RAM are a decent fraction of the price of a whole Pro or Max M chip laptop.
- bloopernova 3y ago> most AI work loads are memory bound. Is that the reason why the number of neural cores stays the same, do you think?
- api 3y agoThe neural engine in Apple chips is really for small models for doing things like Face ID, local inference for Photos.app, etc. It's there to run those small models with very little power consumption. It's not that great for huge models. The GPU is better for those, and they have been adding cores and performance there. But yes, most large models are pretty heavily memory bound.
- Tagbert 3y agoMemory throughput or memory capacity? Is that true for discrete GPUs like Nvidia? Is it the same for unified memory like the M-series?
- api 3y agoBoth really, since models are pretty much 100% "hot" RAM. The bigger the model the more bandwidth it demands too since evaluating it means streaming the entire thing through the processor.