4 ms·
The neural engine in Apple chips is really for small models for doing things like Face ID, local inference for Photos.app, etc. It's there to run those small mo
by api 3y ago
The neural engine in Apple chips is really for small models for doing things like Face ID, local inference for Photos.app, etc. It's there to run those small models with very little power consumption. It's not that great for huge models. The GPU is better for those, and they have been adding cores and performance there.
But yes, most large models are pretty heavily memory bound.
- Tagbert 3y agoMemory throughput or memory capacity? Is that true for discrete GPUs like Nvidia? Is it the same for unified memory like the M-series?
- api 3y agoBoth really, since models are pretty much 100% "hot" RAM. The bigger the model the more bandwidth it demands too since evaluating it means streaming the entire thing through the processor.