3 ms·

UMA may turn out to be visionary. I really wonder if they saw the AI/ML trend or just lucked out. Either way, the apple silicon arch is looking very strong for

UMA may turn out to be visionary. I really wonder if they saw the AI/ML trend or just lucked out. Either way, the apple silicon arch is looking very strong for local AI. It’s a lot easier to beef up the NPU than to redo memory arch.

smoldesu 4y ago
I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. All of them are pretty weak for local training. But having reasonably powerful inferencing hardware isn't very hard at all, UMA doesn't seem very visionary to me in an era of MMAPed AI models.
- microtonal 4y ago
  I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. Apple Silicon AMX units provide the matrix multiplication performance of many core CPUs or faster at a fraction of the wattage. See eg. https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
  - smoldesu 4y ago
    Yes, and generic multicore ARM CPUs can run ARM's standard compute library regardless of their hardware: https://github.com/ARM-software/armnn https://github.com/ARM-software/armnn Plus, the benchmark you've linked to is comparing hardware accelerated inferencing to the notoriously crippled MKL execution. A more appropriate comparison would test Apple's AMX units against the Ryzen's AVX-optimized inferencing.
    - microtonal 4y ago
      Nope, the benchmarks are done by disabling the MKL AMD cripple (I did these benchmarks). It’s not faster with eg. AMD BLIS.
- brookst 4y ago
  The visionary part is having a computer with 64GB RAM that can be used either for ML or for traditional desktop purposes. It means fewer HW SKUs, which improve scale economy. And it means the same HW can be repurposed for different users, versus PCs where you have to replace CPU and/or GPU. For raw ML performance in a hyper-optimized system, UMA is not a big deal. For a company that needs to ship millions of units and estimate demand quarters in advance, it seems like a pretty big deal.
  - smoldesu 4y ago
    Is it very different from their previous desktop lineup? I'm under the impression that Intel Macs can also run ML models with acceleration.
    - brookst 4y ago
      Very different. Intel Macs had separate system RAM and video RAM, like PCs. Apple Silicon doesn't just share address space with memory mapping, it's literally all the same RAM, and it can be allocated to CPU or GPU. If you get a 96GB M2 Mac, it can be an 8GB system with 88GB high speed GPU memory, or a 95.5GB CPU system with a tiny bit of GPU memory. Apple's GPUs are slow today (compared to state of the art nvidia/etc), but if Apple upped the GPU horsepower, the system arch puts them far ahead of PC-based systems.
      - smoldesu 4y ago
        Yeah, that would be great in a world where model splitting didn't exist: https://huggingface.co/blog/accelerate-large-models https://huggingface.co/blog/accelerate-large-models
        brookst 3y ago
        That doesn't have any relevance to the efficiency and cost improvements of having the same very fast RAM connected to both CPU and GPU cores. I can't believe anyone is arguing that bifurcated memory systems are no big deal. Are you like an x86 arch enthusiast? I'm sure Intel is frantically working on UMA for x86/x64, if that makes it more palatable. Though they'll need on-die GPU, which might get interesting.

smoldesu 4y ago

I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. All of them are pretty weak for local training. But having reasonably powerful inferencing hardware isn't very hard at all, UMA doesn't seem very visionary to me in an era of MMAPed AI models.

microtonal 4y ago
I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. Apple Silicon AMX units provide the matrix multiplication performance of many core CPUs or faster at a fraction of the wattage. See eg. https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
- smoldesu 4y ago
  Yes, and generic multicore ARM CPUs can run ARM's standard compute library regardless of their hardware: https://github.com/ARM-software/armnn https://github.com/ARM-software/armnn Plus, the benchmark you've linked to is comparing hardware accelerated inferencing to the notoriously crippled MKL execution. A more appropriate comparison would test Apple's AMX units against the Ryzen's AVX-optimized inferencing.
  - microtonal 4y ago
    Nope, the benchmarks are done by disabling the MKL AMD cripple (I did these benchmarks). It’s not faster with eg. AMD BLIS.
brookst 4y ago
The visionary part is having a computer with 64GB RAM that can be used either for ML or for traditional desktop purposes. It means fewer HW SKUs, which improve scale economy. And it means the same HW can be repurposed for different users, versus PCs where you have to replace CPU and/or GPU. For raw ML performance in a hyper-optimized system, UMA is not a big deal. For a company that needs to ship millions of units and estimate demand quarters in advance, it seems like a pretty big deal.
- smoldesu 4y ago
  Is it very different from their previous desktop lineup? I'm under the impression that Intel Macs can also run ML models with acceleration.
  - brookst 4y ago
    Very different. Intel Macs had separate system RAM and video RAM, like PCs. Apple Silicon doesn't just share address space with memory mapping, it's literally all the same RAM, and it can be allocated to CPU or GPU. If you get a 96GB M2 Mac, it can be an 8GB system with 88GB high speed GPU memory, or a 95.5GB CPU system with a tiny bit of GPU memory. Apple's GPUs are slow today (compared to state of the art nvidia/etc), but if Apple upped the GPU horsepower, the system arch puts them far ahead of PC-based systems.
    - smoldesu 4y ago
      Yeah, that would be great in a world where model splitting didn't exist: https://huggingface.co/blog/accelerate-large-models https://huggingface.co/blog/accelerate-large-models
      - brookst 3y ago
        That doesn't have any relevance to the efficiency and cost improvements of having the same very fast RAM connected to both CPU and GPU cores. I can't believe anyone is arguing that bifurcated memory systems are no big deal. Are you like an x86 arch enthusiast? I'm sure Intel is frantically working on UMA for x86/x64, if that makes it more palatable. Though they'll need on-die GPU, which might get interesting.