3 ms·

I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. All

I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. All of them are pretty weak for local training. But having reasonably powerful inferencing hardware isn't very hard at all, UMA doesn't seem very visionary to me in an era of MMAPed AI models.

microtonal 3y ago
I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. Apple Silicon AMX units provide the matrix multiplication performance of many core CPUs or faster at a fraction of the wattage. See eg. https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
- smoldesu 3y ago
  Yes, and generic multicore ARM CPUs can run ARM's standard compute library regardless of their hardware: https://github.com/ARM-software/armnn https://github.com/ARM-software/armnn Plus, the benchmark you've linked to is comparing hardware accelerated inferencing to the notoriously crippled MKL execution. A more appropriate comparison would test Apple's AMX units against the Ryzen's AVX-optimized inferencing.
  - microtonal 3y ago
    Nope, the benchmarks are done by disabling the MKL AMD cripple (I did these benchmarks). It’s not faster with eg. AMD BLIS.
brookst 3y ago
The visionary part is having a computer with 64GB RAM that can be used either for ML or for traditional desktop purposes. It means fewer HW SKUs, which improve scale economy. And it means the same HW can be repurposed for different users, versus PCs where you have to replace CPU and/or GPU. For raw ML performance in a hyper-optimized system, UMA is not a big deal. For a company that needs to ship millions of units and estimate demand quarters in advance, it seems like a pretty big deal.
- smoldesu 3y ago
  Is it very different from their previous desktop lineup? I'm under the impression that Intel Macs can also run ML models with acceleration.
  - brookst 3y ago
    Very different. Intel Macs had separate system RAM and video RAM, like PCs. Apple Silicon doesn't just share address space with memory mapping, it's literally all the same RAM, and it can be allocated to CPU or GPU. If you get a 96GB M2 Mac, it can be an 8GB system with 88GB high speed GPU memory, or a 95.5GB CPU system with a tiny bit of GPU memory. Apple's GPUs are slow today (compared to state of the art nvidia/etc), but if Apple upped the GPU horsepower, the system arch puts them far ahead of PC-based systems.
    - smoldesu 3y ago
      Yeah, that would be great in a world where model splitting didn't exist: https://huggingface.co/blog/accelerate-large-models https://huggingface.co/blog/accelerate-large-models
      - brookst 3y ago
        That doesn't have any relevance to the efficiency and cost improvements of having the same very fast RAM connected to both CPU and GPU cores. I can't believe anyone is arguing that bifurcated memory systems are no big deal. Are you like an x86 arch enthusiast? I'm sure Intel is frantically working on UMA for x86/x64, if that makes it more palatable. Though they'll need on-die GPU, which might get interesting.
        smoldesu 3y ago
        I'm a computer enthusiast. I've got my M1 in a drawer in my kitchen, it's just not very useful for much unless I'm being paid to fix something on it. MacOS is a miserable mockery of itself nowadays and Apple Silicon is more trouble than it's worth, at least in my experience. As I'm working on AI stuff right now, I have to be a realist. I'm not going to go dig up my Mac Mini so my AI inferencing can run slower and take longer to set up. Nothing I do feels that much faster on my M1 Mini. It feels faster than my 2018 Macbook Pro, but so did my 2014 MBP... and my 2009 x201. Being told to install Colima for Docker with reasonable system temps was the last straw. It's just not worth the hoop-jumping, at least from where I stand. So... when a day comes where I need UMA for something, please let me know. As is, I'm not missing out on any performance uplift though. > I'm sure Intel is frantically working on UMA for x86/x64 Everyone has been working on it. AMD was heavily considering it in the original Ryzen spec iirc. x86 does have an impetus to put more of the system on a chip - there's no good reason for UMA to be forced on it yet. Especially at scale, the idea of consolidating address space does not work out. It works for home users, but so does PCI (as it has for the past... 2 decades). It's just marketing. It's a cool feature (they even gave it a Proper Apple Name) but I'm not hearing anybody clamor for unified memory to hit the datacenter or upend the gaming industry. It's another T2 Security Chip feature, a nicely-worded marketing blurb they can toss in a gradient bubble for their next WWDC keynote.

microtonal 3y ago

I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. Apple Silicon AMX units provide the matrix multiplication performance of many core CPUs or faster at a fraction of the wattage. See eg. https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads

smoldesu 3y ago
Yes, and generic multicore ARM CPUs can run ARM's standard compute library regardless of their hardware: https://github.com/ARM-software/armnn https://github.com/ARM-software/armnn Plus, the benchmark you've linked to is comparing hardware accelerated inferencing to the notoriously crippled MKL execution. A more appropriate comparison would test Apple's AMX units against the Ryzen's AVX-optimized inferencing.
- microtonal 3y ago
  Nope, the benchmarks are done by disabling the MKL AMD cripple (I did these benchmarks). It’s not faster with eg. AMD BLIS.

brookst 3y ago

The visionary part is having a computer with 64GB RAM that can be used either for ML or for traditional desktop purposes. It means fewer HW SKUs, which improve scale economy. And it means the same HW can be repurposed for different users, versus PCs where you have to replace CPU and/or GPU. For raw ML performance in a hyper-optimized system, UMA is not a big deal. For a company that needs to ship millions of units and estimate demand quarters in advance, it seems like a pretty big deal.

smoldesu 3y ago
Is it very different from their previous desktop lineup? I'm under the impression that Intel Macs can also run ML models with acceleration.
- brookst 3y ago
  Very different. Intel Macs had separate system RAM and video RAM, like PCs. Apple Silicon doesn't just share address space with memory mapping, it's literally all the same RAM, and it can be allocated to CPU or GPU. If you get a 96GB M2 Mac, it can be an 8GB system with 88GB high speed GPU memory, or a 95.5GB CPU system with a tiny bit of GPU memory. Apple's GPUs are slow today (compared to state of the art nvidia/etc), but if Apple upped the GPU horsepower, the system arch puts them far ahead of PC-based systems.
  - smoldesu 3y ago
    Yeah, that would be great in a world where model splitting didn't exist: https://huggingface.co/blog/accelerate-large-models https://huggingface.co/blog/accelerate-large-models
    - brookst 3y ago
      That doesn't have any relevance to the efficiency and cost improvements of having the same very fast RAM connected to both CPU and GPU cores. I can't believe anyone is arguing that bifurcated memory systems are no big deal. Are you like an x86 arch enthusiast? I'm sure Intel is frantically working on UMA for x86/x64, if that makes it more palatable. Though they'll need on-die GPU, which might get interesting.
      - smoldesu 3y ago
        I'm a computer enthusiast. I've got my M1 in a drawer in my kitchen, it's just not very useful for much unless I'm being paid to fix something on it. MacOS is a miserable mockery of itself nowadays and Apple Silicon is more trouble than it's worth, at least in my experience. As I'm working on AI stuff right now, I have to be a realist. I'm not going to go dig up my Mac Mini so my AI inferencing can run slower and take longer to set up. Nothing I do feels that much faster on my M1 Mini. It feels faster than my 2018 Macbook Pro, but so did my 2014 MBP... and my 2009 x201. Being told to install Colima for Docker with reasonable system temps was the last straw. It's just not worth the hoop-jumping, at least from where I stand. So... when a day comes where I need UMA for something, please let me know. As is, I'm not missing out on any performance uplift though. > I'm sure Intel is frantically working on UMA for x86/x64 Everyone has been working on it. AMD was heavily considering it in the original Ryzen spec iirc. x86 does have an impetus to put more of the system on a chip - there's no good reason for UMA to be forced on it yet. Especially at scale, the idea of consolidating address space does not work out. It works for home users, but so does PCI (as it has for the past... 2 decades). It's just marketing. It's a cool feature (they even gave it a Proper Apple Name) but I'm not hearing anybody clamor for unified memory to hit the datacenter or upend the gaming industry. It's another T2 Security Chip feature, a nicely-worded marketing blurb they can toss in a gradient bubble for their next WWDC keynote.