5 ms·

I think the author downplays the significance his work because it only applies to "small neural networks". There are a lot of use-cases that can benefit from th

I think the author downplays the significance his work because it only applies to "small neural networks". There are a lot of use-cases that can benefit from this type of optimizations. Discovering how to use an undocumented fast accelerator available on millions of devices is very valuable.

my123 4y ago
Apple did prefer to expose it through their own Accelerator.framework API however...
- capableweb 4y ago
  Of course they do, Apple like to remain as much in control as possible. If suddenly it becomes more efficient/faster to run ML/AI stuff on Asahi Linux on Mac hardware then with macOS, I'm sure they be embarrassed enough to take some sort of action. And I'm pretty sure that action will be towards the side of "closing things down" rather than "opening stuff up", as is tradition.
  - my123 4y ago
    Wrong answer. AMX is an unstable ISA that changes between product generations. That's why it's not publicly documented. Arm SME is the standardisation of the concept, but is not inmarket yet. https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture https://community.arm.com/arm-community-blogs/b/architecture...
- svantana 4y ago
  Has it been verified that they actually use these instructions in Accelerate.framework? I just benchmarked this on my 2019 intel i9 mbp, and got the following speeds for 128x128 matrices, 32 repeats: cblas_sgemm: 36 GFLOP/s vDSP_mmul: 41 GFLOP/s That's a pretty big deal if these functions are >30x faster on the M1...! edit: that seems to be verified in the tlkh.dev blog post above. Interestingly, I ran the same code on my bargain-basement 2020 iphone SE, and got 259GFLOP/s! These apple devices are pretty mindblowing.
  - microtonal 4y ago
    Has it been verified that they actually use these instructions in Accelerate.framework? Yes. Aside from benchmarks, you can easily verify this by profiling an application with Instruments and then inspecting the disassembly. However, it should be said that AMX does not scale linearly with the number of cores, but with the number of core clusters. So, on the M1 if you use Accelerate in two threads (rather than one), performance will barely improve, because the first thread can keep the AMX unit busy enough. However, e.g. the M1 Pro and M1 Max have two performance core clusters with AMX units in them. So matrix multiplication doubles roughly two times compared to the M1. Similarly, the M1 Ultra has fours performance core clusters, so matrix multiplication performance is roughly twice that of the M1 Pro/Max and four times that of the M1. Benchmarks: https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
- discord 4y ago
  Hey not related but you mentioned using kvm to run arm64 macOS on linux aarch64. I would like to give this a shot, but can't find a project for it. Would you mind sharing the deets?
mark_l_watson 4y ago
Apple has done a wonderful job making CoreML smoothly integrated with iOS, iWatchOS, iPadOS, and macOS development.
- brookst 4y ago
  I get the value of the common APIs, but as a developer how do you deal with the wide range of performance in different form factors and product generations? Is there some way to gracefully adapt the same models to a specific device’s capabilities?
  - londons_explore 4y ago
    There are a bunch of easy ways to scale neural nets. Quantization and distillation being the main approaches (or some combination of the two). Both typically require more training time, but not much more human-effort. You can normally expect to get way more than half the 'outcome' from a neural net with half the ram/compute/time/power budget. So neural nets scale 'down' pretty well.
- ur-whale 4y ago
  > Apple has done a wonderful job making CoreML Apple has done a wonderful job of further locking their user into the golden cage they call a platform.
  - gjsman-1000 4y ago
    Every platform is a golden cage in some respect. Ask any business who is stuck on ancient Win32 and even DOS applications, source code long gone. (Looking at you my local McDonalds, Menards, Tractor Supply)…
  - gcr 4y ago
    I think you're right and you're wrong, it's a bit more complicated. ML is one of the few applications that benefit from platform-specific optimizations, so if you need every ounce of performance, you have your choice of which walled garden you want to tether your application to. The "lock-in" comes from the specific capabilities of your special-purpose hardware, and for serious applications, you're already thinking hard about whether to design your entire implementation around Apple, NVidia, Google/TPU, or even Android devices. For big models, platform-specific needs influence every aspect of model design, including data/model sharding, quantization, training loops... For non-scientific applications, it's usual practice to train your model in platform-agnostic ways using PyTorch or Tensorflow or whatever and then deploy it to devices in platform-specific ways, whether that's XLA, CoreML, Edge TPU, Android NNAPI, TensorflowJS, or hell, custom-written GLSL shaders or whatever. We're just starting to see cross-platform frameworks that abstract model inference: TFLite, PyTorch Mobile, ONNX. To their credit, CoreML can act as a backend for any of these, so you don't even need to worry about your platform.
MuffinFlavored 4y ago
Not up to date on a lot of "AI"/"ML" things, why isn't this significant for medium/large neural networks as well?
- lostmsu 4y ago
  RTX 3090 theoretical matmul is 142 TFlops. E.g. about 100x of this.
  - johndough 4y ago
    The RTX 3090 has 35.58 TFlops FP32 performance, or 285.48 FP16 according to https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_30_series https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces... EDIT: I fell for NVIDIA's marketing. The dense FP16 performance is only half of 284.48, which is 142. Thanks to adgjlsfhk1 for the correction.
    - adgjlsfhk1 4y ago
      That 285 is listed as (2:1 sparse) which means it's only valid for matrices where 2 out of every 4 numbers are zero. For dense matrices it's half that.
      - bee_rider 4y ago
        Are 2:1 sparse matrices a common thing? It seems weird, like clearly that’s not sparse enough to want to use, like, sparse matrix “CSR” style storage or something, haha. I would just treat it as dense I guess.
        adgjlsfhk1 4y ago
        They aren't. As far as I can tell, Nvidia does this to be able to double the number of TFlops they put on their website. (this might be a little unfair, the real reason is that in ML it might be possible to train a NN such that your matrices have this structure, but I haven't seen anyone other than Nvidia use it)

my123 4y ago

Apple did prefer to expose it through their own Accelerator.framework API however...

capableweb 4y ago
Of course they do, Apple like to remain as much in control as possible. If suddenly it becomes more efficient/faster to run ML/AI stuff on Asahi Linux on Mac hardware then with macOS, I'm sure they be embarrassed enough to take some sort of action. And I'm pretty sure that action will be towards the side of "closing things down" rather than "opening stuff up", as is tradition.
- my123 4y ago
  Wrong answer. AMX is an unstable ISA that changes between product generations. That's why it's not publicly documented. Arm SME is the standardisation of the concept, but is not inmarket yet. https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture https://community.arm.com/arm-community-blogs/b/architecture...
svantana 4y ago
Has it been verified that they actually use these instructions in Accelerate.framework? I just benchmarked this on my 2019 intel i9 mbp, and got the following speeds for 128x128 matrices, 32 repeats: cblas_sgemm: 36 GFLOP/s vDSP_mmul: 41 GFLOP/s That's a pretty big deal if these functions are >30x faster on the M1...! edit: that seems to be verified in the tlkh.dev blog post above. Interestingly, I ran the same code on my bargain-basement 2020 iphone SE, and got 259GFLOP/s! These apple devices are pretty mindblowing.
- microtonal 4y ago
  Has it been verified that they actually use these instructions in Accelerate.framework? Yes. Aside from benchmarks, you can easily verify this by profiling an application with Instruments and then inspecting the disassembly. However, it should be said that AMX does not scale linearly with the number of cores, but with the number of core clusters. So, on the M1 if you use Accelerate in two threads (rather than one), performance will barely improve, because the first thread can keep the AMX unit busy enough. However, e.g. the M1 Pro and M1 Max have two performance core clusters with AMX units in them. So matrix multiplication doubles roughly two times compared to the M1. Similarly, the M1 Ultra has fours performance core clusters, so matrix multiplication performance is roughly twice that of the M1 Pro/Max and four times that of the M1. Benchmarks: https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
discord 4y ago
Hey not related but you mentioned using kvm to run arm64 macOS on linux aarch64. I would like to give this a shot, but can't find a project for it. Would you mind sharing the deets?

mark_l_watson 4y ago

Apple has done a wonderful job making CoreML smoothly integrated with iOS, iWatchOS, iPadOS, and macOS development.

brookst 4y ago
I get the value of the common APIs, but as a developer how do you deal with the wide range of performance in different form factors and product generations? Is there some way to gracefully adapt the same models to a specific device’s capabilities?
- londons_explore 4y ago
  There are a bunch of easy ways to scale neural nets. Quantization and distillation being the main approaches (or some combination of the two). Both typically require more training time, but not much more human-effort. You can normally expect to get way more than half the 'outcome' from a neural net with half the ram/compute/time/power budget. So neural nets scale 'down' pretty well.
ur-whale 4y ago
> Apple has done a wonderful job making CoreML Apple has done a wonderful job of further locking their user into the golden cage they call a platform.
- gjsman-1000 4y ago
  Every platform is a golden cage in some respect. Ask any business who is stuck on ancient Win32 and even DOS applications, source code long gone. (Looking at you my local McDonalds, Menards, Tractor Supply)…
- gcr 4y ago
  I think you're right and you're wrong, it's a bit more complicated. ML is one of the few applications that benefit from platform-specific optimizations, so if you need every ounce of performance, you have your choice of which walled garden you want to tether your application to. The "lock-in" comes from the specific capabilities of your special-purpose hardware, and for serious applications, you're already thinking hard about whether to design your entire implementation around Apple, NVidia, Google/TPU, or even Android devices. For big models, platform-specific needs influence every aspect of model design, including data/model sharding, quantization, training loops... For non-scientific applications, it's usual practice to train your model in platform-agnostic ways using PyTorch or Tensorflow or whatever and then deploy it to devices in platform-specific ways, whether that's XLA, CoreML, Edge TPU, Android NNAPI, TensorflowJS, or hell, custom-written GLSL shaders or whatever. We're just starting to see cross-platform frameworks that abstract model inference: TFLite, PyTorch Mobile, ONNX. To their credit, CoreML can act as a backend for any of these, so you don't even need to worry about your platform.

MuffinFlavored 4y ago

Not up to date on a lot of "AI"/"ML" things, why isn't this significant for medium/large neural networks as well?

lostmsu 4y ago
RTX 3090 theoretical matmul is 142 TFlops. E.g. about 100x of this.
- johndough 4y ago
  The RTX 3090 has 35.58 TFlops FP32 performance, or 285.48 FP16 according to https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_30_series https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces... EDIT: I fell for NVIDIA's marketing. The dense FP16 performance is only half of 284.48, which is 142. Thanks to adgjlsfhk1 for the correction.
  - adgjlsfhk1 4y ago
    That 285 is listed as (2:1 sparse) which means it's only valid for matrices where 2 out of every 4 numbers are zero. For dense matrices it's half that.
    - bee_rider 4y ago
      Are 2:1 sparse matrices a common thing? It seems weird, like clearly that’s not sparse enough to want to use, like, sparse matrix “CSR” style storage or something, haha. I would just treat it as dense I guess.
      - adgjlsfhk1 4y ago
        They aren't. As far as I can tell, Nvidia does this to be able to double the number of TFlops they put on their website. (this might be a little unfair, the real reason is that in ML it might be possible to train a NN such that your matrices have this structure, but I haven't seen anyone other than Nvidia use it)