11 ms·

The Secret Apple M1 Coprocessor

161 points by socialdemocrat 6y ago

mackman 6y ago
https://archive.is/eeksi https://archive.is/eeksi
- mackman 6y ago
  This article links to this more detailed post which I’d recommend: https://nod.ai/comparing-apple-m1-with-amx2-m1-with-neon/ https://nod.ai/comparing-apple-m1-with-amx2-m1-with-neon/
- sgt 6y ago
  Thanks! @dang - can we not make this an automatic HN feature? That is, automatically linking to archive.is for all stories from medium.com, NY Times, etc.
  - kube-system 6y ago
    I hope not. archive.is doesn’t resolve on all DNS providers.
    - sgt 6y ago
      You mean it is currently being censored by DNS providers and/or ISP's?
      - kube-system 6y ago
        I’m not sure that’s the case. https://community.cloudflare.com/t/archive-is-error-1001/18227 https://community.cloudflare.com/t/archive-is-error-1001/182...
h0l0cube 6y ago
> However more importantly, this is an advantage to Apple. By only letting their libraries use these special instructions Apple retains the freedom to radically change how this hardware works later. They could remove or add AMX instructions. Depends on whether these instructions can be executed with unprivileged user-land code. Once the genie is out of the bottle, backward compatibility means imitating the old instruction set. Since a recompile is often cheap enough (and esp. if coprocessor calls can be remapped at the bitcode compilation time), this should be a non-issue.
- saagarjha 6y ago
  Apple isn't going to support your code using custom instructions.
  - h0l0cube 6y ago
    I forget that this is how Apple's operates. I can just see a bunch of software that is really popular on Mac (e.g. image + video processing, 3d) that could get a massive speed boost with unfettered access to these instructions. Their library ain't going to cater for a fraction of the useful stuff that a matrix coprocessor can do.
- TazeTSchnitzel 6y ago
  When Apple have decided not to document an instruction set to give them freedom to change it, I don't think they will cry about breaking your application in the next hardware version. This is just like how (some) GPU vendors don't document their hardware's data structures and instruction set. You're living dangerously if you choose to rely on their exact behaviour. The vendors really only expect you to use their API (Vulkan/Metal/Direct3D/etc) because that abstraction layer allows them to radically and incompatibly change their hardware on a regular basis to improve performance or functionality.
  - uncledave 6y ago
    Exactly this. You buy hardware for the API that ships with the software on it these days.
saagarjha 6y ago
> However due to pressure from customers ARM relented and announced in 2019 that they would allow extensions. Of course Apple doesn't care and has been shipping custom instructions for longer than that…
- klelatti 6y ago
  Reference to support this? Plus you seem to imply that Apple has done this without Arm's consent which seems unlikely and even if so I would be astonished if that were in the public domain.
  - saagarjha 6y ago
    I have no idea what Arm’s involvement is in all this, but from the outside it has consistently seemed like Apple launches chips that don’t follow what is documented and then the requirements invariably change soon after. Just off the top of my head, they’ve been shipping their own custom memory compression instructions for a while now, and I am pretty sure their chips dropped support for 4K granules much earlier than the rules claimed this was acceptable. AMX showed up in A13, I believe? Which would be just before the linked article is dated.
    - socialdemocrat 6y ago
      What is a memory compression instruction? You mean their CPUs have special machine code instructions as part of their ISA for compressing segments of memory? Or do you mean something more akin to the Thumb format but for ARM-64?
      - saagarjha 6y ago
        > You mean their CPUs have special machine code instructions as part of their ISA for compressing segments of memory? Yes.
    - klelatti 6y ago
      > I have no idea what Arm’s involvement is in all this But you implied that Apple flouted what was permitted by Arm. > Just off the top of my head That's not a reference. As others have commented its much, much more likely that Apple and Arm have a very close relationship and that Apple are just at the leading edge.
ur-whale 6y ago
Let me just say this: if that actually exists, it isn't the first time, by far, Apple has pulled off that trick (or others, like Intel for that matter)
cjdell 6y ago
I recall a relevant piece of history where the Intel 286 didn't have the ability to drop from the new protected mode back to real mode without a full system reset. A team at IBM eventually figured out that with a well crafted dance of instructions this could be achieved by taking advantage of specific quirks within the chip. By the time the 386 was released, this method was so common that Intel had to preserve these quirks to maintain compatibility with existing software. Apple may find themselves in a similar situation if they're not careful.
- fluffything 6y ago
  Apple just breaks such Software.
- socialdemocrat 6y ago
  It is well known that Apple does not follow such policy. If you didn't follow their standards and your software break, then that is on you. Apple will not lift a finger. This has been Apple practice for over 20 years, and if you develop on the Mac platform you should be well aware of it or suffer the consequences.
  - donarb 6y ago
    In the early years, Microsoft had written some system hacks that they used to run Word and Excel on Apple computers. Apple had to be careful not to break those hacks when releasing OS updates. Eventually Apple got tired of a vendor dictating changes on it's own OS and got Microsoft to remove these hacks.
    - mrkstu 6y ago
      Yep, some vendors have that kind of pull, which is exactly why the Carbon API was created and persisted for a long while. But, like Carbon, Apple will eventually sunset anything that drags at them evolving the platform.
    - Someone 6y ago
      I only remember that they postponed the move from a 24-bit address bus to a full 32-bit one because about every program would have crashed (https://en.wikipedia.org/wiki/Classic_Mac_OS_memory_management#32-bit_clean https://en.wikipedia.org/wiki/Classic_Mac_OS_memory_manageme.... Many programs flipped bits because it was a lot faster than making the system call that did the same) I don’t remember any Excel-specific code to keep Excel running, but there was a special error code from the “launch program” system call saying “program has special memory requirements”, but AFAIK, it actually meant “program is Excel version such and such” (changing the programs’s creator code made “launch program” succeed, but the program almost immediately crashed.
xiphias2 6y ago
SIMD was probably the wrong direction to take instead of vector / matrix coprocessors, as SIMD can't do as aggressive pipelining. RISC-V is fixing it, but it's great that Apple is also going into this direction.
- andy_ppp 6y ago
  Cool, how is RISC-V dealing with this? Can’t wait to see what processor in 10 years or so on 1nm lithography look like, they might even be able to run a JavaScript test suit at a reasonable pace or compile Rust fast :-)
  - xiphias2 6y ago
    It has an instruction set where the instructions are independent of the size of the register. It's easier to look some videos about it on YouTube than to explain it actually.
  - rrss 6y ago
    Generally the idea is to make the instruction set "scalable," such that if the CPU's vector ALU doubles in width, applications don't need to be rewritten or recompiled to benefit. (In contrast to existing SIMD instructions, which are for a specific width, which is why e.g. Intel introduces new instructions for wider and wider vectors - AVX1, AVX2, AVX512). ARM and RISC-V are both pursuing this approach, with SVE and the V extension, respectively. Though it is unlikely that vector instructions will help very much for typical JS or compilation workloads.
    - ithkuil 6y ago
      Does any of these new vector instruction sets that only differ from the previous one fir the width of the operands still allow for a new patient to be filed?
      - rrss 6y ago
        I would guess yes, but don't really have any idea. To clarify, though, the changes introduced in newer SIMD extensions are rarely just a width increase. AVX-512, for example, increased the width from 256-bit to 512-bit, but also brought some substantial programmability improvements.
rrss 6y ago
> However due to pressure from customers ARM relented and announced in 2019 that they would allow extensions. The cited announcement here was for ARMv8-M only, the microcontroller variant of ARMv8 for embedded systems. From the linked EETimes article: >> Arm is opening up its instruction set to customers’ customized instructions for Cortex M cores >> For Cortex-A cores, Arm is still a long way from offering any customizable instructions From https://www.arm.com/why-arm/technologies/custom-instructions https://www.arm.com/why-arm/technologies/custom-instructions: > Arm Custom Instructions are a standard feature of the Armv8-M architecture Apple M1 is not an ARMv8-M core, it's an ARMv8-A core. I'd very much like to understand the contractual terms under which Apple put custom instructions into their ARMv8-A core, but I don't think it is as simple as "ARM is totally fine with all licensees doing this now." AFAIK, the latest revisions of ARMv8-A still do not permit custom instructions. Other reasons I don't think AMX is in the category of the publicly announced "custom instructions" part of the ISA: - arm custom instruction are prohibited from accessing memory, but AMXLD* and AMXST* exist. - arm custom instructions are prohibited from having their own register states, but AMX has amx0, amx1, amx2 registers. https://developer.arm.com/architectures/instruction-sets/custom-instructions https://developer.arm.com/architectures/instruction-sets/cus...
- Asmod4n 6y ago
  This only applies if Apple would sell their chips as ARM Cpus to other companies etc.
  - rrss 6y ago
    Do you know this, or is this a guess? This would be surprising to me. Suppose Amazon acquires an architecture license, designs a custom core for Graviton3, and deploys it in AWS. Could they add custom instructions at will? I would think ARM would be very unhappy when there are applications that run on amazon's arm64 CPU that don't run on ARM's arm64 CPUs, and I would imagine that that would be reflected in the license terms.
- aneutron 6y ago
  It should also be noted that ARM has multiple types of licences. Most manufacturers only license cores and adjacent IPs, with no possible "tweaks" or the license with minimal "tweaks". However, my understanding is that apple has an "ISA" architecture, where they basically can do whatever they want, since they can build a custom CPU that uses the ARM ISA, whereas Samsung for example, can only "tweak" the cores. Basically, Apple doesn't use core IP and "only" licenses the ISA (which is far more expensive if my memory is not lying) and so they can do whatever they want.
sigwinch28 6y ago
Off-topic, but relevant due to the author's choice of publishing platform. > You have 1 free member-only story left this month. Sign up for Medium and get an extra one How about no?
proverbialbunny 6y ago
Oh shiny! As a data scientist who currently is in Pandas day-to-day, under the hood it uses BLAS (and I think LAPACK) which should give some nice acceleration benefits when working with larger dataframes on Apple's M series chips.
- jhayward 6y ago
  Numpy recently removed Apple's Accelerate framework as a supported BLAS implementation, due to alleged bugs in Accelerate. The Accelerate framework is the only thing that has the Matrix co-processor support. https://github.com/numpy/numpy/pull/15759 https://github.com/numpy/numpy/pull/15759
- JacobiX 6y ago
  To benefit from the acceleration, one should use the Apple implementation of BLAS/LAPACK: the Accelerate framework, for the moment I think numerical python libs link against OpenBLAS ? But I'm not 100% sure.
andy_ppp 6y ago
I found the article could probably be trimmed down somewhat... is the idea that using a separate processor for SIMD style operating allows pipelining and other optimisations that would be difficult to implement in the context of a more general purpose CPU?
- 95014_refugee 6y ago
  There is no "separate processor". These are just private ISA extensions.
Jkvngt 6y ago
I hate to say it but that article is overly confusing and needs better organization.
- supernova87a 6y ago
  I understood most of the information presented, but missed some basic technical point of what the AMX does. Is it a specialized module + its libraries that's very good at dot products, for example? Matrix inversions? That's what I wanted to know. Maybe I missed it.
JoshTko 6y ago
Maybe this is the hardware based X86 translation.
- als0 6y ago
  There is no hardware based translation. It is done purely in software by Rosetta 2. However, having Total Store Ordering (TSO) in hardware does help simplify Rosetta.
ENOTTY 6y ago
Intel recently disclosed an ISA extension for matrix operations called Advanced Matrix Extensions (AMX). It should appear in Sapphire Rapids. https://fuse.wikichip.org/news/3600/the-x86-advanced-matrix-extension-amx-brings-matrix-operations-to-debut-with-sapphire-rapids/ https://fuse.wikichip.org/news/3600/the-x86-advanced-matrix-...
sradman 6y ago
This article about the undocumented Apple AMX instructions is the missing piece of the puzzle for me. Various HN threads discussed SIMD [1] and ML/DL [2] on the Apple M1. Daniel Lemire's benchmark compared x64 SIMD (SSE, AVX) vs. ARM SIMD (Neon, SVE2 [next gen]). Apple's AMX is a coprocessor that does Matrix math in addition to Vector SIMD math. Maynard Handley clearly understood this in his comment on Lemire's blog [3]: > You (and other commenters) are aware of NEON, but apparently not of AMX. AMX may not work for the sorts of JSON parsing weirdness for which you use AVX256 (that’ll have to wait for SVE/2, probably next year) but it does solve the problem of “I want to execute dense linear algebra fast”. > You might want to run some comparisons of that for your M1 vs Intel MacBooks… The API’s to look at are in Accelerate() The Core ML [4] docs have a nice diagram that demonstrates the relationship with the lower level Accelerate [5] and Accelerate BNNS [6]: > Core ML is the foundation for domain-specific frameworks and functionality. Core ML supports Vision for analyzing images, Natural Language for processing text, Speech for converting audio to text, and Sound Analysis for identifying sounds in audio. Core ML itself builds on top of low-level primitives like Accelerate and BNNS, as well as Metal Performance Shaders. Apple's TensorFlow implementation for the M1 is analogous to Core ML. It probably invokes the appropriate Accelerate BNNS and/or Metal library calls. Maynard Handley was exactly right, SIMD/ML/DL code that targets Apple silicon should link against Apple's Accelerate or Core ML libraries. These libraries in turn invoke the appropriate AMX, Neural Engine, and GPU instructions for each target platform. [1] https://news.ycombinator.com/item?id=25408853 https://news.ycombinator.com/item?id=25408853 [2] https://news.ycombinator.com/item?id=25773109 https://news.ycombinator.com/item?id=25773109 [3] https://lemire.me/blog/2020/12/11/arm-macbook-vs-intel-macbook/#comment-561810 https://lemire.me/blog/2020/12/11/arm-macbook-vs-intel-macbo... [4] https://developer.apple.com/documentation/coreml https://developer.apple.com/documentation/coreml [5] https://developer.apple.com/documentation/accelerate https://developer.apple.com/documentation/accelerate [6] https://developer.apple.com/documentation/accelerate/bnns https://developer.apple.com/documentation/accelerate/bnns
lallysingh 6y ago
The action is in the linked gist https://gist.github.com/dougallj/7a75a3be1ec69ca550e7c36dc75e0d6f https://gist.github.com/dougallj/7a75a3be1ec69ca550e7c36dc75... # These # instructions have been reversed from Accelerate (vImage, libBLAS, libBNNS, # libvDSP and libLAPACK all use them), and by experimenting with their # behaviour on the M1.
eyesee 6y ago
Apple actually did announce this capability in their “Time Flies” keynote which introduced the A14 with the new iPad. They call it “Second-generation machine learning accelerators” in the CPU, with “accelerated matrix multiplication”. Start at 53:14: https://youtu.be/b13xnFp_Los https://youtu.be/b13xnFp_Los
- joseph_grobbles 6y ago
  They've advertised this many times in many venues. It has been a featured part of their keynote. It has been talked about endlessly. There is positively nothing "secret" about this.
  - 95014_refugee 6y ago
    ... it's also not a "coprocessor"
- zaroth 6y ago
  Thank you. Mods should change the title to avoid clickbait.

https://archive.is/eeksi https://archive.is/eeksi

mackman 6y ago
This article links to this more detailed post which I’d recommend: https://nod.ai/comparing-apple-m1-with-amx2-m1-with-neon/ https://nod.ai/comparing-apple-m1-with-amx2-m1-with-neon/
sgt 6y ago
Thanks! @dang - can we not make this an automatic HN feature? That is, automatically linking to archive.is for all stories from medium.com, NY Times, etc.
- kube-system 6y ago
  I hope not. archive.is doesn’t resolve on all DNS providers.
  - sgt 6y ago
    You mean it is currently being censored by DNS providers and/or ISP's?
    - kube-system 6y ago
      I’m not sure that’s the case. https://community.cloudflare.com/t/archive-is-error-1001/18227 https://community.cloudflare.com/t/archive-is-error-1001/182...

> However more importantly, this is an advantage to Apple. By only letting their libraries use these special instructions Apple retains the freedom to radically change how this hardware works later. They could remove or add AMX instructions. Depends on whether these instructions can be executed with unprivileged user-land code. Once the genie is out of the bottle, backward compatibility means imitating the old instruction set. Since a recompile is often cheap enough (and esp. if coprocessor calls can be remapped at the bitcode compilation time), this should be a non-issue.

saagarjha 6y ago
Apple isn't going to support your code using custom instructions.
- h0l0cube 6y ago
  I forget that this is how Apple's operates. I can just see a bunch of software that is really popular on Mac (e.g. image + video processing, 3d) that could get a massive speed boost with unfettered access to these instructions. Their library ain't going to cater for a fraction of the useful stuff that a matrix coprocessor can do.
TazeTSchnitzel 6y ago
When Apple have decided not to document an instruction set to give them freedom to change it, I don't think they will cry about breaking your application in the next hardware version. This is just like how (some) GPU vendors don't document their hardware's data structures and instruction set. You're living dangerously if you choose to rely on their exact behaviour. The vendors really only expect you to use their API (Vulkan/Metal/Direct3D/etc) because that abstraction layer allows them to radically and incompatibly change their hardware on a regular basis to improve performance or functionality.
- uncledave 6y ago
  Exactly this. You buy hardware for the API that ships with the software on it these days.

saagarjha 6y ago

> However due to pressure from customers ARM relented and announced in 2019 that they would allow extensions. Of course Apple doesn't care and has been shipping custom instructions for longer than that…

klelatti 6y ago
Reference to support this? Plus you seem to imply that Apple has done this without Arm's consent which seems unlikely and even if so I would be astonished if that were in the public domain.
- saagarjha 6y ago
  I have no idea what Arm’s involvement is in all this, but from the outside it has consistently seemed like Apple launches chips that don’t follow what is documented and then the requirements invariably change soon after. Just off the top of my head, they’ve been shipping their own custom memory compression instructions for a while now, and I am pretty sure their chips dropped support for 4K granules much earlier than the rules claimed this was acceptable. AMX showed up in A13, I believe? Which would be just before the linked article is dated.
  - socialdemocrat 6y ago
    What is a memory compression instruction? You mean their CPUs have special machine code instructions as part of their ISA for compressing segments of memory? Or do you mean something more akin to the Thumb format but for ARM-64?
    - saagarjha 6y ago
      > You mean their CPUs have special machine code instructions as part of their ISA for compressing segments of memory? Yes.
  - klelatti 6y ago
    > I have no idea what Arm’s involvement is in all this But you implied that Apple flouted what was permitted by Arm. > Just off the top of my head That's not a reference. As others have commented its much, much more likely that Apple and Arm have a very close relationship and that Apple are just at the leading edge.

ur-whale 6y ago

Let me just say this: if that actually exists, it isn't the first time, by far, Apple has pulled off that trick (or others, like Intel for that matter)

cjdell 6y ago

I recall a relevant piece of history where the Intel 286 didn't have the ability to drop from the new protected mode back to real mode without a full system reset. A team at IBM eventually figured out that with a well crafted dance of instructions this could be achieved by taking advantage of specific quirks within the chip. By the time the 386 was released, this method was so common that Intel had to preserve these quirks to maintain compatibility with existing software. Apple may find themselves in a similar situation if they're not careful.

fluffything 6y ago
Apple just breaks such Software.
socialdemocrat 6y ago
It is well known that Apple does not follow such policy. If you didn't follow their standards and your software break, then that is on you. Apple will not lift a finger. This has been Apple practice for over 20 years, and if you develop on the Mac platform you should be well aware of it or suffer the consequences.
- donarb 6y ago
  In the early years, Microsoft had written some system hacks that they used to run Word and Excel on Apple computers. Apple had to be careful not to break those hacks when releasing OS updates. Eventually Apple got tired of a vendor dictating changes on it's own OS and got Microsoft to remove these hacks.
  - mrkstu 6y ago
    Yep, some vendors have that kind of pull, which is exactly why the Carbon API was created and persisted for a long while. But, like Carbon, Apple will eventually sunset anything that drags at them evolving the platform.
  - Someone 6y ago
    I only remember that they postponed the move from a 24-bit address bus to a full 32-bit one because about every program would have crashed (https://en.wikipedia.org/wiki/Classic_Mac_OS_memory_management#32-bit_clean https://en.wikipedia.org/wiki/Classic_Mac_OS_memory_manageme.... Many programs flipped bits because it was a lot faster than making the system call that did the same) I don’t remember any Excel-specific code to keep Excel running, but there was a special error code from the “launch program” system call saying “program has special memory requirements”, but AFAIK, it actually meant “program is Excel version such and such” (changing the programs’s creator code made “launch program” succeed, but the program almost immediately crashed.

xiphias2 6y ago

SIMD was probably the wrong direction to take instead of vector / matrix coprocessors, as SIMD can't do as aggressive pipelining. RISC-V is fixing it, but it's great that Apple is also going into this direction.

andy_ppp 6y ago
Cool, how is RISC-V dealing with this? Can’t wait to see what processor in 10 years or so on 1nm lithography look like, they might even be able to run a JavaScript test suit at a reasonable pace or compile Rust fast :-)
- xiphias2 6y ago
  It has an instruction set where the instructions are independent of the size of the register. It's easier to look some videos about it on YouTube than to explain it actually.
- rrss 6y ago
  Generally the idea is to make the instruction set "scalable," such that if the CPU's vector ALU doubles in width, applications don't need to be rewritten or recompiled to benefit. (In contrast to existing SIMD instructions, which are for a specific width, which is why e.g. Intel introduces new instructions for wider and wider vectors - AVX1, AVX2, AVX512). ARM and RISC-V are both pursuing this approach, with SVE and the V extension, respectively. Though it is unlikely that vector instructions will help very much for typical JS or compilation workloads.
  - ithkuil 6y ago
    Does any of these new vector instruction sets that only differ from the previous one fir the width of the operands still allow for a new patient to be filed?
    - rrss 6y ago
      I would guess yes, but don't really have any idea. To clarify, though, the changes introduced in newer SIMD extensions are rarely just a width increase. AVX-512, for example, increased the width from 256-bit to 512-bit, but also brought some substantial programmability improvements.

rrss 6y ago

> However due to pressure from customers ARM relented and announced in 2019 that they would allow extensions. The cited announcement here was for ARMv8-M only, the microcontroller variant of ARMv8 for embedded systems. From the linked EETimes article: >> Arm is opening up its instruction set to customers’ customized instructions for Cortex M cores >> For Cortex-A cores, Arm is still a long way from offering any customizable instructions From https://www.arm.com/why-arm/technologies/custom-instructions https://www.arm.com/why-arm/technologies/custom-instructions: > Arm Custom Instructions are a standard feature of the Armv8-M architecture Apple M1 is not an ARMv8-M core, it's an ARMv8-A core. I'd very much like to understand the contractual terms under which Apple put custom instructions into their ARMv8-A core, but I don't think it is as simple as "ARM is totally fine with all licensees doing this now." AFAIK, the latest revisions of ARMv8-A still do not permit custom instructions. Other reasons I don't think AMX is in the category of the publicly announced "custom instructions" part of the ISA: - arm custom instruction are prohibited from accessing memory, but AMXLD* and AMXST* exist. - arm custom instructions are prohibited from having their own register states, but AMX has amx0, amx1, amx2 registers. https://developer.arm.com/architectures/instruction-sets/custom-instructions https://developer.arm.com/architectures/instruction-sets/cus...

Asmod4n 6y ago
This only applies if Apple would sell their chips as ARM Cpus to other companies etc.
- rrss 6y ago
  Do you know this, or is this a guess? This would be surprising to me. Suppose Amazon acquires an architecture license, designs a custom core for Graviton3, and deploys it in AWS. Could they add custom instructions at will? I would think ARM would be very unhappy when there are applications that run on amazon's arm64 CPU that don't run on ARM's arm64 CPUs, and I would imagine that that would be reflected in the license terms.
aneutron 6y ago
It should also be noted that ARM has multiple types of licences. Most manufacturers only license cores and adjacent IPs, with no possible "tweaks" or the license with minimal "tweaks". However, my understanding is that apple has an "ISA" architecture, where they basically can do whatever they want, since they can build a custom CPU that uses the ARM ISA, whereas Samsung for example, can only "tweak" the cores. Basically, Apple doesn't use core IP and "only" licenses the ISA (which is far more expensive if my memory is not lying) and so they can do whatever they want.

sigwinch28 6y ago

Off-topic, but relevant due to the author's choice of publishing platform. > You have 1 free member-only story left this month. Sign up for Medium and get an extra one How about no?

proverbialbunny 6y ago

Oh shiny! As a data scientist who currently is in Pandas day-to-day, under the hood it uses BLAS (and I think LAPACK) which should give some nice acceleration benefits when working with larger dataframes on Apple's M series chips.

jhayward 6y ago
Numpy recently removed Apple's Accelerate framework as a supported BLAS implementation, due to alleged bugs in Accelerate. The Accelerate framework is the only thing that has the Matrix co-processor support. https://github.com/numpy/numpy/pull/15759 https://github.com/numpy/numpy/pull/15759
JacobiX 6y ago
To benefit from the acceleration, one should use the Apple implementation of BLAS/LAPACK: the Accelerate framework, for the moment I think numerical python libs link against OpenBLAS ? But I'm not 100% sure.

andy_ppp 6y ago

I found the article could probably be trimmed down somewhat... is the idea that using a separate processor for SIMD style operating allows pipelining and other optimisations that would be difficult to implement in the context of a more general purpose CPU?

95014_refugee 6y ago
There is no "separate processor". These are just private ISA extensions.

Jkvngt 6y ago

I hate to say it but that article is overly confusing and needs better organization.

supernova87a 6y ago
I understood most of the information presented, but missed some basic technical point of what the AMX does. Is it a specialized module + its libraries that's very good at dot products, for example? Matrix inversions? That's what I wanted to know. Maybe I missed it.

JoshTko 6y ago

Maybe this is the hardware based X86 translation.

als0 6y ago
There is no hardware based translation. It is done purely in software by Rosetta 2. However, having Total Store Ordering (TSO) in hardware does help simplify Rosetta.

ENOTTY 6y ago

Intel recently disclosed an ISA extension for matrix operations called Advanced Matrix Extensions (AMX). It should appear in Sapphire Rapids. https://fuse.wikichip.org/news/3600/the-x86-advanced-matrix-extension-amx-brings-matrix-operations-to-debut-with-sapphire-rapids/ https://fuse.wikichip.org/news/3600/the-x86-advanced-matrix-...

sradman 6y ago

This article about the undocumented Apple AMX instructions is the missing piece of the puzzle for me. Various HN threads discussed SIMD [1] and ML/DL [2] on the Apple M1. Daniel Lemire's benchmark compared x64 SIMD (SSE, AVX) vs. ARM SIMD (Neon, SVE2 [next gen]). Apple's AMX is a coprocessor that does Matrix math in addition to Vector SIMD math. Maynard Handley clearly understood this in his comment on Lemire's blog [3]: > You (and other commenters) are aware of NEON, but apparently not of AMX. AMX may not work for the sorts of JSON parsing weirdness for which you use AVX256 (that’ll have to wait for SVE/2, probably next year) but it does solve the problem of “I want to execute dense linear algebra fast”. > You might want to run some comparisons of that for your M1 vs Intel MacBooks… The API’s to look at are in Accelerate() The Core ML [4] docs have a nice diagram that demonstrates the relationship with the lower level Accelerate [5] and Accelerate BNNS [6]: > Core ML is the foundation for domain-specific frameworks and functionality. Core ML supports Vision for analyzing images, Natural Language for processing text, Speech for converting audio to text, and Sound Analysis for identifying sounds in audio. Core ML itself builds on top of low-level primitives like Accelerate and BNNS, as well as Metal Performance Shaders. Apple's TensorFlow implementation for the M1 is analogous to Core ML. It probably invokes the appropriate Accelerate BNNS and/or Metal library calls. Maynard Handley was exactly right, SIMD/ML/DL code that targets Apple silicon should link against Apple's Accelerate or Core ML libraries. These libraries in turn invoke the appropriate AMX, Neural Engine, and GPU instructions for each target platform. [1] https://news.ycombinator.com/item?id=25408853 https://news.ycombinator.com/item?id=25408853 [2] https://news.ycombinator.com/item?id=25773109 https://news.ycombinator.com/item?id=25773109 [3] https://lemire.me/blog/2020/12/11/arm-macbook-vs-intel-macbook/#comment-561810 https://lemire.me/blog/2020/12/11/arm-macbook-vs-intel-macbo... [4] https://developer.apple.com/documentation/coreml https://developer.apple.com/documentation/coreml [5] https://developer.apple.com/documentation/accelerate https://developer.apple.com/documentation/accelerate [6] https://developer.apple.com/documentation/accelerate/bnns https://developer.apple.com/documentation/accelerate/bnns

lallysingh 6y ago

The action is in the linked gist https://gist.github.com/dougallj/7a75a3be1ec69ca550e7c36dc75e0d6f https://gist.github.com/dougallj/7a75a3be1ec69ca550e7c36dc75... # These # instructions have been reversed from Accelerate (vImage, libBLAS, libBNNS, # libvDSP and libLAPACK all use them), and by experimenting with their # behaviour on the M1.

eyesee 6y ago

Apple actually did announce this capability in their “Time Flies” keynote which introduced the A14 with the new iPad. They call it “Second-generation machine learning accelerators” in the CPU, with “accelerated matrix multiplication”. Start at 53:14: https://youtu.be/b13xnFp_Los https://youtu.be/b13xnFp_Los

joseph_grobbles 6y ago
They've advertised this many times in many venues. It has been a featured part of their keynote. It has been talked about endlessly. There is positively nothing "secret" about this.
- 95014_refugee 6y ago
  ... it's also not a "coprocessor"
zaroth 6y ago
Thank you. Mods should change the title to avoid clickbait.