11 ms·
How to get 1.5 TFlops of FP32 performance on a single M1 CPU core
- FL33TW00D 4y agoI love all the posts by Bram. Please keep writing them!
- Tepix 4y agoSo, this is about Apple's undocumented AMX instructions, issued from CPU, executed on a special accelerator execution unit. Is there one such unit per CPU core?
- my123 4y agoM1 has one AMX unit per cluster AFAIK. This however can and does change between different chips.
- microtonal 4y agoYes, there is one per core cluster. The title is a bit misleading, because it suggests that going to two or three cores will scale linearly, though it won't be much faster. See here for sgemm benchmarks for everything from the M1 to M1 Ultra and 1 to 16 threads: https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
- adrian_b 4y agoNo. So the title is misleading, even if it is true that you get this performance with a program that uses a single CPU core.
- MuffinFlavored 4y ago> So, this is about Apple's undocumented AMX instructions, issued from CPU, executed on a special accelerator execution unit. CPU instruction -> AMX instruction -> AMX result -> CPU? How are these kinds of things usually kept in sync/in a manageable state? Like does the CPU block until the AMX returns?
- mochomocha 4y agoI think the author downplays the significance his work because it only applies to "small neural networks". There are a lot of use-cases that can benefit from this type of optimizations. Discovering how to use an undocumented fast accelerator available on millions of devices is very valuable.
- my123 4y agoApple did prefer to expose it through their own Accelerator.framework API however...
- capableweb 4y agoOf course they do, Apple like to remain as much in control as possible. If suddenly it becomes more efficient/faster to run ML/AI stuff on Asahi Linux on Mac hardware then with macOS, I'm sure they be embarrassed enough to take some sort of action. And I'm pretty sure that action will be towards the side of "closing things down" rather than "opening stuff up", as is tradition.
- my123 4y agoWrong answer. AMX is an unstable ISA that changes between product generations. That's why it's not publicly documented. Arm SME is the standardisation of the concept, but is not inmarket yet. https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture https://community.arm.com/arm-community-blogs/b/architecture...
- svantana 4y agoHas it been verified that they actually use these instructions in Accelerate.framework? I just benchmarked this on my 2019 intel i9 mbp, and got the following speeds for 128x128 matrices, 32 repeats: cblas_sgemm: 36 GFLOP/s vDSP_mmul: 41 GFLOP/s That's a pretty big deal if these functions are >30x faster on the M1...! edit: that seems to be verified in the tlkh.dev blog post above. Interestingly, I ran the same code on my bargain-basement 2020 iphone SE, and got 259GFLOP/s! These apple devices are pretty mindblowing.
- Thaxll 4y agoWhy would Apple hide such optimization from public APIs?
- esskay 4y agoIt's only really used for their internal applications and the OS level stuff so assume they want to prevent performance issues with it having to deal with 3rd party stuff.
- frogblast 4y agoIt is available via public APIs, but the hardware instructions themselves are not documented. This lets the instructions change in future CPUs, vs having the ISA be baked in stone forever. Example: AMX predated the standard ARM matrix multiply instructions. Perhaps Apple will add the ARM versions someday and now can remove AMX without breaking compatibility. Or maybe there will be a non-additive AMXv2.
- olliej 4y agoIt's not hidden from the public APIs. It's exposed through Accelerate.framework, etc. The instructions themselves aren't stable so there's no guarantee they'd exist in other generations - as others have said these instructions predate ARM actually adding the standardized scalable vector extension, presumably Apple could switch to the standard ISA in future, but if the AMX instructions were exposed directly to developers they'd have to continue to support them in future and thus continue to burn silicon space for them.
- deleted 4y ago[deleted]
- gcr 4y agoIt's amazing to me that there are four separate pieces of hardware in M1 devices that can do matrix multiplies. In addition to running on the CPU, M1 Max devices have three separate kinds of hardware-accelerated `gemm`: the GPU, the ANE (Apple Neural Engine), and this special matrix coprocessor. Here's a fairly detailed post that benchmarks each: https://tlkh.dev/benchmarking-the-apple-m1-max https://tlkh.dev/benchmarking-the-apple-m1-max And here's a great post about the justification for having so much special-purpose hardware: https://medium.com/swlh/apples-m1-secret-coprocessor-6599492fc1e1 https://medium.com/swlh/apples-m1-secret-coprocessor-6599492... As for the matrix coprocessor, Apple's built-in BLAS implementation (Accelerate.framework) uses this chip. You can link Numpy against this to benefit in your Python programs, for example. Here are some old instructions: https://gist.github.com/MarkDana/a9481b8134cf38a556cf23e1e815dafb https://gist.github.com/MarkDana/a9481b8134cf38a556cf23e1e81... All this represents yet another cycle on the Wheel of Reincarnation... (http://catb.org/jargon/html/W/wheel-of-reincarnation.html http://catb.org/jargon/html/W/wheel-of-reincarnation.html)
- amelius 4y agoIsn't this wheel of reincarnation simply a result of a shifting bottleneck? A computation can be CPU-bound or memory-bound, and this can change over hardware generations.
- gcr 4y agoMakes sense... We're also seeing energy efficiency and model size and latency becoming significant constraints these days, and the more unique constraints an application has, perhaps the more beneficial it is to have many different implementations with different tradeoffs.
- ElectricalUnion 4y ago> energy efficiency (...) many different implementations Yep, thermal throttling is a thing, and sometimes all you need is either useless silicon padding or some specialized, most of the time dark silicon to both make it feasible to cool and prevent it from melting.
- bob1029 4y agoThis is a fairly ridiculous amount of performance, all things considered. It always seemed to me like SIMD/AVX/etc would eventually come for the GPU's lunch money... How many more product generations of "SIMD on steroids" before this is practically true? The latency factor is the biggest thing for me. The GPU is a turtle compared to CPU-bound techniques. I can see emerging applications for this in real-time/streaming where every millisecond counts.
- thechao 4y agoThat's how we felt when we were writing the software rasterizer for Larrabee! The issue is that that 1.5TFLOP is probably way more power than the M1 GPU's ~2.5TFLOP. The second issue is that a SW rasterizer is going to spend ~50% of its budget emulating fixed function. So, now you're way more power, for 1/4 the perf (best case). Also, you can't run any other apps, and you're probably going to have bandwidth issues to the display controller. GPUs are an optimization to try to use the excess Moore's law we have to get to the ghost of Dennard's law.
- bob1029 4y agoI think a power/latency/perf tradeoff could be agreeable for certain applications. GPUs in the cloud are not exactly cheap. Many gaming experiences do not require nanite-level graphics. Building something that can reliably output reasonable-quality 3d graphics without relying on specific GPU technologies will give you a much broader realm to operate with. I believe something along this path is the solution for streaming gaming. I perceive the failure of Stadia, et. al. as being a consequence of trying to bolt streaming onto existing GPU-based, local gaming solutions. Build something from scratch with streaming/latency as a #1 priority, and you can dramatically expand the operational radius of each datacenter (e.g. ~100km per millisecond saved).
- jvanderbot 4y agoLight is 300km/ms in a vacuum. Is it that much slower through switched fiber?
- nimish 4y agoHow is fp64? Nvidia crippled the 4090 to just 1.3T fp64 flops so if a mac mini with m1 could match that it'd be a solid win
- bsdnoob 4y agoYou know you can see your upvoted stories right?
- kolbusa 4y agoNitpick... This paragraph is somewhat confusing. I think it is worded incorrectly: > Let's simplify the problem and implicitly transpose the matrix multiplication. Both A and B (our inputs) will have K (our reduction dimension) as the leading dimension. This doesn't really matter much in practice, but it simplifies our code a lot. The code is C[n * 16 + m] += A[k * 16 + m] * B[k * 16 + n]; Which means that actually *m* is the leading dimension of A with stride 16, and for B it is *n* with stride 16.
- NelsonMinar 4y agoDoes Apple use the AMX in their own code? Is anything like the AMX present in their mobile CPUs?
- microtonal 4y agoThe AMX units are really nice, especially because you can use them simply with standard sgemm through the Accelerate framework. However, in most applications where latency is not an issue, you'll probably want to use Metal Performance shaders instead, not only are they much faster most applications, they can also be more energy efficient. For instance, we did benchmarks of spaCy (natural language processing) transformer models across various Apple Silicon SoCs and MPS was 1.9x (M1) to 5.5x faster (M1 Ultra) while providing far more performance per Watt. E.g. using MPS on an M2 MacBook Air used 4W less energy while being 2.7x faster than AMX. Full benchmarks are at the end of this post: https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders
- londons_explore 4y agoFor comparison... A single Google TPUv4 'pod' (entire row of datacenter racks) gives 1,126,400 TFlops. Thats why your pet ML projects will always be behind those done at big companies.
- mhuffman 4y agoI have always been under the impression that there will eventually be a way to distribute ML projects across many personal computers (like the Folding@home or SETI@home) that could give even Google a run for their money! A few hundred million personal computers is a lot of processing!
- dontwearitout 4y agoOne issue with this is that many ML tasks are more memory bandwidth bound than compute bound. A forward pass through a neural network is a bunch of simple operations on gigabytes of data, whereas with protein folding there's a much higher ratio of compute to data.
- londons_explore 4y agoSETI@home had 90,000 users at its peak. Even assuming all these users have the most powerful Macbook, and are happy to run it 24/7, the whole fleet still doesn't equal just one of Googles pods. And googles pods have microsecond latency terabit interconnects, while a fleet of macbooks would have hundreds of milliseconds of latency and low bandwidth... And Google has many pods... I'm afraid even a massive team of home users will never beat companies with dedicated hardware.
- mhuffman 4y ago>SETI@home had 90,000 users at its peak. Even assuming all these users have the most powerful Macbook, and are happy to run it 24/7, the whole fleet still doesn't equal just one of Googles pods. Well ... SETI@home wasn't trying to create ML boobs for anime waifus! ... I suspect the army of young guys that want to do that would dwarf any other distributed service including bitcoin!
- superkuh 4y agoIt's a fast console, there's no doubt of that. Kind of like the Playstation 3 when it came out. Fast, not much software support without lots of special considerations, non-upgradable hardware, limited peripheral support. All in all, a fast CPU embedded in a marginal console-like "computer". People out there who were tricked into buying the M1 8GB ram version can confirm.
- 2fast4you 4y agoTricked how? I’ve got an M2 8GB and loving it
- smoldesu 4y agoEven with swap, 8gb is pretty paltry on a memory-hungry system like MacOS, let alone a system that shares GPU and system memory. 16gb is the minimum for me even though I really only edit/compile code, and even then it can be pretty easy to max out your system memory after a couple docker containers... It might not be a 'trick' per-se, but anyone who intends to use a Mac for work should consider upgraded memory (IMO).
- 2fast4you 4y agoI will agree with your last point. If I had bought this machine for doing seriously development I would've gone for 16gb. Saying that, I've been pleasantly surprised with its power. I've been playing with Metal, throwing together a native version ShaderToy, and it hasn't felt unpowered once. Even when running the iPad emulator. I did feel a little duped when I learned that some M1/M2 machines can only support one external monitor. Now I have to replace my two monitors with a widescreen.
- smoldesu 4y agoIMO, the 'problem' is that MacOS will use 4-5gb OOB, and using an Electron app with a browser open will easily push that into swap space. For most daily drivers, even light users, they'll be happy to have upgraded memory.
- muricula 4y agoSome folks may be interested in the Armv9 Scalable Matrix Extensions which appear to do something very very similar. https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture https://community.arm.com/arm-community-blogs/b/architecture...
- atonse 4y agoI remember driving to college nearly 20 years ago and one of the headlines on the radio (NPR morning show) was that the DOE had unveiled the world's fastest supercomputer at the White House that day. It was going to do a whopping 6 Teraflops, and they explained what that meant. And I remember thinking about all the possibilities with that kind of compute. I understand that this 1.5 TFlops may not be an exact comparison (or maybe it's the same), but if it's even within an order of magnitude, it is beyond mind-blowing, and we've just crossed over into at Exaflops at the supercomputer level.
- varunkmohan 4y agoPosts like these are always awesome to look at how much we can push consumer hardware. It's hard not to really appreciate some of the devices we have today. For instance, an RTX 4090 is capable of 660 TFlops of FP8 (MSRP 1600). Would not be surprised if we soon have laptops that can do petaflops of computation soon!
- kristianp 4y agoAnyone with a comparison with Intel's deep learning boost or VNNI, which is available on avx-512 processors such as https://ark.intel.com/content/www/us/en/ark/products/213805/intel-core-i511400h-processor-12m-cache-up-to-4-50-ghz.html https://ark.intel.com/content/www/us/en/ark/products/213805/...
- brrrrrm 4y agoI don't see throughput information on Intel's AMX, but their VNNI has information: https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#avx512techs=AVX512_VNNI&ig_expand=2856 https://www.intel.com/content/www/us/en/docs/intrinsics-guid... 0.5 cycles per instruction max, 3.7GHz clock, that's 7.4e9 instructions per second. If I'm reading it right, that instruction does 16 4-wide dot products, which is ~128 ops. So ~950Gops peak in int8 precision on a server class Xeon assuming no clock throttling. (edit: flops -> ops)
- stephencanon 4y agoInt8 operations are not "flops" =)
- Weryj 4y agoSomeone should run this on a M2 model.