6 ms·

You can help with the reverse engineering of Apple Silicon done by a dozen people worldwide, that is how we find out the GPU and NPU instructions[1-4]. There is

You can help with the reverse engineering of Apple Silicon done by a dozen people worldwide, that is how we find out the GPU and NPU instructions[1-4]. There is over 43 trillion float operations per second to unlock at 8 terabit per second 'unified' memory bandwidth and 270 gigabits per second networking (less on the smaller chips).... [1] https://github.com/AsahiLinux/gpu https://github.com/AsahiLinux/gpu [2] https://github.com/dougallj/applegpu https://github.com/dougallj/applegpu [3] https://github.com/antgroup-skyward/ANETools/tree/main/ANEDisassembler https://github.com/antgroup-skyward/ANETools/tree/main/ANEDi... [4] https://github.com/hollance/neural-engine https://github.com/hollance/neural-engine You can use a high level APIs like MLX, Metal or CoreML to compute other things on the GPU and NPU. Shadama [5] is an example programming language that translates (with Ometa) matrix calculations into WebGPU or WebGL APIs (I forget which). You can do exactly the same with the MLX, Metal or CoreML APIs and only pay around 3% overhead going through the translation stages. [5] https://github.com/yoshikiohshima/Shadama https://github.com/yoshikiohshima/Shadama I estimate it will cost around $22K at my hourly rate to completely reverse engineer the latest A16 and M4 CPU (ARMV9), GPU and NPU instruction sets. I think I am halfway on the reverse engineering, the debugging part is the hardest problem. You would however not be able to sell software with it on the APP Store as Apple forbids undocumented API's or bare metal instructions.

KeplerBoy 2y ago
Where does the 270 gbit/s networking figure come from? Is it the aggregate bandwidth from the pcie slots on the mac pro, which could support nics at that speeds (and above according to my quick maths#), but there is not really any driver support for modern Intel or Mellanox/Nvidia NICs as far as I can tell. My use case would be hooking up a device which spews out sensor data at 100 gbit/s over qsfp28 ethernet as directly to a GPU as possible. The new mac mini has the GPU power, but there's no way to get the data into it. # 2x Gen4x16 + 4x Gen3x8 = 2 * 31.508 GB/s + 4 * 7.877 GB/s ≈ 90 GB/s = 720 gbit/s
- morphle 2y ago
  > Where does the 270 gbit/s networking figure come from? Is it the aggregate bandwidth from the pcie slots on the Mac pro We both should restate and specify the calculation for each different Apple Silicon chip and the PCB/machine model it is wired onto. The $599 M4 Mac mini base model networking (aggregated Wifi, USB-C, 10G Ethernet, Thunderbolt PCIe) is almost 270 Gbps. Your 720 Gbps is for a >$8000 Mac Pro M2 Ultra but the number is to high because the 2x Gen4x16 is shared/oversubscribed with the other PCIe lanes for x8 PCIe slots, SSD and Thunderbolt. You need to measure/benchmark it, not read the marketing PR. I estimate the $1400 M4 Pro Mac mini networking bandwidth by adding the external WiFi, 10 Gbps Ethernet, two USC-C ports (2 x 10 Gbps) and three Thunderbolt 4 ports (3 x 80/120 Gbps) but subtracting the PCIe 64 Gbps limit and not counting the internal SSD. Two $599 M4 Mac mini base models are faster and cheaper than one M4 Pro Mac mini. The point of the precise actual measurements I did of the trillion opereations per second and the billion of bits per second networking/interconnect of the M4 Mac mini against all the other Apple silicon machines is to find which package (chip plus pcb plus case) has the best price/performance/watt balanced against them networked together. On januari 2025 you can build the cheapest fastest supercomputer in the world from just off the shelf M4 16Gb Mac mini base models with 10G Ethernet, Mikrotek 100G switches and a few FPGA's. It would outperform all Nvidia, Cerebras, Tenstorrent and datacenter clusters I know of, mainly because of the low power Apple Silicon. Note that the M4 has only 1,2 Tips unified memory bandwidth and the M4 Pro has double that. The 8 Tops unified memory bandwidth is on the M1 and M2 Studio Ultra with 64/128/192GB DRAM. Without it you cant's reach 50 trillion operations per second. A Mac Studio has only around 190 Gbps external networking bandwidth but does not reach 43 trillion TOPS, as does the 720 Gbps of your Mac Pro estimate. By reverse engineering the instruction set you could squeeze a few percent extra performance out of this M4 cluster. The 43 trillion TOPS of the M4 itself is an estimate. The ANE does 34 TOPS, the CPU less than 5 TOP depending on float type and we have no reliable benchmarks for the CPU floating point.
  - KeplerBoy 2y ago
    The pcie configuration was taken from the mac pro and it's m2 ultra. https://www.apple.com/mac-pro/ https://www.apple.com/mac-pro/ I'd assume the mac mini has a less extensive pcie/tb subsystem. No idea what people are doing with all those pcie slots except for nvme cards. I wonder how hard it would be to talk to a pcie fpga.
- deleted 2y ago
  [deleted]
- morphle 2y ago
  >but there's no way to get the data into it at 100 Gbps I'm confident you can get 100 Gbps in by aggregating M4 Mac mini ports. I resell a $199 Microtik CCR2004-1G-2XS-PCIe SmartNIC with 2 x 25 Gbps QSFP28 that connects to a x8 PCIe 3.0. (I still have a few available for $140 plus shipping plus a few refurbished 16 x 10 Gbps for $400 and 8 x 100 Gbps switches for $800). Theoretically you can connect that SmartNIC to two of the three M4 Mac mini Thunderbolt 4/USB4 ports that pass through 2 x x4 PCIe 3.0, if you can figure out how to aggregate the two x4 PCIe lanes into a single x8 port. The driver source code is for Linux and could be ported to MacOS. You then aggregate the ports with the 100 Gbps switch. I'm pretty sure you could create a new PCB design with a larger Broadcom switch chip model to attach to the 10G Ethernet, two 10 Gbps USB-C ports plus the three Thunderbolt 4/USB4 port and write a new driver to aggregate over the 6 ports. You'd have 126 Gbps minus the PCIe overhead and could combine it into a single 100 Gbps QSFP28 port. I already warned this is still theoretical. Broadcom might not sell you the switch chip, Intel might not sell you the Thunderbolt chip and Apple might block the installation of your device driver code. But people already proved the interconnect with the Apple Thunderbolt Bridge driver at 3 x 10 Gbps connected via large expensive Thunderbolt hubs [2]. Others just connect each port to different M4 Macs [1][3][4] in various ways. [1] https://x.com/alexocheema/status/1807882764261417000 https://x.com/alexocheema/status/1807882764261417000 [2] https://www.youtube.com/watch?v=GBR6pHZ68Ho https://www.youtube.com/watch?v=GBR6pHZ68Ho [3] https://www.youtube.com/watch?v=2eNVV0ouBxg https://www.youtube.com/watch?v=2eNVV0ouBxg [4] https://www.youtube.com/watch?v=SkmrUWyZThQ https://www.youtube.com/watch?v=SkmrUWyZThQ
  - KeplerBoy 2y ago
    I know this should all be possible, but isnt really because Apple doesnt care about this use case. I'll just stick to Orin AGX modules with their proper pcie slot and real Linux support. I want to do something meaningful with the incoming data and not waste years just getting the link up and running.
dgfitz 2y ago
It’s too bad they don’t make this easier on developers, Apple. Is there a reason I don’t see?
- morphle 2y ago
  There certainly is a reason and indeed you don't see it because Apple downplays these things in their PR. It might be the same reason that is behind NVDIA's CUDA moat. CUDA lock-in prevented competitors like AMD and Intel to convince programmers and their customers to switch away from CUDA. So there was no software ported to their competitive GPU's. So you get anti-trust lawsuits [1]. I think you should put yourself in Apples management mindset and then reason. I suspect they think they will not sell more iPhones or Macs if they let third party developers access the low level APIs and write faster software. They might reason that if no one knows the instruction sets hackers will write less code to break security. Security by obscurity. They certainly think that blocking competitors from reverse engineering the low power Apple Silicon and blocking them from using TSMC manufacturing capacity will keep them the most profitable company for another decade. [1] https://news.ycombinator.com/item?id=40593576 https://news.ycombinator.com/item?id=40593576
  - dylan604 2y ago
    At this point, Apple is absolutely not afraid of an anti-trust lawsuit. To them, it is part of the cost of doing business
    - morphle 2y ago
      I concur, they are virtually untouchable in this respect. No one else will throw a trillion or more into developing lower power faster silicon.
  - _zoltan_ 2y ago
    CUDA didn't prevent anything at least not in the way you believe. Intel and AMD had no competitive offer, period. They still don't. NVIDIA is simply offering an ecosystem that is battle tested and is ready out of the box. Look at the recent semianalysis test to see how not ready AMD is, who would be the only company to have a real shot at this. Their HW on paper is better or equal, yet their software ecosystem is nowhere ready.
MuffinFlavored 2y ago
This would get rid of needing Metal to be the blackbox and enable things like "nvptx CUDA" equivalent / https://libc.llvm.org/gpu/ https://libc.llvm.org/gpu/ right? Very interesting. A steal for $22k but I guess very niche for now...
- morphle 2y ago
  Yes, knowing the exact CPU and ANE assembly instructions (or the underlying microcode!!) allows for general purpose software to adaptively compile processes on all the core types, not just the CPU ones. Its won't always be faster, you get more cache misses (some cores don't have cache) and different DMA and thread scheduling, some registers can't fit the floats or large integers, etc etc. But yes, it will be possible to use all 140 cores of the M2 Ultra or the 36 cores of the M4. There will be an M6 Extreme some day, maybe 500 cores? Actually, the GPU and ANE cores themselves are built from teams of smaller cores, maybe a few dozens, hundreds or thousand in all, same as in most NVDIA chips. >A steal for $22k but I guess very niche for now... A single iPhone or Mac app (a game, an LLM, pattern recognition, security app, VPN, de/encryption, video en/dec coder) that can be sped up by 80%-200% can afford my faster assembly level API. A whole series of hardware level zero-day exploits for iPhone and Mac would become possible, now that won't be very niche at all. It is worth millions to reverse Apple Silicon instruction sets.
  - MuffinFlavored 2y ago
    What would a "llvm compilable" hello world look like that matches the libc GPU example for "AGX" (Apple Graphics)? It's not possible from MacOS, right? It'd have to be done from Linux?
    - morphle 2y ago
      No, I don't think it is impossible for MacOS. I might be missing a detail here, not sure. I have to think it over. I have seen [1] you can patch ANECompilerService, so you can even speed up existing code, because Apple compiles your code just in time (at runtime) on each machine. We could do that for MacOS libc too. [1] Some how-to hints in https://discussions.apple.com/thread/254758525?sortBy=rank https://discussions.apple.com/thread/254758525?sortBy=rank
JackYoustra 2y ago
any place you have your current progress written up on? Any methodology I could help contribute on? I've read each one of the four links you've given over the years and it seems vague with how far people have currently gotten and exact issues.
- morphle 2y ago
  >Any methodology I could help contribute on? Several people have already contacted me today with this request. This is how I give out details and share current progress with you. Yes, you can help, most people on HN could. It is not that difficult work and it is not just low level debugging, coding and FPGA hardware. It is also organizing and even simple sales, talking to funders. With patience, you could even get paid to help. >any place you have your current progress written up on? Not any place in public, because of its value for zero-day exploits. This knowledge is worth millions. I'm in the process of rewriting my three scientific papers on reverse engineering Apple Silicon low level instructions. >it seems vague with how far people have currently gotten and exact issues. Yes, I'm afraid you're right, my apologies. It's very much detailed and technical stuff, some of it under patent and NDA, some even sensitive for winning economic wars and ongoing wars (you can guess those are exiting stories). It even plays a role in the $52.7 billion US, €43 billion EU and $150 billion (unconfirmed) Chinese Chips Acts. Apple Silicon is the main reason TSMC opened a US factory [1], keeping its instruction set details secret is deemed important. If you want more information, you should join our offline video discussions for more info. Maybe sometimes sign an NDA for the juicy bits. [1] https://www.cnbc.com/2024/12/13/inside-tsmcs-new-chip-fab-where-apple-will-make-chips-in-the-us-.html https://www.cnbc.com/2024/12/13/inside-tsmcs-new-chip-fab-wh...
  - saagarjha 2y ago
    While understanding the GPU’s microarchitecture might be useful for exploits it’s definitely not worth “millions”.
    - morphle 2y ago
      You are right. The zero-day exploits might be worth roughly a million each, but not the family tree of native GPU's, ANE, CPU instruction sets and microarchitecture on which they would be based. My apology for writing unclearly, English is not my native language. I'm surprised it is yours. Saving on energy, programming effort and purchase cost of a supercomputer in case of M4 instruction sets and microarchitecture knowledge would also save millions.

KeplerBoy 2y ago

Where does the 270 gbit/s networking figure come from? Is it the aggregate bandwidth from the pcie slots on the mac pro, which could support nics at that speeds (and above according to my quick maths#), but there is not really any driver support for modern Intel or Mellanox/Nvidia NICs as far as I can tell. My use case would be hooking up a device which spews out sensor data at 100 gbit/s over qsfp28 ethernet as directly to a GPU as possible. The new mac mini has the GPU power, but there's no way to get the data into it. # 2x Gen4x16 + 4x Gen3x8 = 2 * 31.508 GB/s + 4 * 7.877 GB/s ≈ 90 GB/s = 720 gbit/s

morphle 2y ago
> Where does the 270 gbit/s networking figure come from? Is it the aggregate bandwidth from the pcie slots on the Mac pro We both should restate and specify the calculation for each different Apple Silicon chip and the PCB/machine model it is wired onto. The $599 M4 Mac mini base model networking (aggregated Wifi, USB-C, 10G Ethernet, Thunderbolt PCIe) is almost 270 Gbps. Your 720 Gbps is for a >$8000 Mac Pro M2 Ultra but the number is to high because the 2x Gen4x16 is shared/oversubscribed with the other PCIe lanes for x8 PCIe slots, SSD and Thunderbolt. You need to measure/benchmark it, not read the marketing PR. I estimate the $1400 M4 Pro Mac mini networking bandwidth by adding the external WiFi, 10 Gbps Ethernet, two USC-C ports (2 x 10 Gbps) and three Thunderbolt 4 ports (3 x 80/120 Gbps) but subtracting the PCIe 64 Gbps limit and not counting the internal SSD. Two $599 M4 Mac mini base models are faster and cheaper than one M4 Pro Mac mini. The point of the precise actual measurements I did of the trillion opereations per second and the billion of bits per second networking/interconnect of the M4 Mac mini against all the other Apple silicon machines is to find which package (chip plus pcb plus case) has the best price/performance/watt balanced against them networked together. On januari 2025 you can build the cheapest fastest supercomputer in the world from just off the shelf M4 16Gb Mac mini base models with 10G Ethernet, Mikrotek 100G switches and a few FPGA's. It would outperform all Nvidia, Cerebras, Tenstorrent and datacenter clusters I know of, mainly because of the low power Apple Silicon. Note that the M4 has only 1,2 Tips unified memory bandwidth and the M4 Pro has double that. The 8 Tops unified memory bandwidth is on the M1 and M2 Studio Ultra with 64/128/192GB DRAM. Without it you cant's reach 50 trillion operations per second. A Mac Studio has only around 190 Gbps external networking bandwidth but does not reach 43 trillion TOPS, as does the 720 Gbps of your Mac Pro estimate. By reverse engineering the instruction set you could squeeze a few percent extra performance out of this M4 cluster. The 43 trillion TOPS of the M4 itself is an estimate. The ANE does 34 TOPS, the CPU less than 5 TOP depending on float type and we have no reliable benchmarks for the CPU floating point.
- KeplerBoy 2y ago
  The pcie configuration was taken from the mac pro and it's m2 ultra. https://www.apple.com/mac-pro/ https://www.apple.com/mac-pro/ I'd assume the mac mini has a less extensive pcie/tb subsystem. No idea what people are doing with all those pcie slots except for nvme cards. I wonder how hard it would be to talk to a pcie fpga.
deleted 2y ago
[deleted]
morphle 2y ago
>but there's no way to get the data into it at 100 Gbps I'm confident you can get 100 Gbps in by aggregating M4 Mac mini ports. I resell a $199 Microtik CCR2004-1G-2XS-PCIe SmartNIC with 2 x 25 Gbps QSFP28 that connects to a x8 PCIe 3.0. (I still have a few available for $140 plus shipping plus a few refurbished 16 x 10 Gbps for $400 and 8 x 100 Gbps switches for $800). Theoretically you can connect that SmartNIC to two of the three M4 Mac mini Thunderbolt 4/USB4 ports that pass through 2 x x4 PCIe 3.0, if you can figure out how to aggregate the two x4 PCIe lanes into a single x8 port. The driver source code is for Linux and could be ported to MacOS. You then aggregate the ports with the 100 Gbps switch. I'm pretty sure you could create a new PCB design with a larger Broadcom switch chip model to attach to the 10G Ethernet, two 10 Gbps USB-C ports plus the three Thunderbolt 4/USB4 port and write a new driver to aggregate over the 6 ports. You'd have 126 Gbps minus the PCIe overhead and could combine it into a single 100 Gbps QSFP28 port. I already warned this is still theoretical. Broadcom might not sell you the switch chip, Intel might not sell you the Thunderbolt chip and Apple might block the installation of your device driver code. But people already proved the interconnect with the Apple Thunderbolt Bridge driver at 3 x 10 Gbps connected via large expensive Thunderbolt hubs [2]. Others just connect each port to different M4 Macs [1][3][4] in various ways. [1] https://x.com/alexocheema/status/1807882764261417000 https://x.com/alexocheema/status/1807882764261417000 [2] https://www.youtube.com/watch?v=GBR6pHZ68Ho https://www.youtube.com/watch?v=GBR6pHZ68Ho [3] https://www.youtube.com/watch?v=2eNVV0ouBxg https://www.youtube.com/watch?v=2eNVV0ouBxg [4] https://www.youtube.com/watch?v=SkmrUWyZThQ https://www.youtube.com/watch?v=SkmrUWyZThQ
- KeplerBoy 2y ago
  I know this should all be possible, but isnt really because Apple doesnt care about this use case. I'll just stick to Orin AGX modules with their proper pcie slot and real Linux support. I want to do something meaningful with the incoming data and not waste years just getting the link up and running.

dgfitz 2y ago

It’s too bad they don’t make this easier on developers, Apple. Is there a reason I don’t see?

morphle 2y ago
There certainly is a reason and indeed you don't see it because Apple downplays these things in their PR. It might be the same reason that is behind NVDIA's CUDA moat. CUDA lock-in prevented competitors like AMD and Intel to convince programmers and their customers to switch away from CUDA. So there was no software ported to their competitive GPU's. So you get anti-trust lawsuits [1]. I think you should put yourself in Apples management mindset and then reason. I suspect they think they will not sell more iPhones or Macs if they let third party developers access the low level APIs and write faster software. They might reason that if no one knows the instruction sets hackers will write less code to break security. Security by obscurity. They certainly think that blocking competitors from reverse engineering the low power Apple Silicon and blocking them from using TSMC manufacturing capacity will keep them the most profitable company for another decade. [1] https://news.ycombinator.com/item?id=40593576 https://news.ycombinator.com/item?id=40593576
- dylan604 2y ago
  At this point, Apple is absolutely not afraid of an anti-trust lawsuit. To them, it is part of the cost of doing business
  - morphle 2y ago
    I concur, they are virtually untouchable in this respect. No one else will throw a trillion or more into developing lower power faster silicon.
- _zoltan_ 2y ago
  CUDA didn't prevent anything at least not in the way you believe. Intel and AMD had no competitive offer, period. They still don't. NVIDIA is simply offering an ecosystem that is battle tested and is ready out of the box. Look at the recent semianalysis test to see how not ready AMD is, who would be the only company to have a real shot at this. Their HW on paper is better or equal, yet their software ecosystem is nowhere ready.

MuffinFlavored 2y ago

This would get rid of needing Metal to be the blackbox and enable things like "nvptx CUDA" equivalent / https://libc.llvm.org/gpu/ https://libc.llvm.org/gpu/ right? Very interesting. A steal for $22k but I guess very niche for now...

morphle 2y ago
Yes, knowing the exact CPU and ANE assembly instructions (or the underlying microcode!!) allows for general purpose software to adaptively compile processes on all the core types, not just the CPU ones. Its won't always be faster, you get more cache misses (some cores don't have cache) and different DMA and thread scheduling, some registers can't fit the floats or large integers, etc etc. But yes, it will be possible to use all 140 cores of the M2 Ultra or the 36 cores of the M4. There will be an M6 Extreme some day, maybe 500 cores? Actually, the GPU and ANE cores themselves are built from teams of smaller cores, maybe a few dozens, hundreds or thousand in all, same as in most NVDIA chips. >A steal for $22k but I guess very niche for now... A single iPhone or Mac app (a game, an LLM, pattern recognition, security app, VPN, de/encryption, video en/dec coder) that can be sped up by 80%-200% can afford my faster assembly level API. A whole series of hardware level zero-day exploits for iPhone and Mac would become possible, now that won't be very niche at all. It is worth millions to reverse Apple Silicon instruction sets.
- MuffinFlavored 2y ago
  What would a "llvm compilable" hello world look like that matches the libc GPU example for "AGX" (Apple Graphics)? It's not possible from MacOS, right? It'd have to be done from Linux?
  - morphle 2y ago
    No, I don't think it is impossible for MacOS. I might be missing a detail here, not sure. I have to think it over. I have seen [1] you can patch ANECompilerService, so you can even speed up existing code, because Apple compiles your code just in time (at runtime) on each machine. We could do that for MacOS libc too. [1] Some how-to hints in https://discussions.apple.com/thread/254758525?sortBy=rank https://discussions.apple.com/thread/254758525?sortBy=rank

JackYoustra 2y ago

any place you have your current progress written up on? Any methodology I could help contribute on? I've read each one of the four links you've given over the years and it seems vague with how far people have currently gotten and exact issues.

morphle 2y ago
>Any methodology I could help contribute on? Several people have already contacted me today with this request. This is how I give out details and share current progress with you. Yes, you can help, most people on HN could. It is not that difficult work and it is not just low level debugging, coding and FPGA hardware. It is also organizing and even simple sales, talking to funders. With patience, you could even get paid to help. >any place you have your current progress written up on? Not any place in public, because of its value for zero-day exploits. This knowledge is worth millions. I'm in the process of rewriting my three scientific papers on reverse engineering Apple Silicon low level instructions. >it seems vague with how far people have currently gotten and exact issues. Yes, I'm afraid you're right, my apologies. It's very much detailed and technical stuff, some of it under patent and NDA, some even sensitive for winning economic wars and ongoing wars (you can guess those are exiting stories). It even plays a role in the $52.7 billion US, €43 billion EU and $150 billion (unconfirmed) Chinese Chips Acts. Apple Silicon is the main reason TSMC opened a US factory [1], keeping its instruction set details secret is deemed important. If you want more information, you should join our offline video discussions for more info. Maybe sometimes sign an NDA for the juicy bits. [1] https://www.cnbc.com/2024/12/13/inside-tsmcs-new-chip-fab-where-apple-will-make-chips-in-the-us-.html https://www.cnbc.com/2024/12/13/inside-tsmcs-new-chip-fab-wh...
- saagarjha 2y ago
  While understanding the GPU’s microarchitecture might be useful for exploits it’s definitely not worth “millions”.
  - morphle 2y ago
    You are right. The zero-day exploits might be worth roughly a million each, but not the family tree of native GPU's, ANE, CPU instruction sets and microarchitecture on which they would be based. My apology for writing unclearly, English is not my native language. I'm surprised it is yours. Saving on energy, programming effort and purchase cost of a supercomputer in case of M4 instruction sets and microarchitecture knowledge would also save millions.