4 ms·

If Hector Martin doesn't say that they fixed the memory-mapping bug in their PCIe controller[0] I think it's safe to say that we're not getting a new Mac Pro th

If Hector Martin doesn't say that they fixed the memory-mapping bug in their PCIe controller[0] I think it's safe to say that we're not getting a new Mac Pro this year. And to be honest, the whole concept of a Mac Pro goes against the design philosophy of Apple Silicon. The whole point of those chips is to maximize efficiency by having everything on-die or very close to the chip. For example, the M2 Max has like eight memory channels in a laptop - you could not do that even with CAMM[1] modules. A desktop machine without replaceable RAM or slots that can accommodate GPUs is going to be a nonstarter no matter how good the the SoC's integrated accelerators are. A significant chunk of the cost of the Mac Pro is just the fancy desktop case, specifically designed to accommodate lots of expansion cards and compute accelerators. If you don't care about slots, you can just buy a Mac Studio for less money than the Pro. If you do want slots, only being able to plug in storage and I/O devices and nothing else is going to be a huge drag. [0] Apple Silicon PCIe controllers have a unique limitation in that they prohibit accessing external memory devices. This is why Thunderbolt eGPUs don't work - the M1 and M2 can't actually write to the memory on the GPUs. [1] Compression Attached Memory Module, a JEDEC standard for a thin dual-channel RAM card promoted by Dell to allow socketed high-performance RAM on business laptops.

s3p 4y ago
Thank you for the citations with definitions :)
dijit 4y ago
> This is why Thunderbolt eGPUs don't work - the M1 and M2 can't actually write to the memory on the GPUs. I thought the major reason was simply lack of drivers. When I consider that I can use direct mapped M.2 storage over Thunderbolt; or use 100GBit network cards over Thunderbolt 4 (with a PCIe adapter) I wouldn't think that I was limited to mapping memory in the SoC.
- kmeisthax 4y ago
  I'm skipping over a few details here[0]. ARM has several different classes of memory access mappings; the normal one is called... well, Normal. I/O and storage is mapped using Device mappings, which don't cache reads and can be further divided into how little the CPU is allowed to try and optimize writes[1]. GPUs need Normal memory because applications written for modern graphics APIs expect to be able to map GPU memory into themselves and read and write to it like CPU memory. The ARM spec for I/O is that you are always allowed to use whatever mapping type the device needs, and that less-strict mappings should, in the worst case, "fall back" to stricter ones. Apple handles this differently; the SoC fabric requires you use the specific device mapping that it expects for a particular device, and if you try to use something looser or stricter than what it wants, it will drop the transaction and raise an exception. And of course the SoC fabric will not allow Normal memory reads or writes to hit a PCIe device. As far as the Asahi Linux team is aware, there isn't a way from the CPU to turn off this behavior. It's also not the only implementation of PCIe on ARM that locks out PCIe memory. Raspberry Pi 4's PCIe support[3] also has the same design flaw. If it was just a driver problem, someone would have ported AMDGPU to ARM and ran it on Asahi Linux by now, and we'd be posting cool benchmarks between the internal and external GPUs. You don't notice this problem for I/O or storage because those never need to be mapped as Normal. [0] And probably still skipping over more details, since I'm not an ARM expert. This is just what I've gleaned from reading other kernel developers' Mastodon and Twitter feeds. [1] Which, BTW, the M1 also screws up. You're supposed to be able to pick posted writes[2] or non-posted writes; Apple Silicon specifically refuses transaction types that don't match what the hardware expects. See https://lore.kernel.org/linux-arm-kernel/20210120132717.395873-4-mohamed.mediouni@caramail.com/ https://lore.kernel.org/linux-arm-kernel/20210120132717.3958... for more info. [2] Write instructions finish before the write is actually sent to the device. [3] Yes, it has PCIe. One lane of it, used to drive an external USB 3 controller. You can of course repurpose it for other things.
  - AceJohnny2 4y ago
    Considering the complexity of a SoC like the M1, that there is zero documentation for it, and the Asahi team can only reverse-engineer it, what are the odds that they just don't know the magic bits to twiddle to get this functionality?
    - csdvrx 4y ago
      > Considering the complexity of a SoC like the M1, that there is zero documentation for it, and the Asahi team can only reverse-engineer it, what are the odds that they just don't know the magic bits to twiddle to get this functionality? I'd consider that very low because you've got a lot of eyeballs trying to get the most out of a COTS easily available hardware. The goal is to make it the best-in-class Linux support. Personally, I believe it can be done. I follow their progress with great interest, as some of the non-technical limitations of the macbook (OLED and touchscreen) could easily be fixed.
      - smoldesu 4y ago
        Just because something is a COTS doesn't mean it will get great Linux support. The Nintendo Switch should be the best-supported Linux board availible by that logic, but alas, even an unlocked Tegra is only so capable. At a certain point, vendor cooperation becomes outright necessary. Asahi has come a long way, but getting to "best-in-class Linux support" takes more than community effort with commodity hardware. Stalled-out projects like Nouveau don't lend a lot of hope for the future.
trollied 4y ago
I was thinking that they'd go down the Risc PC route ( https://en.wikipedia.org/wiki/Risc_PC https://en.wikipedia.org/wiki/Risc_PC ) - allow expansion cards with both CPUs and memory on. Apple already trialled this with the accelerator cards in the current Mac Pro. Obviously there would be a penalty for off-die memory access, but I'm sure somebody cleverer than me could think of a nice way of architecting such a beast.
agloeregrets 4y ago
One of the thoughts on how a mac pro would work was to make the SOC/Memory combo an Add-in card and then use an Infinity Fabric-like interconnect on the motherboard with some nutty controller to allow clean intercommunication to split off instructions. By doing this, massively multithreaded workloads would not see any real impact...as long as you don't need to send anything card-to-card. Finally, (assuming the M2 Ultra maxes out at 192Gb ram ) this enables adding in 1 to maybe 6 M2 Ultras, topping out just over 1TB ram. This almost starts to work but will fail if your tasks can't be cleanly cut up into 192GB chunks, maybe that's the magic of that controller.

s3p 4y ago

Thank you for the citations with definitions :)

dijit 4y ago

> This is why Thunderbolt eGPUs don't work - the M1 and M2 can't actually write to the memory on the GPUs. I thought the major reason was simply lack of drivers. When I consider that I can use direct mapped M.2 storage over Thunderbolt; or use 100GBit network cards over Thunderbolt 4 (with a PCIe adapter) I wouldn't think that I was limited to mapping memory in the SoC.

kmeisthax 4y ago
I'm skipping over a few details here[0]. ARM has several different classes of memory access mappings; the normal one is called... well, Normal. I/O and storage is mapped using Device mappings, which don't cache reads and can be further divided into how little the CPU is allowed to try and optimize writes[1]. GPUs need Normal memory because applications written for modern graphics APIs expect to be able to map GPU memory into themselves and read and write to it like CPU memory. The ARM spec for I/O is that you are always allowed to use whatever mapping type the device needs, and that less-strict mappings should, in the worst case, "fall back" to stricter ones. Apple handles this differently; the SoC fabric requires you use the specific device mapping that it expects for a particular device, and if you try to use something looser or stricter than what it wants, it will drop the transaction and raise an exception. And of course the SoC fabric will not allow Normal memory reads or writes to hit a PCIe device. As far as the Asahi Linux team is aware, there isn't a way from the CPU to turn off this behavior. It's also not the only implementation of PCIe on ARM that locks out PCIe memory. Raspberry Pi 4's PCIe support[3] also has the same design flaw. If it was just a driver problem, someone would have ported AMDGPU to ARM and ran it on Asahi Linux by now, and we'd be posting cool benchmarks between the internal and external GPUs. You don't notice this problem for I/O or storage because those never need to be mapped as Normal. [0] And probably still skipping over more details, since I'm not an ARM expert. This is just what I've gleaned from reading other kernel developers' Mastodon and Twitter feeds. [1] Which, BTW, the M1 also screws up. You're supposed to be able to pick posted writes[2] or non-posted writes; Apple Silicon specifically refuses transaction types that don't match what the hardware expects. See https://lore.kernel.org/linux-arm-kernel/20210120132717.395873-4-mohamed.mediouni@caramail.com/ https://lore.kernel.org/linux-arm-kernel/20210120132717.3958... for more info. [2] Write instructions finish before the write is actually sent to the device. [3] Yes, it has PCIe. One lane of it, used to drive an external USB 3 controller. You can of course repurpose it for other things.
- AceJohnny2 4y ago
  Considering the complexity of a SoC like the M1, that there is zero documentation for it, and the Asahi team can only reverse-engineer it, what are the odds that they just don't know the magic bits to twiddle to get this functionality?
  - csdvrx 4y ago
    > Considering the complexity of a SoC like the M1, that there is zero documentation for it, and the Asahi team can only reverse-engineer it, what are the odds that they just don't know the magic bits to twiddle to get this functionality? I'd consider that very low because you've got a lot of eyeballs trying to get the most out of a COTS easily available hardware. The goal is to make it the best-in-class Linux support. Personally, I believe it can be done. I follow their progress with great interest, as some of the non-technical limitations of the macbook (OLED and touchscreen) could easily be fixed.
    - smoldesu 4y ago
      Just because something is a COTS doesn't mean it will get great Linux support. The Nintendo Switch should be the best-supported Linux board availible by that logic, but alas, even an unlocked Tegra is only so capable. At a certain point, vendor cooperation becomes outright necessary. Asahi has come a long way, but getting to "best-in-class Linux support" takes more than community effort with commodity hardware. Stalled-out projects like Nouveau don't lend a lot of hope for the future.

trollied 4y ago

I was thinking that they'd go down the Risc PC route ( https://en.wikipedia.org/wiki/Risc_PC https://en.wikipedia.org/wiki/Risc_PC ) - allow expansion cards with both CPUs and memory on. Apple already trialled this with the accelerator cards in the current Mac Pro. Obviously there would be a penalty for off-die memory access, but I'm sure somebody cleverer than me could think of a nice way of architecting such a beast.

agloeregrets 4y ago

One of the thoughts on how a mac pro would work was to make the SOC/Memory combo an Add-in card and then use an Infinity Fabric-like interconnect on the motherboard with some nutty controller to allow clean intercommunication to split off instructions. By doing this, massively multithreaded workloads would not see any real impact...as long as you don't need to send anything card-to-card. Finally, (assuming the M2 Ultra maxes out at 192Gb ram ) this enables adding in 1 to maybe 6 M2 Ultras, topping out just over 1TB ram. This almost starts to work but will fail if your tasks can't be cleanly cut up into 192GB chunks, maybe that's the magic of that controller.