3 ms·
One problem I've found specific to the Pi (and likely most of the other lower powered ARM chips out there, from what I've seen from Rock Pi, Pine boards, etc.)
by geerlingguy 6y ago
One problem I've found specific to the Pi (and likely most of the other lower powered ARM chips out there, from what I've seen from Rock Pi, Pine boards, etc.) is that the PCIe layer is not 100% supported on the hardware level as it should be if you want to support features like IO bar space (nonexistent on ARM).
On top of that, until the past couple years, there were only a few exotic ARM builds (probably mostly in custom or enterprise land) that even had standard PCIe slots, so many vendors didn't (and still don't) test their drivers—even if compiled for ARM—on any real hardware.
I hope this changes, especially since Macs now have ARM64, the Pi CM4 supports it, and some other inexpensive SBCs have 4x or even 8x lanes on the more expensive ones (is there an ARM motherboard out there with a 16x lane bus?).
For the Pi in particular, until the CM4 came out, very few people would ever get access to the PCIe bus on the regular Pi 4 model B, so only enough work was done on the kernel side to ensure the VL805 USB controller worked, and a few bits were just never fully implemented or tested (e.g. BAR allocations, MSI-X support, some memory access functions).
- CyberRabbi 6y agoYou mentioned work on the kernel side. Is this entirely a hardware issue or are the Linux pcie drivers for the Pi also lacking functionality?
- geerlingguy 6y agoIt's a mix, and from my work and discussions on getting a GPU running (or not...), it seems to make debugging even harder if I can't figure out if it's bad memory access or an actual hardware feature that's just not there. I've been tracking progress on this site / repo: https://pipci.jeffgeerling.com/ https://pipci.jeffgeerling.com/
- monocasa 6y agoHave you talked to broadcom about the PCIe IO space problem? Most other RISCs handle IO space by just mapping into an MMIO space instead. PowerPC, MIPS, and other ARM SoCs have been handling this for decades at this point. It feels weird that their root complex would choke on that, and more likely that there's some unknown config knock you have to setup that hasn't been documented properly. I feel like you might have enough of a publicity reach to get an actual answer on this one.
- my123 6y ago> there an ARM motherboard out there with a 16x lane bus? For servers at sky-high prices only. For $699, your closest choice might be the NVIDIA Jetson AGX Xavier... PCIe 4.0 but with a x8 slot, and an NVMe M.2 x4 PCIe 4.0 slot too.
- floatboth 6y agoSolidRun HoneyComb has about the same slots and price, but instead of weird nvidia closedness you get first-class upstream EDK2 support, everything is about plug & play with ACPI.
- my123 6y agoThe HoneyComb is at PCIe 3.0. Sadly it's at ARMv8.0 (w/ Cortex-A72) so that removes it from consideration totally for me. And the Xavier has an official-but-still-beta EDK2 firmware available too, which includes ACPI support and can even boot Windows. (https://developer.nvidia.com/embedded/downloads#?search=UEFI&cl=date,-1 https://developer.nvidia.com/embedded/downloads#?search=UEFI...) Yes, GPU 3D acceleration is currently not supported there yet, but the LX2K doesn't have a GPU at all anyway...