4 ms·

Getting the RK3588 SoC supported in upstream Linux (2023)

86 points by transpute 3y ago

madduci 3y ago
Great presentation..one of the major issues with the support of RK3588 is that the GPU doesn't have proper drivers, so GPU acceleration relies on patches made by small contributors and these are obviously not updated regularly, so they tend to be obsolete easily.
- not_your_vase 3y ago
  A story as old as the embedded ARM ecosystem itself :(
  - jsheard 3y ago
    AMD is probably the best hope nowadays, they've started licensing their RDNA GPU architecture out to ARM SOC vendors and they already have excellent open source drivers. Mali and Adreno are still much more common though, especially on cheap SBCs which nearly always use Mali.
    - my123 3y ago
      I hoped so but nope. Exynos with RDNA has closed source user-space, with AMD/Samsung not making public the differences, and no attempt at upstreaming the kernel-side ones too. The best mobile platform on Arm with open-source GPU drivers rn is Qualcomm...
- mort96 3y ago
  Either that, or using the proprietary Mali drivers which rely on userspace X11 GPU drivers which don't work in current versions of X or current kernels so you end up running 5.10 using a separate x11-legacy package. ARM GPUs on Linux are truly in a terrible state right now. I hope ARM gets serious about ensuring that upstream supports their GPUs soon after release.
  - snvzz 3y ago
    >I hope ARM gets serious about ensuring that upstream supports their GPUs soon after release. Ironically, Imagination Technologies of PowerVR fame (infamy?) did, with a funded mesa3d driver effort. And embraced RISC-V while at it.
  - zokier 3y ago
    At least ARM is not completely antagonist anymore and is actually funding some this effort, that is already pretty good step forwards. But yes, there is long way to go for everyone here. https://newsroom.arm.com/news/arm-expands-open-source-partnerships https://newsroom.arm.com/news/arm-expands-open-source-partne...
    - mort96 3y ago
      I've seen that too, I'm (very) cautiously optimistic. Ideally, ARM would be doing a bunch of the development work and actually ensure that the driver support is ready to be upstreamed on day 1 (like what AMD pretty much does AFAIU). However, the Mali-G610 (used in the RK3588) is still not supported despite being released in 2021. ARM's "work with Collabora" started before the release of the Mali-G610, so the fact that we're so far from upstream support even after this many years of development seems like a bad sign. I don't understand why future Mali architectures wouldn't also take 4+ years unless ARM really really steps it up.
      - throw354279 3y ago
        > I don't understand why future Mali architectures wouldn't also take 4+ years unless ARM really really steps it up. G610 (and the larger G710) are sort of special since there was a major change to the internal architecture and how work is submitted to the GPUs. Later versions didn't have such a big, breaking change.
- zokier 3y ago
  The situation doesn't seem quite so dire, Collabora is apparently working on driver for this now: https://www.phoronix.com/news/Panthor-DRM-Newer-Mali https://www.phoronix.com/news/Panthor-DRM-Newer-Mali They have pretty decent track record in producing these sort of drivers so there is a chance
- mntmn 3y ago
  This is not true, the panthor driver (previously pancsf) for mesa is actively being developed and funded, and already works quite well (I had a demo desktop running on prototye MNT Reform Next laptop at 37c3).
  - mort96 3y ago
    I feel like "already" is a bit strong considering the GPU has been out for almost 3 years and it's still seemingly far from upstream.
    - imtringued 3y ago
      And the main panfrost developer Alyssa Rosenzweig has moved on, so the pace of development has declined significantly. At this point the Raspberry Pi 5 is more than capable and the Rock 5 has lost the primary advantage it once had.
      - fooishbar 3y ago
        It's nothing to do with that really. It's much more to do with the gen10 (Gxxx) Mali requiring a completely new kernel driver. Given that nouveau and other drivers required the same thing, this ended up being a long diversion into a lot of new common DRM infrastructure to make things easier for the next drivers who need to do the same thing. By comparison, enabling gen9 in the kernel was more like adding a couple of device IDs. See bbrezillon & dakr's talk here for more details: https://indico.freedesktop.org/event/4/contributions/181/ https://indico.freedesktop.org/event/4/contributions/181/
dd_xplore 3y ago
Joshua makes some good improvement and upstreams for this SOC https://github.com/Joshua-Riek/ubuntu-rockchip https://github.com/Joshua-Riek/ubuntu-rockchip
- cmrdporcupine 3y ago
  Very cool, I wonder if I could get that working for the FriendlyElec CM3588 board I have here. Hm.
  - dd_xplore 3y ago
    Yes, I'm happy cuz I finally got a updated distro. P.S I have a NanoPi R6C.
    - cmrdporcupine 3y ago
      I ended up doing surgery and using the provided debian install but moving /usr and /var and /home over to nVME SSD (from SDCard) but still booting from the card. The debian itself seems mostly fine and packages upgradeable etc. But kernel, yeah that's another story
cmrdporcupine 3y ago
I have a little RK3588 (FriendlyElec CM3588 "NAS" kit) board here and it's awesome. 4 n.VME M2 slots, 2.5gbps Ethernet. 8 cores. It replaced a 2011 Mac Mini as local Plex media server, photo storage/backup, etc. and it draws 3.5 Watts idle vs 7-10 on the Mini. It's running fanless for weeks now, and sitting in an old wooden Cuban cigar box (lid open). But yes, I ventured into... building the kernel & image ... and like a lot of these systems it was a nightmare of brittle shell & Python scripts, patches, many steps, and multiple, multiple READMEs. And because of its origin in the world of phone/media-box SoCs, the kernel defaults set up include annoying things like overriden highly-reduced VmMalloc limit (I work on my own DB component that uses VM overcommit at very large sizes) which I can't find the right knob to fix, etc. etc. But really, this is a very promising board. I hope to see more boards like this that lean into more "PC-class" capabilities than the usual RPi type sytems.
- rcarmo 3y ago
  I went down this rabbit hole just the other week: https://taoofmac.com/space/blog/2024/02/10/2000 https://taoofmac.com/space/blog/2024/02/10/2000 I managed to get recovery booting but my rootfs didn't finish loading, and I still haven't figured out why. And I am no novice to these things (I currently have around half a dozen RK3588/3588S boards around my desk as I do some advisory work on IoT projects, and right now my strongest recommendations are all Armbian-compatible).
p0w3n3d 3y ago
I don't get it. 1. Rockchip sells a chip in some boards (let's focus on one, e.g. in Orange Pi 5) 2. Orange Pi 5 has some software pre-installed (I guess linux right? even if it's android then it's still linux, right?) 3. Some people need to find out by themselves how to make normal upstream linux running on this chip Questions: A. How the system provided by Orange Pi works on the RK3588? Does it not use all the power/features? B. Why Rockchip only "suggests" how to handle problems instead of fixing it? C. Who and why created the "OEM" linux that has been installed on Orange Pi 5 for selling? C.1. Why the people who created it won't release their work into linux?
- mcpherrinm 3y ago
  They will ship images using a probably old kernel with various patches and drivers which may not be in suitable shape for simply including upstream. The vendor may ship this initially, but often won't work on upstreaming the changes or porting to newer kernels, so you end up with devices stuck running forks of old kernels unless somebody takes the kind of effort discussed here to upstream the changes.
  - p0w3n3d 3y ago
    or unless vulnerabilities of old kernels will allow someone to take them over
- mafuyu 3y ago
  This is a common story in ARM / embedded linux. SoC vendors’ main focus is to make the chip and give it enough BSP support for customers to buy it and build their products around it. The bare minimum for this is typically a set of hacky kernel patches targeting a giver kernel version. The customer then needs to figure out what they want to do: Take the kernel blob as-is? Get the vendor’s kernel source and try and modify it for their specific needs? Rebase it to a newer kernel? The customer typically wants to just get enough support to ship their product; they’re not in the business of maintaining some other company’s kernel patches. They’ll do it if they have to, but the engineers will grumble about it the entire time. Thus, getting support landed into Linux upstream is primarily the task of the chip vendor. There’s a good business case for it, as better and widely available software support will help sell the chip. But it’s also incredibly expensive to write all that kernel code and take it through the process of upstreaming. A lot of this stuff won’t meet the standards of the Linux kernel without lots of work. The ARM SBC market flips this, where you have a bunch of enthusiasts, and then a smaller market of businesses wanting to use them in their products. That can provide enough momentum for a community-driven upstreaming effort.

86 points by transpute 3y ago

Great presentation..one of the major issues with the support of RK3588 is that the GPU doesn't have proper drivers, so GPU acceleration relies on patches made by small contributors and these are obviously not updated regularly, so they tend to be obsolete easily.

not_your_vase 3y ago
A story as old as the embedded ARM ecosystem itself :(
- jsheard 3y ago
  AMD is probably the best hope nowadays, they've started licensing their RDNA GPU architecture out to ARM SOC vendors and they already have excellent open source drivers. Mali and Adreno are still much more common though, especially on cheap SBCs which nearly always use Mali.
  - my123 3y ago
    I hoped so but nope. Exynos with RDNA has closed source user-space, with AMD/Samsung not making public the differences, and no attempt at upstreaming the kernel-side ones too. The best mobile platform on Arm with open-source GPU drivers rn is Qualcomm...
mort96 3y ago
Either that, or using the proprietary Mali drivers which rely on userspace X11 GPU drivers which don't work in current versions of X or current kernels so you end up running 5.10 using a separate x11-legacy package. ARM GPUs on Linux are truly in a terrible state right now. I hope ARM gets serious about ensuring that upstream supports their GPUs soon after release.
- snvzz 3y ago
  >I hope ARM gets serious about ensuring that upstream supports their GPUs soon after release. Ironically, Imagination Technologies of PowerVR fame (infamy?) did, with a funded mesa3d driver effort. And embraced RISC-V while at it.
- zokier 3y ago
  At least ARM is not completely antagonist anymore and is actually funding some this effort, that is already pretty good step forwards. But yes, there is long way to go for everyone here. https://newsroom.arm.com/news/arm-expands-open-source-partnerships https://newsroom.arm.com/news/arm-expands-open-source-partne...
  - mort96 3y ago
    I've seen that too, I'm (very) cautiously optimistic. Ideally, ARM would be doing a bunch of the development work and actually ensure that the driver support is ready to be upstreamed on day 1 (like what AMD pretty much does AFAIU). However, the Mali-G610 (used in the RK3588) is still not supported despite being released in 2021. ARM's "work with Collabora" started before the release of the Mali-G610, so the fact that we're so far from upstream support even after this many years of development seems like a bad sign. I don't understand why future Mali architectures wouldn't also take 4+ years unless ARM really really steps it up.
    - throw354279 3y ago
      > I don't understand why future Mali architectures wouldn't also take 4+ years unless ARM really really steps it up. G610 (and the larger G710) are sort of special since there was a major change to the internal architecture and how work is submitted to the GPUs. Later versions didn't have such a big, breaking change.
zokier 3y ago
The situation doesn't seem quite so dire, Collabora is apparently working on driver for this now: https://www.phoronix.com/news/Panthor-DRM-Newer-Mali https://www.phoronix.com/news/Panthor-DRM-Newer-Mali They have pretty decent track record in producing these sort of drivers so there is a chance
mntmn 3y ago
This is not true, the panthor driver (previously pancsf) for mesa is actively being developed and funded, and already works quite well (I had a demo desktop running on prototye MNT Reform Next laptop at 37c3).
- mort96 3y ago
  I feel like "already" is a bit strong considering the GPU has been out for almost 3 years and it's still seemingly far from upstream.
  - imtringued 3y ago
    And the main panfrost developer Alyssa Rosenzweig has moved on, so the pace of development has declined significantly. At this point the Raspberry Pi 5 is more than capable and the Rock 5 has lost the primary advantage it once had.
    - fooishbar 3y ago
      It's nothing to do with that really. It's much more to do with the gen10 (Gxxx) Mali requiring a completely new kernel driver. Given that nouveau and other drivers required the same thing, this ended up being a long diversion into a lot of new common DRM infrastructure to make things easier for the next drivers who need to do the same thing. By comparison, enabling gen9 in the kernel was more like adding a couple of device IDs. See bbrezillon & dakr's talk here for more details: https://indico.freedesktop.org/event/4/contributions/181/ https://indico.freedesktop.org/event/4/contributions/181/

dd_xplore 3y ago

Joshua makes some good improvement and upstreams for this SOC https://github.com/Joshua-Riek/ubuntu-rockchip https://github.com/Joshua-Riek/ubuntu-rockchip

cmrdporcupine 3y ago
Very cool, I wonder if I could get that working for the FriendlyElec CM3588 board I have here. Hm.
- dd_xplore 3y ago
  Yes, I'm happy cuz I finally got a updated distro. P.S I have a NanoPi R6C.
  - cmrdporcupine 3y ago
    I ended up doing surgery and using the provided debian install but moving /usr and /var and /home over to nVME SSD (from SDCard) but still booting from the card. The debian itself seems mostly fine and packages upgradeable etc. But kernel, yeah that's another story

cmrdporcupine 3y ago

I have a little RK3588 (FriendlyElec CM3588 "NAS" kit) board here and it's awesome. 4 n.VME M2 slots, 2.5gbps Ethernet. 8 cores. It replaced a 2011 Mac Mini as local Plex media server, photo storage/backup, etc. and it draws 3.5 Watts idle vs 7-10 on the Mini. It's running fanless for weeks now, and sitting in an old wooden Cuban cigar box (lid open). But yes, I ventured into... building the kernel & image ... and like a lot of these systems it was a nightmare of brittle shell & Python scripts, patches, many steps, and multiple, multiple READMEs. And because of its origin in the world of phone/media-box SoCs, the kernel defaults set up include annoying things like overriden highly-reduced VmMalloc limit (I work on my own DB component that uses VM overcommit at very large sizes) which I can't find the right knob to fix, etc. etc. But really, this is a very promising board. I hope to see more boards like this that lean into more "PC-class" capabilities than the usual RPi type sytems.

rcarmo 3y ago
I went down this rabbit hole just the other week: https://taoofmac.com/space/blog/2024/02/10/2000 https://taoofmac.com/space/blog/2024/02/10/2000 I managed to get recovery booting but my rootfs didn't finish loading, and I still haven't figured out why. And I am no novice to these things (I currently have around half a dozen RK3588/3588S boards around my desk as I do some advisory work on IoT projects, and right now my strongest recommendations are all Armbian-compatible).

p0w3n3d 3y ago

I don't get it. 1. Rockchip sells a chip in some boards (let's focus on one, e.g. in Orange Pi 5) 2. Orange Pi 5 has some software pre-installed (I guess linux right? even if it's android then it's still linux, right?) 3. Some people need to find out by themselves how to make normal upstream linux running on this chip Questions: A. How the system provided by Orange Pi works on the RK3588? Does it not use all the power/features? B. Why Rockchip only "suggests" how to handle problems instead of fixing it? C. Who and why created the "OEM" linux that has been installed on Orange Pi 5 for selling? C.1. Why the people who created it won't release their work into linux?

mcpherrinm 3y ago
They will ship images using a probably old kernel with various patches and drivers which may not be in suitable shape for simply including upstream. The vendor may ship this initially, but often won't work on upstreaming the changes or porting to newer kernels, so you end up with devices stuck running forks of old kernels unless somebody takes the kind of effort discussed here to upstream the changes.
- p0w3n3d 3y ago
  or unless vulnerabilities of old kernels will allow someone to take them over
mafuyu 3y ago
This is a common story in ARM / embedded linux. SoC vendors’ main focus is to make the chip and give it enough BSP support for customers to buy it and build their products around it. The bare minimum for this is typically a set of hacky kernel patches targeting a giver kernel version. The customer then needs to figure out what they want to do: Take the kernel blob as-is? Get the vendor’s kernel source and try and modify it for their specific needs? Rebase it to a newer kernel? The customer typically wants to just get enough support to ship their product; they’re not in the business of maintaining some other company’s kernel patches. They’ll do it if they have to, but the engineers will grumble about it the entire time. Thus, getting support landed into Linux upstream is primarily the task of the chip vendor. There’s a good business case for it, as better and widely available software support will help sell the chip. But it’s also incredibly expensive to write all that kernel code and take it through the process of upstreaming. A lot of this stuff won’t meet the standards of the Linux kernel without lots of work. The ARM SBC market flips this, where you have a bunch of enthusiasts, and then a smaller market of businesses wanting to use them in their products. That can provide enough momentum for a community-driven upstreaming effort.