18 ms·

The Linux graphics stack in a nutshell

274 points by zorgmonkey 3y ago

anArbitraryOne 3y ago
[flagged]
- PaulDavisThe1st 3y ago
  Wanna know about acorns? No? Well, in a nutshell, they're an oak tree.
skullone 3y ago
Is systemd going to add it's own DRI layer?
- feitingen 3y ago
  I really hope not.
userbinator 3y ago
This is about 3D rendering, to be precise; I believe 2D acceleration goes through the same lower layers but the higher ones are very different. Incidentally, one thing I noticed when I was trying to port Linux GPU drivers to Windows some time ago is what appeared to be an excessive amount of indirection; there are so many layers and places where things could be simpler.
- Jasper_ 3y ago
  The most viable approach these days is for 2D is to use the 3D hardware. There's no standard, usable API for 2D accelerated drawing the way there is for 3D, nor does it quite make sense for there to be one. (No, OpenVG is not viable. No, Xrender is not viable. cairo and Skia both use the 3D hardware in combination with a CPU render engine.)
  - pcwalton 3y ago
    For the most part that's true, but simple 2D compositing is a bit of a different beast, because it can sometimes be done at scanout time, saving a blit. Last I checked, (non-Android) Linux rarely makes use of this except for the mouse cursor. But in general you can save a good bit of energy and memory bandwidth on HiDPI displays if you try to use 2D hardware layers where you can. You can virtually never use them for the UI itself, because they're far too limited, but the windowing system can often use them to composite windows together. It'd be nice if Wayland compositors made more use of this, e.g. to avoid having to blit the foreground window every frame.
    - kimixa 3y ago
      I used to work on mobile graphics and the android HWC stack. The scanout-time hardware was often less useful that you might think - only in dynamic scenes where the GPU is otherwise idle (like playing video possibly with a static UI overlay was the premier use case). For static scenes it's more efficient to render out to a buffer (using the GPU as the scanout overlay pipes often had limited feedback capability) and just output that using overlays disabled. It didn't take many frames for that to be worth it. For apps that were animating or otherwise updating it's window, most UI toolkits used the GPU for widget rendering. And often the scanout pipes didn't hook into the (relatively large) system caches like the GPU did, so there were times it was again faster to composite the screen on the GPU to a single scanout buffer than flush already cached data, the get the scanout hardware to read it back from the memory bus. And there weren't as cheap as people thought - one stat I remember was that the total area of the GPU on the omap4 platform was smaller than the display pipes. Though that is now a pretty old chip, and always had a bit of focus on "multimedia".
      - deleted 3y ago
        [deleted]
      - kllrnohj 3y ago
        I think your information is quite outdated. The HWC overlay planes are heavily used, you can see this trivially just doing a 'dumpsys SurfaceFlinger' or grabbing a systrace/perfetto trace. When it falls back to GPU composition it's very obvious as there's a significant hit to latency and more GPU contention. The overlay capabilities of the modern Snapdragons are also quite absurd. They support like upwards of a dozen overlays now and even have FP16 extended sRGB support. Some HWCs (like the one in the steam deck) even have per plane 3D LUTs for HDR tone mapping (ex https://github.com/ValveSoftware/gamescope/blob/master/src/docs/Steam%20Deck%20Display%20Pipeline.png https://github.com/ValveSoftware/gamescope/blob/master/src/d... ) The composition is bandwidth heavy of course, but for static scenes there's a cache after the HWC in the form of panel self refresh.
- pcwalton 3y ago
  2D acceleration is generally done through the same APIs, specifically OpenGL and Vulkan. Classically, the X compositor would use the GLX_EXT_texture_from_pixmap extension to import an X pixmap representing a window surface into OpenGL, where it can be used like any other texture. For the Wayland compositor, I believe you'd use EGL_WL_bind_wayland_display to bind a Wayland surface to an EGLImage, and then glEGLImageTargetTexture2DOES (can't believe I have that function name memorized) to bind that EGLImage to an OpenGL texture, where it can be used in the same way. Vulkan has similar extensions. On the client side, I think most Linux apps still draw their UIs on CPU, usually accelerated with SIMD. Firefox and Chrome (I think SkiaGL is enabled on Linux?) are exceptions; they use OpenGL and/or Vulkan to draw their UI. Video playback is a different beast and in theory relies on vendor-specific extensions to decode the video in hardware. However, the last time I looked at Linux video decoding (which was years ago), the drivers were awful and interfacing with each vendor's APIs was a huge pain, and so most apps just did video decoding on CPU. (Besides, the Linux ecosystem prefers open codecs, and hardware has only recently gotten support for non-patent-encumbered video formats.)
  - okanat 3y ago
    For client side Qt has good GPU support but only for QML. All QML is drawn on GPU by default (expect text I think, which uses Harfbuzz) but all Qt Widgets are drawn on CPU. However things like KDE's Wayland uses direct OpenGL calls for faster composition. Firefox has Web Render running on top of ANGLE which is a generic OpenGL layer that converts the OpenGL calls into native platform calls. ANGLE is a Google project and it is the base library for Skia which is used by Chromium to render everything. IIRC Qt / QML also uses ANGLE for Windows.
  - MBCook 3y ago
    Why are toolkits still rendered on the CPU?
    - pcwalton 3y ago
      It's a ton of effort to write a GPU vector renderer that's both compatible with existing apps and faster than the CPU. Switching to SkiaGL would probably be the easiest approach to migrate to GPU rendering, but Skia is notoriously difficult to use outside of Google's codebases. (The running joke being "the recommended way to build Skia is to get a job at Google, but there are some workarounds available if for some reason that isn't practical.")
- sonicanatidae 3y ago
  Was the complexity dedicated to backwards compat or just needlessly complex for reasons sane people will never understand?
MBCook 3y ago
Before the Wayland complaints take over the thread, I’d like to post a link to a very short thread by Drew DeVault. https://fosstodon.org/@drewdevault/111607882208898175 https://fosstodon.org/@drewdevault/111607882208898175 Here are the two important posts: ——— The story of Wayland: 1. No one wanted to maintain X11 because it sucked 2. We made Wayland and it's much better 3. A vocal minority of change-averse people complained with little to no factual basis 4. They were asked to muster some labor to maintain X11 5. None of them did 6. All of the people who actually do get work done eventually stopped listening to them and moved on with Wayland —— Some of these detractors built a tottering pile of godawful hacks on top of X11 where every piece depends on another critical design flaw of X11 and are upset that by fixing all of these design flaws their pile of hacks fell over when no one wanted to maintain the load bearing side of their hacks
- deleted 3y ago
  [deleted]
- bee_rider 3y ago
  In the thread about Firefox switching defaults to Wayland, there were some complaints about some accessibility software not being supported by Wayland. If the “tottering pile of godawful hacks” is required to not exclude blind people, it doesn’t seem that godawful… Personally I’d prefer to use Sway, but last time I tried Zoom on Sway it gave me a lot of trouble. X11 might not be getting much future development, but it is done and it works, so who cares? It can just stay the same in perpetuity for all I care as long as it keeps working.
  - shmerl 3y ago
    Does Zoom even support Wayland or you are running it through XWayland? All these proprietary clients usually have a lot of inertia with implementing Wayland support.
    - bee_rider 3y ago
      I haven’t the slightest clue, it is a terrible program and I just wanted to do the minimal to get it working. Switching to X11 meant I was able to waste fewer brain-cycles thinking about Zoom.
charcircuit 3y ago
The scene graph isn't part of the renderer any more than a player object that contains the player's location. The scene graph's purpose is to make updating transforms efficient. Just because it references transforms that may need to be sent to the GPU, that doesn't mean it is part of the renderer.
- kllrnohj 3y ago
  Yes and no. Some type of deferred structure is almost always part of a GPU renderer as it's necessary for batching and reordering which help performance tremendously. Sometimes this is entirely an internal system behind an imperitive API, though. Like skia only offers an imperitive API even though it builds a deferred rendering structure from that under the covers. So you end up doing a scene graph to imperitive API to internal renderer scene graph in some UI toolkits. In others they may share the same scene graph structure, applying those optimizations directly from the initial graph. Although all that said a "pure" scene graph is often overkill and slow. They were all the rage in the early 2000s but less so these days. QML's QtQuick looks like the primary remaining example?
Already__Taken 3y ago
my Google Fu has failed me but isn't there a simple way to compile rust or go to a Linux os that boots to a gui? lots of embedded talks about tiny hardware and framebuffers etc. I can throw hardware around, it's UX/dx I just want ssh in the background and a UI Infront.
- leonheld 3y ago
  I think the closest to this is https://doc.qt.io/Boot2Qt/ https://doc.qt.io/Boot2Qt/.
- c0balt 3y ago
  Yesn't, there aren't many tools for it. The closest you will get for a generic GUI app is various window managers/ DM for Wayland/ X11 that run in a "kiosk mode" + autologin. This way the OS (is supposed to) just boots and opens your app in effectively Fullscreen.
  - Already__Taken 3y ago
    That did help me find https://github.com/cage-kiosk/cage/wiki https://github.com/cage-kiosk/cage/wiki initially I'm here there's other going down that path. Thanks
- ogoffart 3y ago
  You can do that with Slint (https://slint.dev https://slint.dev) and its linuxkms backend. No need for a xorg server or wayland compositor, just run the application made with Slint from the init script.
  - yencabulator 3y ago
    Beware Slint's license, it's either proprietary or GPLv3, or whatever this kooky thing is that doesn't allow mobile apps: https://github.com/slint-ui/slint/blob/master/LICENSES/LicenseRef-Slint-Royalty-free-1.1.md https://github.com/slint-ui/slint/blob/master/LICENSES/Licen...
lfmunoz4 3y ago
Could barely understand this article. Is it just me or can none explain the graphics stack in a sensible way?
- rijoja 3y ago
  Neither, it's quite a huge subject, so if you put in the time you'll surely be able to figure it out! I don't think that the article isn't sensible, from what I saw it seemed quite well-written and the site usually has good quality content. The thing is that it targets readers with a certain knowledge-level. It's difficult for me to tell how much you know about graphics programming in general, so I can't really point you to any resources that would be better in your particular case!

274 points by zorgmonkey 3y ago

anArbitraryOne 3y ago

[flagged]

PaulDavisThe1st 3y ago
Wanna know about acorns? No? Well, in a nutshell, they're an oak tree.

skullone 3y ago

Is systemd going to add it's own DRI layer?

feitingen 3y ago
I really hope not.

userbinator 3y ago

This is about 3D rendering, to be precise; I believe 2D acceleration goes through the same lower layers but the higher ones are very different. Incidentally, one thing I noticed when I was trying to port Linux GPU drivers to Windows some time ago is what appeared to be an excessive amount of indirection; there are so many layers and places where things could be simpler.

Jasper_ 3y ago
The most viable approach these days is for 2D is to use the 3D hardware. There's no standard, usable API for 2D accelerated drawing the way there is for 3D, nor does it quite make sense for there to be one. (No, OpenVG is not viable. No, Xrender is not viable. cairo and Skia both use the 3D hardware in combination with a CPU render engine.)
- pcwalton 3y ago
  For the most part that's true, but simple 2D compositing is a bit of a different beast, because it can sometimes be done at scanout time, saving a blit. Last I checked, (non-Android) Linux rarely makes use of this except for the mouse cursor. But in general you can save a good bit of energy and memory bandwidth on HiDPI displays if you try to use 2D hardware layers where you can. You can virtually never use them for the UI itself, because they're far too limited, but the windowing system can often use them to composite windows together. It'd be nice if Wayland compositors made more use of this, e.g. to avoid having to blit the foreground window every frame.
  - kimixa 3y ago
    I used to work on mobile graphics and the android HWC stack. The scanout-time hardware was often less useful that you might think - only in dynamic scenes where the GPU is otherwise idle (like playing video possibly with a static UI overlay was the premier use case). For static scenes it's more efficient to render out to a buffer (using the GPU as the scanout overlay pipes often had limited feedback capability) and just output that using overlays disabled. It didn't take many frames for that to be worth it. For apps that were animating or otherwise updating it's window, most UI toolkits used the GPU for widget rendering. And often the scanout pipes didn't hook into the (relatively large) system caches like the GPU did, so there were times it was again faster to composite the screen on the GPU to a single scanout buffer than flush already cached data, the get the scanout hardware to read it back from the memory bus. And there weren't as cheap as people thought - one stat I remember was that the total area of the GPU on the omap4 platform was smaller than the display pipes. Though that is now a pretty old chip, and always had a bit of focus on "multimedia".
    - deleted 3y ago
      [deleted]
    - kllrnohj 3y ago
      I think your information is quite outdated. The HWC overlay planes are heavily used, you can see this trivially just doing a 'dumpsys SurfaceFlinger' or grabbing a systrace/perfetto trace. When it falls back to GPU composition it's very obvious as there's a significant hit to latency and more GPU contention. The overlay capabilities of the modern Snapdragons are also quite absurd. They support like upwards of a dozen overlays now and even have FP16 extended sRGB support. Some HWCs (like the one in the steam deck) even have per plane 3D LUTs for HDR tone mapping (ex https://github.com/ValveSoftware/gamescope/blob/master/src/docs/Steam%20Deck%20Display%20Pipeline.png https://github.com/ValveSoftware/gamescope/blob/master/src/d... ) The composition is bandwidth heavy of course, but for static scenes there's a cache after the HWC in the form of panel self refresh.
pcwalton 3y ago
2D acceleration is generally done through the same APIs, specifically OpenGL and Vulkan. Classically, the X compositor would use the GLX_EXT_texture_from_pixmap extension to import an X pixmap representing a window surface into OpenGL, where it can be used like any other texture. For the Wayland compositor, I believe you'd use EGL_WL_bind_wayland_display to bind a Wayland surface to an EGLImage, and then glEGLImageTargetTexture2DOES (can't believe I have that function name memorized) to bind that EGLImage to an OpenGL texture, where it can be used in the same way. Vulkan has similar extensions. On the client side, I think most Linux apps still draw their UIs on CPU, usually accelerated with SIMD. Firefox and Chrome (I think SkiaGL is enabled on Linux?) are exceptions; they use OpenGL and/or Vulkan to draw their UI. Video playback is a different beast and in theory relies on vendor-specific extensions to decode the video in hardware. However, the last time I looked at Linux video decoding (which was years ago), the drivers were awful and interfacing with each vendor's APIs was a huge pain, and so most apps just did video decoding on CPU. (Besides, the Linux ecosystem prefers open codecs, and hardware has only recently gotten support for non-patent-encumbered video formats.)
- okanat 3y ago
  For client side Qt has good GPU support but only for QML. All QML is drawn on GPU by default (expect text I think, which uses Harfbuzz) but all Qt Widgets are drawn on CPU. However things like KDE's Wayland uses direct OpenGL calls for faster composition. Firefox has Web Render running on top of ANGLE which is a generic OpenGL layer that converts the OpenGL calls into native platform calls. ANGLE is a Google project and it is the base library for Skia which is used by Chromium to render everything. IIRC Qt / QML also uses ANGLE for Windows.
- MBCook 3y ago
  Why are toolkits still rendered on the CPU?
  - pcwalton 3y ago
    It's a ton of effort to write a GPU vector renderer that's both compatible with existing apps and faster than the CPU. Switching to SkiaGL would probably be the easiest approach to migrate to GPU rendering, but Skia is notoriously difficult to use outside of Google's codebases. (The running joke being "the recommended way to build Skia is to get a job at Google, but there are some workarounds available if for some reason that isn't practical.")
sonicanatidae 3y ago
Was the complexity dedicated to backwards compat or just needlessly complex for reasons sane people will never understand?

MBCook 3y ago

Before the Wayland complaints take over the thread, I’d like to post a link to a very short thread by Drew DeVault. https://fosstodon.org/@drewdevault/111607882208898175 https://fosstodon.org/@drewdevault/111607882208898175 Here are the two important posts: ——— The story of Wayland: 1. No one wanted to maintain X11 because it sucked 2. We made Wayland and it's much better 3. A vocal minority of change-averse people complained with little to no factual basis 4. They were asked to muster some labor to maintain X11 5. None of them did 6. All of the people who actually do get work done eventually stopped listening to them and moved on with Wayland —— Some of these detractors built a tottering pile of godawful hacks on top of X11 where every piece depends on another critical design flaw of X11 and are upset that by fixing all of these design flaws their pile of hacks fell over when no one wanted to maintain the load bearing side of their hacks

deleted 3y ago
[deleted]
bee_rider 3y ago
In the thread about Firefox switching defaults to Wayland, there were some complaints about some accessibility software not being supported by Wayland. If the “tottering pile of godawful hacks” is required to not exclude blind people, it doesn’t seem that godawful… Personally I’d prefer to use Sway, but last time I tried Zoom on Sway it gave me a lot of trouble. X11 might not be getting much future development, but it is done and it works, so who cares? It can just stay the same in perpetuity for all I care as long as it keeps working.
- shmerl 3y ago
  Does Zoom even support Wayland or you are running it through XWayland? All these proprietary clients usually have a lot of inertia with implementing Wayland support.
  - bee_rider 3y ago
    I haven’t the slightest clue, it is a terrible program and I just wanted to do the minimal to get it working. Switching to X11 meant I was able to waste fewer brain-cycles thinking about Zoom.

charcircuit 3y ago

The scene graph isn't part of the renderer any more than a player object that contains the player's location. The scene graph's purpose is to make updating transforms efficient. Just because it references transforms that may need to be sent to the GPU, that doesn't mean it is part of the renderer.

kllrnohj 3y ago
Yes and no. Some type of deferred structure is almost always part of a GPU renderer as it's necessary for batching and reordering which help performance tremendously. Sometimes this is entirely an internal system behind an imperitive API, though. Like skia only offers an imperitive API even though it builds a deferred rendering structure from that under the covers. So you end up doing a scene graph to imperitive API to internal renderer scene graph in some UI toolkits. In others they may share the same scene graph structure, applying those optimizations directly from the initial graph. Although all that said a "pure" scene graph is often overkill and slow. They were all the rage in the early 2000s but less so these days. QML's QtQuick looks like the primary remaining example?

Already__Taken 3y ago

my Google Fu has failed me but isn't there a simple way to compile rust or go to a Linux os that boots to a gui? lots of embedded talks about tiny hardware and framebuffers etc. I can throw hardware around, it's UX/dx I just want ssh in the background and a UI Infront.

leonheld 3y ago
I think the closest to this is https://doc.qt.io/Boot2Qt/ https://doc.qt.io/Boot2Qt/.
c0balt 3y ago
Yesn't, there aren't many tools for it. The closest you will get for a generic GUI app is various window managers/ DM for Wayland/ X11 that run in a "kiosk mode" + autologin. This way the OS (is supposed to) just boots and opens your app in effectively Fullscreen.
- Already__Taken 3y ago
  That did help me find https://github.com/cage-kiosk/cage/wiki https://github.com/cage-kiosk/cage/wiki initially I'm here there's other going down that path. Thanks
ogoffart 3y ago
You can do that with Slint (https://slint.dev https://slint.dev) and its linuxkms backend. No need for a xorg server or wayland compositor, just run the application made with Slint from the init script.
- yencabulator 3y ago
  Beware Slint's license, it's either proprietary or GPLv3, or whatever this kooky thing is that doesn't allow mobile apps: https://github.com/slint-ui/slint/blob/master/LICENSES/LicenseRef-Slint-Royalty-free-1.1.md https://github.com/slint-ui/slint/blob/master/LICENSES/Licen...

lfmunoz4 3y ago

Could barely understand this article. Is it just me or can none explain the graphics stack in a sensible way?

rijoja 3y ago
Neither, it's quite a huge subject, so if you put in the time you'll surely be able to figure it out! I don't think that the article isn't sensible, from what I saw it seemed quite well-written and the site usually has good quality content. The thing is that it targets readers with a certain knowledge-level. It's difficult for me to tell how much you know about graphics programming in general, so I can't really point you to any resources that would be better in your particular case!