15 ms·

GTK: Introducing Graphics Offload

279 points by signa11 3y ago

diath 3y ago
I wonder if there are plans to make it work with X11 in the future, I've yet to see the benefit of trying to switch to Wayland on my desktop, it just doesn't work as-is the way my 8 year old setup works.
- knocte 3y ago
  I doubt they have the energy to backport bleeding edge tech.
- PlutoIsAPlanet 3y ago
  This is one of the benefits of the Wayland protocol over X, being able to do this kind of thing relatively straightforwardly. Once support for hardware planes becomes more common in Wayland compositors, this can be tied to ultimately allow no-copy rendering to the display for non-fullscreen applications, which for video playback (incl. likes of Youtube) equals to reduced CPU & GPU usage and less power draw, as well as reduced latency.
  - AshamedCaptain 3y ago
    > This is one of the benefits of the Wayland protocol over X What. The original design of X actually encouraged a separate surface / Window for each single widget on your UI. This was actually removed in Gtk+3 ("windowless widgets"). And now they are bringing it back just for wayland ("subsurfaces"). As far as I can read, it is practically the same concept.
    - tadfisher 3y ago
      The original design of X had clients send drawing commands to the X server, basically treating the server as a remote Cairo/Skia-like 2D rasterizer, and subwindows were a cheap way to avoid pixel-level damage calculations. This was obviated in the common case by the Xdamage extension. Later use of windows as a rendering surface for shared client/server buffers was added with Xshm, then for device video buffers with Xv. GTK3 got rid of windowed widgets because Keith Packard introduced the Xrender extension, which basically added 2D compositing to X, which was the last remaining use for subwindows for every widget.
      - AshamedCaptain 3y ago
        This is completely wrong. Xrender is completely orthogonal to having windows or not. Heck, Xrender takes a _window_ as target -- Xrender is just an extension to allow more complicated drawing commands to be sent to the server (like alpha composition). You make your toolkit's programmer's life more complicated, not less, by having windowless widgets (at the very minimum you now have to complicate your rendering & event handling code with offsets and clip regions and the like). The excuse that was used when introducing windowless widgets is to reduce tearing/noise during resizing, as Gtk+ had trouble synchronizing the resizing of all the windows at the same time.
        play_ac 3y ago
        >Xrender is completely orthogonal to having windows or not. Heck, Xrender takes a _window_ as target -- Xrender is just an extension to allow more complicated drawing commands to be sent to the server (like alpha composition). Yes, that's the point. When you can tell Xrender to efficiently composite some pixmaps then there's really no reason to use sub-windows ever. >You make your toolkit's programmer's life more complicated, not less, by having windowless widgets (at the very minimum you now have to complicate your code with offsets and clip regions and the like). No, you still had to have offsets and clip regions before too because the client still had to set and update those. And it was more complicated because when you made a sub-window every single bit of state like that had to be synchronized with the X server and repeatedly copied over the wire. With client-side rendering everything is simply stored in the client and never has to deal with that problem.
        AshamedCaptain 3y ago
        > When you can tell Xrender to efficiently composite some pixmaps then there's really no reason to use sub-windows ever. There is, or we would not be having subsurfaces on Wayland or this entire discussion in the first place. Are you seriously arguing that the only reason to using windows in Xorg is to have composition? People were using Xshape/Xmisc and the like to handle the lack of alpha channels in the core protocol? This is not what I remember. I would be surprised if Xshape even worked on non-top level windows. heck, even MOTIF had windowless widgets (called gadgets iirc), and the purpose most definitely was not composition-related.
- hurryer 3y ago
  No screen tearing is a major benefit of using a compositor.
  - mrob 3y ago
    And screen tearing is a major benefit of not using a compositor. There's an unavoidable tradeoff between image quality and latency. Neither is objectively better than the other. Xorg has the unique advantage that you can easily switch between them by changing the TearFree setting with xrandr.
    - RVuRnvbM2e 3y ago
      It's not unique. Wayland has a tearing protocol. https://gitlab.freedesktop.org/wayland/wayland-protocols/-/tree/main/staging/tearing-control?ref_type=heads https://gitlab.freedesktop.org/wayland/wayland-protocols/-/t...
      - mrob 3y ago
        This is something that every application has to opt in to individually. It's not a global setting like TearFree.
        RVuRnvbM2e 3y ago
        This is just untrue.
        mrob 3y ago
        From the XML file that describes the protocol: "This global is a factory interface, allowing clients to inform which type of presentation the content of their surfaces is suitable for." Note that "global" refers to the interface, not the setting. Which Wayland compositor has the equivalent feature of "xrandr --output [name] --set TearFree off"?
        kaba0 3y ago
        Which is the correct default? No application should unknowingly render half-ready frames, that’s stupid. The few niches where it makes sense (games, 3D applications) can opt into it, and do their own thing.
- maccard 3y ago
  > I've yet to see the benefit of trying to switch to Wayland on my desktop how about Graphics Offload?
  - diath 3y ago
    This feature would be nice-to-have but is not impactful enough (at least to me) to outweigh the cons of having to switch to Wayland, which would include migrating my DE and getting accustomed to it as well as looking for replacement applications for these that do not work properly with Wayland (most notably ones that deal with global keyboard hooks). Admittedly I have never tried XWayland which I think could potentially solve some of these issues.
    - maccard 3y ago
      I think if your waiting for a magic bullet of a feature to upgrade you might be waiting a long time, and even Wayland will be replaced at that point. Instead, look at the combination of features (like this) and think about it and future upgrades. I think you're right that xwayland is probably a compromise for now if you need it for things like global shortcuts.
- mnd999 3y ago
  If it worked exactly the same there would indeed be no benefit. If you’re happy with that you have then there’s no reason to switch.
- aktuel 3y ago
  I am sorry to tell you that X11 is completely unmaintained by now. So the chances of that happening are zero.
  - NGRhodes 3y ago
    FYI 21.1.9 was released less than a month ago (https://lists.x.org/archives/xorg/2023-October/061515.html https://lists.x.org/archives/xorg/2023-October/061515.html), they are still fixing bugs.
    - dralley 3y ago
      You mean, they're still fixing critical CVEs.
      - badsectoracula 3y ago
        Which are still bugs. Also only two of the four changes mentioned in the mail are about CVEs.
- AshamedCaptain 3y ago
  Frankly, it was X11 which introduced "Graphics Offload" in the first place, with stuff like XV, chroma keying, and hardware overlays. Then compositors came and we moved to texture_from_surface extensions and uploading things into GPUs. This is just the eternal wheel of reinventing things in computing (TM) doing yet another iteration and unlikely to give any tangible benefits over the situation from decades ago.
  - play_ac 3y ago
    No, nothing like this exists in X11. Xorg still doesn't really have support for non-RGB surfaces. DRI3 gets you part of the way there for attaching GPU buffers but the way surfaces work would have to be overhauled to work more like Wayland, where they can be any format supported by the GPU. There isn't any incentive to implement this in X11 either because X11 is supposed to work over the network and none of this stuff would. Yes, you're technically right that this would have been possible years ago but it wasn't actually ever done, because X11 never had the ability to do it at the same time as using compositing.
    - AshamedCaptain 3y ago
      > Xorg still doesn't really have support for non-RGB surfaces You really need to add context to these statements, because _right now_ I am using through Xorg a program which uses a frigging colormap, which is as non-RGB as it gets. The entire reason Xlib has this "WhitePixel" and XGetPixel and XYPixmap and other useless functions which normally fetch a lot of ire is because it tries to go out of its way to support practically other-wordly color visuals and image formats. If anything, I'd say it is precisely RGB which has the most problems with X11, specially when you go more than 24bpp. > there for attaching GPU buffers None of this is about the GPU, but about about directly presenting images for _hardware_ composition using direct scan-out, hardware layers or not. Exactly what Xv is about, and the reason Xv supports formats like YUV. > There isn't any incentive to implement this in X11 either because X11 is supposed to work over the network and none of this stuff would As if that prevented any of the extensions done to X11 in the last three decades, including Xv.
- audidude 3y ago
  This would require protocol changes for X11 at best, and nobody is adding new protocols. Especially when nobody does Drawable of Drawables anymore and all use client-side drawing with Xshm. You need to dynamically change stacking of subsurfaces on a per-frame basis when doing the CRTC.
  - AshamedCaptain 3y ago
    I really don't see why it would need a new protocol. You can change stacking of "subsurfaces" in the traditional X11 fashion and you can most definitely do "drawables of drawables". At the very least I'd bet most clients still create a separate window for video content. I agree though it would require a lot of changes to the server and no one is in the mood (like, dynamically decide whether I composite this window or push it to a Xv port or hardware plane? practically inconceivable in the current graphics stack, albeit it is not a technical X limitation per-se). This entire feature is also going to be pretty pointless in Wayland desktop space either way because no one is in the mood either -- your dmabufs are going to end up in the GPU anyway for the foreseeable future, just because of the complexity of liftoff, variability of GPUs, and the like.
    - audidude 3y ago
      > I really don't see why it would need a new protocol. You'll need API to remap the Drawable to the scanout plane from a compositor on a per-frame basis (so when submitting the CRTC) and the compositor isn't in control of the CRTC. So...
      - AshamedCaptain 3y ago
        This assumes it would be the role of the window manager or compositor rather than the server who decides that, which I didn't think that way. But I guess it'd make sense (policy vs mechanism). "per-frame basis" I don't see why and it just mirrors Wayland concepts. Still, as protocols go, it's quite a minor change, and one applications don't necessarily have to support.
- kelnos 3y ago
  I would very much doubt it. This would likely require work on Xorg itself (a new protocol extension, maybe; I don't believe X11 supports anything but RGB, [+A, with XRender] for windows, and you'd probably need YUV support for this to be useful), which no one seems to care to do. And the GTK developers seem to see their X11 windowing backend as legacy code that they want to remove as soon as they can do so without getting too many complaints.
ng55QPSK 3y ago
Is the same infrastructure available in Windows and MacOS?
- deleted 3y ago
  [deleted]
- diath 3y ago
  The last paragraph says: > At the moment, graphics offload will only work with Wayland on Linux. There is some hope that we may be able to implement similar things on MacOS, but for now, this is Wayland-only. It also depends on the content being in dmabufs.
- knocte 3y ago
  From the article: > What are the limitations? > At the moment, graphics offload will only work with Wayland on Linux. There is some hope that we may be able to implement similar things on MacOS, but for now, this is Wayland-only. It also depends on the content being in dmabufs.
  - PlutoIsAPlanet 3y ago
    macOS supports similar things in its own native stack, but GTK doesn't make use of it.
- pjmlp 3y ago
  Nope, it is yet another step making Gtk only relevant for Linux development.
  - andersa 3y ago
    Has it been relevant for something else before?
    - pjmlp 3y ago
      Yes, back when G stood for Gimp, and not GNOME.
MarcusE1W 3y ago
Is this something where it would be helpful if the Linux (environment) developers worked together? Like the (graphics) kernel, GTK, KDE, Wayland, … guys all in one room (or video conference) to discuss requirements iron out one graphics architecture that is efficient and transparent? I think it’s good that different graphics systems exist but it feels unnecessary that every team has to make their own discoveries how to handle the existing pieces. If work were coordinated at least on the requirements and architecture level then I think a lot of synergies could be achieved. After that everyone can implement the architecture the way that works best for their use case, but on some common elements could be relied on.
- rawoul 3y ago
  They are: https://indico.freedesktop.org/event/4/ https://indico.freedesktop.org/event/4/ https://emersion.fr/blog/2023/hdr-hackfest-wrap-up/ https://emersion.fr/blog/2023/hdr-hackfest-wrap-up/ https://gitlab.freedesktop.org/wayland/wayland-protocols/-/wikis/meetings https://gitlab.freedesktop.org/wayland/wayland-protocols/-/w... ...
- jdub 3y ago
  They do, it's just not hugely visible. Two great conferences where some of that work happened were linux.conf.au and the Linux Plumbers Conference.
- dontlaugh 3y ago
  That’s exactly how Wayland came to be.
- BearOso 3y ago
  That's exactly what happened. This is the original intent for subsurfaces. A bunch of Wayland developers got together and wrote the spec a long time ago. The only thing happening now is Gtk making use of them transparently in the toolkit. Subsurfaces didn't have bug-free implementations for a while, so maybe some people avoided them. But I know some of us emulator programmers have been using them for output (especially because they can update asynchronously from the parent surface), and I think a couple media players do, too. It's not something that most applications really need.
jiehong 3y ago
I’m not sure I understand why an overlay allows partial offloading while rounding the corner of the video does not. Couldn’t the rounded corners of a video also be an overlay? I’m sure I’m missing something here, but the article does not explain that point.
- andyferris 3y ago
  I think it’s that the _window_ has rounded corners, and you don’t want the content appearing outside the window.
  - audidude 3y ago
    No, you can already be sure it's the right size. This has to do with what it takes to occlude the rounded area from the final display.
- orra 3y ago
  I'd love to know the answer to that. This is fantastic work, but it'd be a shame for it to be scunnered by rounded corners.
  - phkahler 3y ago
    Because the UX folks want what they want. I want my UI out of the way, including the corners of video and my CPU load.
    - audidude 3y ago
      Easily solved by black bars just like people are used to on a TV. I assume most video players will do this when in windowed mode.
      - orra 3y ago
        That's a good observation. Plus, you'll get black bars anyway, if you resize the window to not be the same aspect ratio as the video.
unwind 3y ago
Very cool! I think I found a minor typo: GTK 4.14 will introduce a GtkGraphicsOffload widget, whose only job it is to give a hint that GTK should try to offload the content of its child widget by attaching it to a subsurface instead of letting GSK process it like it usually does. I think that "GSK" nesr the end should just be "GTK". It's not a very near miss on standard qwerty, though ...
- pja 3y ago
  GSK is the GTK Scene Graph Kit: https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit
  - unwind 3y ago
    Wow thanks, TIL! Goes to show how far removed I am from GTK these days I guess. :/
- caslon 3y ago
  Not a typo: https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit
ahartmetz 3y ago
It is strange that the article doesn't compare and contrast to full-screen direct scanout, which most X11 and presumably Wayland compositors implement, e.g. KDE's kwin-wayland since 2021: https://invent.kde.org/plasma/kwin/-/merge_requests/502 https://invent.kde.org/plasma/kwin/-/merge_requests/502 Maybe that is because full-screen direct scanout doesn't take much (if anything) in a toolkit, it's almost purely a compositor feature.
- kaba0 3y ago
  Is there a significant difference? Hardware planes are basically that, just optionally not full-screen.
  - ahartmetz 3y ago
    Fullscreen direct scanout doesn't require (probably tricky) coordination to blend the graphical outputs of multiple processes. How that coordination works is the interesting technical question.
smallstepforman 3y ago
BeOS and Haiku allowed exposure to kernel graphics buffers decades ago (https://www.haiku-os.org/legacy-docs/bebook/BDirectWindow.html https://www.haiku-os.org/legacy-docs/bebook/BDirectWindow.ht...), which bypass the roundtrip to compositor and back. A companion article describing the design is here (https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12.html https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12....) After 25 years, GTK joins the party … With no native video drivers, moving a Vesa Haiku window with video playback still seems smoother than doing the same in Gnome/Kde under X11.
- pjmlp 3y ago
  Like most OSes where the desktop APIs are part of the whole developer experience, and not yet another pluggable piece.
- tmountain 3y ago
  BeOS was truly ahead of its time. It’s a shame that it didn’t get more traction.
  - ris 3y ago
    There must be a name for the logical fallacy I see whenever someone pines over a past "clearly superior" technology that wasn't adopted or had its project cancelled, but I can never grasp it. I guess the closest thing to it is an unfalsifiable statement. The problem comes from people remembering all of the positive traits (often just promises) of a technology but either forgetting all the problems it had or the technology never being given enough of a chance for people to discover all the areas in which it is a bit crap. BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even! This is also a big phenomenon in e.g. aviation. The greatest fighter jet ever is always the one that (tragically!) got cancelled before anyone could discover its weaknesses.
    - darkwater 3y ago
      This, plus nostalgia pink glasses plus rooting for the underdog (who lost) plus my niche tech is better than your mainstream one.
thriftwy 3y ago
I can't imagine anybody passing video frames one by one by a system call as a n array of pixels. I believe neither Xv nor GL-based renderers do that, even before we discuss hw accel.
- chlorion 3y ago
  Emulators for older systems very often do this! The older consoles like the NES had a pixel processing unit that generated the picture, you need to emulate it's state and interaction with the rest of the system, possibly cycle-by-cycle, which makes it not possible to do on the GPU as a shader or whatever. This is kind of a niche use case for sure but it's interesting.
  - thriftwy 3y ago
    They usually use libSDL instead of GTK, though.
    - anthk 3y ago
      They used SDL almost as an display buffer output. Oh, and for easy multimedia.
donatj 3y ago
Is this the same sort of thing Windows 95-XP had before Vista added DWM? Back when videos wouldn't show up in screen shots and their position on screen could sometimes gets out of sync with the window they were playing in? I never fully understood what was happening but my theory at the time was that the video was being sent to the video card separate from the rendered windows.
- anthk 3y ago
  With Linux/BSD you have a fun issue with X. If you used the XV overlayed video output, you got no screenshot except for a blue one. Both with video files and screenshots on TVTime. Some setting allowed it, I can't remember.
superkuh 3y ago
By going wayland only Gtk is becoming less of a GUI toolkit and more of just an extremely specific and arbitrary lib for GNOME desktop environment.
- mixedCase 3y ago
  Gtk is not going wayland only.
  - superkuh 3y ago
    https://www.phoronix.com/news/GTK5-Might-Drop-X11 https://www.phoronix.com/news/GTK5-Might-Drop-X11 "Red Hat's Matthias Clasen opened an issue on GTK entitled "Consider dropping the X11 backend"" https://gitlab.gnome.org/GNOME/gtk/-/issues/5004 https://gitlab.gnome.org/GNOME/gtk/-/issues/5004
- walteweiss 3y ago
  Everyone goes Wayland, not just Gnome, X is obsolete.
  - superkuh 3y ago
    And the handful of feature incompatible waylands are feature incomplete. You still can't keyboard/mouse share under any of them. That's just one of innumberable things you can't do.
    - walteweiss 3y ago
      I assume that will happen over time, won’t it?
      - gnulinux 3y ago
        No it won't Wayland protocol doesn't support it, besides Wayland has been here for such a long time. Pretending like Wayland is ready to replace X is like pretending like we can ban cars tomorrow and bikes will be good enough. It's completely delusional.
amelius 3y ago
This sounds so overly complicated considering that you can do all this in HTML without much effort.
- freedomben 3y ago
  I wonder why then, don't they just implement GTK with HTML? Probably because HTML is at the very top of the stack, while this is much lower... Without everything below it on the stack, HTML is just a text file.
  - amelius 3y ago
    You're reading into it too much. All I meant was: if the API of HTML is simple, then why does GTK's API have to be that complicated.
    - kaba0 3y ago
      This is literally an implementation detail, the API part is a single flag you can set if you want your subsurface to potentially make use of this future.
bogwog 3y ago
Semi-related question: Are there any real benefits to having the compositor deal with compositing your buffers as opposed to doing it yourself? Especially if you're already using hardware acceleration, passing your buffers to the system compositor seems like it could potentially introduce some latency. I guess it would allow the user/system to customize/tweak the compositing process somehow, but for what purpose?
- kaba0 3y ago
  Any kind of post-process effect like transparency, zoom; many stuff like window previews, overviews screens (these are sometimes possible without as well), and tear-freedom.
- phkahler 3y ago
  >> Are there any real benefits to having the compositor deal with compositing your buffers as opposed to doing it yourself? I would guess lower latency since the compositor is running the show trying to maintain full frame rate. The processes still have to cooperate somehow.
- ack_complete 3y ago
  The compositor can potentially map your layers to hardware scanout planes instead of compositing with the GPU, or skip the composition if all layers are obscured.
neurostimulant 3y ago
Rounded corners seems like a feature that has unexpectedly high performance penalty but the ui designers refused to let it go.
- bee_rider 3y ago
  Is it possible that they are just the well know representative example? I vaguely suspect they that is the case, but I can’t think of the broader class they are an example of, haha. The play button they show seems to be a good one, though. It is really nice to have it overlaid on the video.
  - DonHopkins 3y ago
    Fortunately the Play button disappears when the video starts playing, so it has no effect on the frame rate! Or instead of a triangular Play button, you could draw a big funny nose in some position and orientation, and the game would be to pause the video on a frame with somebody's face in it, with the nose in just the right spot. I don't know why the vlc project is ignoring my prs.
- solarkraft 3y ago
  It's something I as a user would also refuse to let go, given that the performance penalty is reasonably small (I think it is).
  - torginus 3y ago
    I think the point is that it's not - rather than just copying a rectangular area to the screen, you have to go through the intermediate step of rendering everything to a temporary buffer, and compositing the results via a shader.
    - chris_wot 3y ago
      But… the example given shows that they place the video frame behind the window an make the front window transparent except for the round play button. This apparently offloads the frame… so why not just do the same for rounded corners? What am I missing?
sylware 3y ago
steam deck wayland compositor is built on drm(dmabufs)/vulkan. (Sad it is written in c++) But what did surprise me even more: I was expecting GTK, and moreover the 4, to be on par with valve software. On linux, I would not event think to code a modern system hardware accelerated GFX component which is not drm(dmabuf)/vulkan.
- charcircuit 3y ago
  This blog post is not talking about Mutter, GNOME's compositor. GTK's hardware acceleration had already been using dmabufs before adding this graphics offload feature.
  - sylware 3y ago
    But as the article states, with GL, not vulkan. Unless the article is obsolete itself?
    - charcircuit 3y ago
      Sure, but OpenGL itself is still useful and used in modern software.
      - sylware 3y ago
        It is legacy and started to be retired. Not to mention GL is a massive and gigantic kludge/bloat compared to vulkan (mostly due to the glsl compiler). So this is good to let it go.
        tristan957 3y ago
        If you have spare time, the GTK maintainers want people to work on the Vulkan renderer. Benjamin Otte and Georges Stavracas Neto have put in a bit of effort to make the Vulkan renderer better. GL is only deprecated on Mac from what I understand.
ori_b 3y ago
The thing that's always felt slow to me in GTK was resizing windows, not getting pixels to the screen. I'm wondering if adding all these composited surfaces adds a cost when resizing the windows and their associated out of process surfaces.
- rollcat 3y ago
  More likely it removes costs. This is very specifically an optimization.
matheusmoreira 3y ago
> A dmabuf is a memory buffer in kernel space that is identified by a file descriptor. > The idea is that you don’t have to copy lots of pixel data around, and instead just pass a file descriptor between kernel subsystems. So like sendfile for graphics? https://www.man7.org/linux/man-pages/man2/sendfile.2.html https://www.man7.org/linux/man-pages/man2/sendfile.2.html It's a pretty awesome system call. We should have more of those.

279 points by signa11 3y ago

I wonder if there are plans to make it work with X11 in the future, I've yet to see the benefit of trying to switch to Wayland on my desktop, it just doesn't work as-is the way my 8 year old setup works.

knocte 3y ago
I doubt they have the energy to backport bleeding edge tech.
PlutoIsAPlanet 3y ago
This is one of the benefits of the Wayland protocol over X, being able to do this kind of thing relatively straightforwardly. Once support for hardware planes becomes more common in Wayland compositors, this can be tied to ultimately allow no-copy rendering to the display for non-fullscreen applications, which for video playback (incl. likes of Youtube) equals to reduced CPU & GPU usage and less power draw, as well as reduced latency.
- AshamedCaptain 3y ago
  > This is one of the benefits of the Wayland protocol over X What. The original design of X actually encouraged a separate surface / Window for each single widget on your UI. This was actually removed in Gtk+3 ("windowless widgets"). And now they are bringing it back just for wayland ("subsurfaces"). As far as I can read, it is practically the same concept.
  - tadfisher 3y ago
    The original design of X had clients send drawing commands to the X server, basically treating the server as a remote Cairo/Skia-like 2D rasterizer, and subwindows were a cheap way to avoid pixel-level damage calculations. This was obviated in the common case by the Xdamage extension. Later use of windows as a rendering surface for shared client/server buffers was added with Xshm, then for device video buffers with Xv. GTK3 got rid of windowed widgets because Keith Packard introduced the Xrender extension, which basically added 2D compositing to X, which was the last remaining use for subwindows for every widget.
    - AshamedCaptain 3y ago
      This is completely wrong. Xrender is completely orthogonal to having windows or not. Heck, Xrender takes a _window_ as target -- Xrender is just an extension to allow more complicated drawing commands to be sent to the server (like alpha composition). You make your toolkit's programmer's life more complicated, not less, by having windowless widgets (at the very minimum you now have to complicate your rendering & event handling code with offsets and clip regions and the like). The excuse that was used when introducing windowless widgets is to reduce tearing/noise during resizing, as Gtk+ had trouble synchronizing the resizing of all the windows at the same time.
      - play_ac 3y ago
        >Xrender is completely orthogonal to having windows or not. Heck, Xrender takes a _window_ as target -- Xrender is just an extension to allow more complicated drawing commands to be sent to the server (like alpha composition). Yes, that's the point. When you can tell Xrender to efficiently composite some pixmaps then there's really no reason to use sub-windows ever. >You make your toolkit's programmer's life more complicated, not less, by having windowless widgets (at the very minimum you now have to complicate your code with offsets and clip regions and the like). No, you still had to have offsets and clip regions before too because the client still had to set and update those. And it was more complicated because when you made a sub-window every single bit of state like that had to be synchronized with the X server and repeatedly copied over the wire. With client-side rendering everything is simply stored in the client and never has to deal with that problem.
        AshamedCaptain 3y ago
        > When you can tell Xrender to efficiently composite some pixmaps then there's really no reason to use sub-windows ever. There is, or we would not be having subsurfaces on Wayland or this entire discussion in the first place. Are you seriously arguing that the only reason to using windows in Xorg is to have composition? People were using Xshape/Xmisc and the like to handle the lack of alpha channels in the core protocol? This is not what I remember. I would be surprised if Xshape even worked on non-top level windows. heck, even MOTIF had windowless widgets (called gadgets iirc), and the purpose most definitely was not composition-related.
hurryer 3y ago
No screen tearing is a major benefit of using a compositor.
- mrob 3y ago
  And screen tearing is a major benefit of not using a compositor. There's an unavoidable tradeoff between image quality and latency. Neither is objectively better than the other. Xorg has the unique advantage that you can easily switch between them by changing the TearFree setting with xrandr.
  - RVuRnvbM2e 3y ago
    It's not unique. Wayland has a tearing protocol. https://gitlab.freedesktop.org/wayland/wayland-protocols/-/tree/main/staging/tearing-control?ref_type=heads https://gitlab.freedesktop.org/wayland/wayland-protocols/-/t...
    - mrob 3y ago
      This is something that every application has to opt in to individually. It's not a global setting like TearFree.
      - RVuRnvbM2e 3y ago
        This is just untrue.
        mrob 3y ago
        From the XML file that describes the protocol: "This global is a factory interface, allowing clients to inform which type of presentation the content of their surfaces is suitable for." Note that "global" refers to the interface, not the setting. Which Wayland compositor has the equivalent feature of "xrandr --output [name] --set TearFree off"?
      - kaba0 3y ago
        Which is the correct default? No application should unknowingly render half-ready frames, that’s stupid. The few niches where it makes sense (games, 3D applications) can opt into it, and do their own thing.
maccard 3y ago
> I've yet to see the benefit of trying to switch to Wayland on my desktop how about Graphics Offload?
- diath 3y ago
  This feature would be nice-to-have but is not impactful enough (at least to me) to outweigh the cons of having to switch to Wayland, which would include migrating my DE and getting accustomed to it as well as looking for replacement applications for these that do not work properly with Wayland (most notably ones that deal with global keyboard hooks). Admittedly I have never tried XWayland which I think could potentially solve some of these issues.
  - maccard 3y ago
    I think if your waiting for a magic bullet of a feature to upgrade you might be waiting a long time, and even Wayland will be replaced at that point. Instead, look at the combination of features (like this) and think about it and future upgrades. I think you're right that xwayland is probably a compromise for now if you need it for things like global shortcuts.
mnd999 3y ago
If it worked exactly the same there would indeed be no benefit. If you’re happy with that you have then there’s no reason to switch.
aktuel 3y ago
I am sorry to tell you that X11 is completely unmaintained by now. So the chances of that happening are zero.
- NGRhodes 3y ago
  FYI 21.1.9 was released less than a month ago (https://lists.x.org/archives/xorg/2023-October/061515.html https://lists.x.org/archives/xorg/2023-October/061515.html), they are still fixing bugs.
  - dralley 3y ago
    You mean, they're still fixing critical CVEs.
    - badsectoracula 3y ago
      Which are still bugs. Also only two of the four changes mentioned in the mail are about CVEs.
AshamedCaptain 3y ago
Frankly, it was X11 which introduced "Graphics Offload" in the first place, with stuff like XV, chroma keying, and hardware overlays. Then compositors came and we moved to texture_from_surface extensions and uploading things into GPUs. This is just the eternal wheel of reinventing things in computing (TM) doing yet another iteration and unlikely to give any tangible benefits over the situation from decades ago.
- play_ac 3y ago
  No, nothing like this exists in X11. Xorg still doesn't really have support for non-RGB surfaces. DRI3 gets you part of the way there for attaching GPU buffers but the way surfaces work would have to be overhauled to work more like Wayland, where they can be any format supported by the GPU. There isn't any incentive to implement this in X11 either because X11 is supposed to work over the network and none of this stuff would. Yes, you're technically right that this would have been possible years ago but it wasn't actually ever done, because X11 never had the ability to do it at the same time as using compositing.
  - AshamedCaptain 3y ago
    > Xorg still doesn't really have support for non-RGB surfaces You really need to add context to these statements, because _right now_ I am using through Xorg a program which uses a frigging colormap, which is as non-RGB as it gets. The entire reason Xlib has this "WhitePixel" and XGetPixel and XYPixmap and other useless functions which normally fetch a lot of ire is because it tries to go out of its way to support practically other-wordly color visuals and image formats. If anything, I'd say it is precisely RGB which has the most problems with X11, specially when you go more than 24bpp. > there for attaching GPU buffers None of this is about the GPU, but about about directly presenting images for _hardware_ composition using direct scan-out, hardware layers or not. Exactly what Xv is about, and the reason Xv supports formats like YUV. > There isn't any incentive to implement this in X11 either because X11 is supposed to work over the network and none of this stuff would As if that prevented any of the extensions done to X11 in the last three decades, including Xv.
audidude 3y ago
This would require protocol changes for X11 at best, and nobody is adding new protocols. Especially when nobody does Drawable of Drawables anymore and all use client-side drawing with Xshm. You need to dynamically change stacking of subsurfaces on a per-frame basis when doing the CRTC.
- AshamedCaptain 3y ago
  I really don't see why it would need a new protocol. You can change stacking of "subsurfaces" in the traditional X11 fashion and you can most definitely do "drawables of drawables". At the very least I'd bet most clients still create a separate window for video content. I agree though it would require a lot of changes to the server and no one is in the mood (like, dynamically decide whether I composite this window or push it to a Xv port or hardware plane? practically inconceivable in the current graphics stack, albeit it is not a technical X limitation per-se). This entire feature is also going to be pretty pointless in Wayland desktop space either way because no one is in the mood either -- your dmabufs are going to end up in the GPU anyway for the foreseeable future, just because of the complexity of liftoff, variability of GPUs, and the like.
  - audidude 3y ago
    > I really don't see why it would need a new protocol. You'll need API to remap the Drawable to the scanout plane from a compositor on a per-frame basis (so when submitting the CRTC) and the compositor isn't in control of the CRTC. So...
    - AshamedCaptain 3y ago
      This assumes it would be the role of the window manager or compositor rather than the server who decides that, which I didn't think that way. But I guess it'd make sense (policy vs mechanism). "per-frame basis" I don't see why and it just mirrors Wayland concepts. Still, as protocols go, it's quite a minor change, and one applications don't necessarily have to support.
kelnos 3y ago
I would very much doubt it. This would likely require work on Xorg itself (a new protocol extension, maybe; I don't believe X11 supports anything but RGB, [+A, with XRender] for windows, and you'd probably need YUV support for this to be useful), which no one seems to care to do. And the GTK developers seem to see their X11 windowing backend as legacy code that they want to remove as soon as they can do so without getting too many complaints.

ng55QPSK 3y ago

Is the same infrastructure available in Windows and MacOS?

deleted 3y ago
[deleted]
diath 3y ago
The last paragraph says: > At the moment, graphics offload will only work with Wayland on Linux. There is some hope that we may be able to implement similar things on MacOS, but for now, this is Wayland-only. It also depends on the content being in dmabufs.
knocte 3y ago
From the article: > What are the limitations? > At the moment, graphics offload will only work with Wayland on Linux. There is some hope that we may be able to implement similar things on MacOS, but for now, this is Wayland-only. It also depends on the content being in dmabufs.
- PlutoIsAPlanet 3y ago
  macOS supports similar things in its own native stack, but GTK doesn't make use of it.
pjmlp 3y ago
Nope, it is yet another step making Gtk only relevant for Linux development.
- andersa 3y ago
  Has it been relevant for something else before?
  - pjmlp 3y ago
    Yes, back when G stood for Gimp, and not GNOME.

MarcusE1W 3y ago

Is this something where it would be helpful if the Linux (environment) developers worked together? Like the (graphics) kernel, GTK, KDE, Wayland, … guys all in one room (or video conference) to discuss requirements iron out one graphics architecture that is efficient and transparent? I think it’s good that different graphics systems exist but it feels unnecessary that every team has to make their own discoveries how to handle the existing pieces. If work were coordinated at least on the requirements and architecture level then I think a lot of synergies could be achieved. After that everyone can implement the architecture the way that works best for their use case, but on some common elements could be relied on.

rawoul 3y ago
They are: https://indico.freedesktop.org/event/4/ https://indico.freedesktop.org/event/4/ https://emersion.fr/blog/2023/hdr-hackfest-wrap-up/ https://emersion.fr/blog/2023/hdr-hackfest-wrap-up/ https://gitlab.freedesktop.org/wayland/wayland-protocols/-/wikis/meetings https://gitlab.freedesktop.org/wayland/wayland-protocols/-/w... ...
jdub 3y ago
They do, it's just not hugely visible. Two great conferences where some of that work happened were linux.conf.au and the Linux Plumbers Conference.
dontlaugh 3y ago
That’s exactly how Wayland came to be.
BearOso 3y ago
That's exactly what happened. This is the original intent for subsurfaces. A bunch of Wayland developers got together and wrote the spec a long time ago. The only thing happening now is Gtk making use of them transparently in the toolkit. Subsurfaces didn't have bug-free implementations for a while, so maybe some people avoided them. But I know some of us emulator programmers have been using them for output (especially because they can update asynchronously from the parent surface), and I think a couple media players do, too. It's not something that most applications really need.

jiehong 3y ago

I’m not sure I understand why an overlay allows partial offloading while rounding the corner of the video does not. Couldn’t the rounded corners of a video also be an overlay? I’m sure I’m missing something here, but the article does not explain that point.

andyferris 3y ago
I think it’s that the _window_ has rounded corners, and you don’t want the content appearing outside the window.
- audidude 3y ago
  No, you can already be sure it's the right size. This has to do with what it takes to occlude the rounded area from the final display.
orra 3y ago
I'd love to know the answer to that. This is fantastic work, but it'd be a shame for it to be scunnered by rounded corners.
- phkahler 3y ago
  Because the UX folks want what they want. I want my UI out of the way, including the corners of video and my CPU load.
  - audidude 3y ago
    Easily solved by black bars just like people are used to on a TV. I assume most video players will do this when in windowed mode.
    - orra 3y ago
      That's a good observation. Plus, you'll get black bars anyway, if you resize the window to not be the same aspect ratio as the video.

unwind 3y ago

Very cool! I think I found a minor typo: GTK 4.14 will introduce a GtkGraphicsOffload widget, whose only job it is to give a hint that GTK should try to offload the content of its child widget by attaching it to a subsurface instead of letting GSK process it like it usually does. I think that "GSK" nesr the end should just be "GTK". It's not a very near miss on standard qwerty, though ...

pja 3y ago
GSK is the GTK Scene Graph Kit: https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit
- unwind 3y ago
  Wow thanks, TIL! Goes to show how far removed I am from GTK these days I guess. :/
caslon 3y ago
Not a typo: https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit https://en.wikipedia.org/wiki/GTK_Scene_Graph_Kit

ahartmetz 3y ago

It is strange that the article doesn't compare and contrast to full-screen direct scanout, which most X11 and presumably Wayland compositors implement, e.g. KDE's kwin-wayland since 2021: https://invent.kde.org/plasma/kwin/-/merge_requests/502 https://invent.kde.org/plasma/kwin/-/merge_requests/502 Maybe that is because full-screen direct scanout doesn't take much (if anything) in a toolkit, it's almost purely a compositor feature.

kaba0 3y ago
Is there a significant difference? Hardware planes are basically that, just optionally not full-screen.
- ahartmetz 3y ago
  Fullscreen direct scanout doesn't require (probably tricky) coordination to blend the graphical outputs of multiple processes. How that coordination works is the interesting technical question.

smallstepforman 3y ago

BeOS and Haiku allowed exposure to kernel graphics buffers decades ago (https://www.haiku-os.org/legacy-docs/bebook/BDirectWindow.html https://www.haiku-os.org/legacy-docs/bebook/BDirectWindow.ht...), which bypass the roundtrip to compositor and back. A companion article describing the design is here (https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12.html https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12....) After 25 years, GTK joins the party … With no native video drivers, moving a Vesa Haiku window with video playback still seems smoother than doing the same in Gnome/Kde under X11.

pjmlp 3y ago
Like most OSes where the desktop APIs are part of the whole developer experience, and not yet another pluggable piece.
tmountain 3y ago
BeOS was truly ahead of its time. It’s a shame that it didn’t get more traction.
- ris 3y ago
  There must be a name for the logical fallacy I see whenever someone pines over a past "clearly superior" technology that wasn't adopted or had its project cancelled, but I can never grasp it. I guess the closest thing to it is an unfalsifiable statement. The problem comes from people remembering all of the positive traits (often just promises) of a technology but either forgetting all the problems it had or the technology never being given enough of a chance for people to discover all the areas in which it is a bit crap. BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even! This is also a big phenomenon in e.g. aviation. The greatest fighter jet ever is always the one that (tragically!) got cancelled before anyone could discover its weaknesses.
  - darkwater 3y ago
    This, plus nostalgia pink glasses plus rooting for the underdog (who lost) plus my niche tech is better than your mainstream one.

thriftwy 3y ago

I can't imagine anybody passing video frames one by one by a system call as a n array of pixels. I believe neither Xv nor GL-based renderers do that, even before we discuss hw accel.

chlorion 3y ago
Emulators for older systems very often do this! The older consoles like the NES had a pixel processing unit that generated the picture, you need to emulate it's state and interaction with the rest of the system, possibly cycle-by-cycle, which makes it not possible to do on the GPU as a shader or whatever. This is kind of a niche use case for sure but it's interesting.
- thriftwy 3y ago
  They usually use libSDL instead of GTK, though.
  - anthk 3y ago
    They used SDL almost as an display buffer output. Oh, and for easy multimedia.

donatj 3y ago

Is this the same sort of thing Windows 95-XP had before Vista added DWM? Back when videos wouldn't show up in screen shots and their position on screen could sometimes gets out of sync with the window they were playing in? I never fully understood what was happening but my theory at the time was that the video was being sent to the video card separate from the rendered windows.

anthk 3y ago
With Linux/BSD you have a fun issue with X. If you used the XV overlayed video output, you got no screenshot except for a blue one. Both with video files and screenshots on TVTime. Some setting allowed it, I can't remember.

superkuh 3y ago

By going wayland only Gtk is becoming less of a GUI toolkit and more of just an extremely specific and arbitrary lib for GNOME desktop environment.

mixedCase 3y ago
Gtk is not going wayland only.
- superkuh 3y ago
  https://www.phoronix.com/news/GTK5-Might-Drop-X11 https://www.phoronix.com/news/GTK5-Might-Drop-X11 "Red Hat's Matthias Clasen opened an issue on GTK entitled "Consider dropping the X11 backend"" https://gitlab.gnome.org/GNOME/gtk/-/issues/5004 https://gitlab.gnome.org/GNOME/gtk/-/issues/5004
walteweiss 3y ago
Everyone goes Wayland, not just Gnome, X is obsolete.
- superkuh 3y ago
  And the handful of feature incompatible waylands are feature incomplete. You still can't keyboard/mouse share under any of them. That's just one of innumberable things you can't do.
  - walteweiss 3y ago
    I assume that will happen over time, won’t it?
    - gnulinux 3y ago
      No it won't Wayland protocol doesn't support it, besides Wayland has been here for such a long time. Pretending like Wayland is ready to replace X is like pretending like we can ban cars tomorrow and bikes will be good enough. It's completely delusional.

amelius 3y ago

This sounds so overly complicated considering that you can do all this in HTML without much effort.

freedomben 3y ago
I wonder why then, don't they just implement GTK with HTML? Probably because HTML is at the very top of the stack, while this is much lower... Without everything below it on the stack, HTML is just a text file.
- amelius 3y ago
  You're reading into it too much. All I meant was: if the API of HTML is simple, then why does GTK's API have to be that complicated.
  - kaba0 3y ago
    This is literally an implementation detail, the API part is a single flag you can set if you want your subsurface to potentially make use of this future.

bogwog 3y ago

Semi-related question: Are there any real benefits to having the compositor deal with compositing your buffers as opposed to doing it yourself? Especially if you're already using hardware acceleration, passing your buffers to the system compositor seems like it could potentially introduce some latency. I guess it would allow the user/system to customize/tweak the compositing process somehow, but for what purpose?

kaba0 3y ago
Any kind of post-process effect like transparency, zoom; many stuff like window previews, overviews screens (these are sometimes possible without as well), and tear-freedom.
phkahler 3y ago
>> Are there any real benefits to having the compositor deal with compositing your buffers as opposed to doing it yourself? I would guess lower latency since the compositor is running the show trying to maintain full frame rate. The processes still have to cooperate somehow.
ack_complete 3y ago
The compositor can potentially map your layers to hardware scanout planes instead of compositing with the GPU, or skip the composition if all layers are obscured.

neurostimulant 3y ago

Rounded corners seems like a feature that has unexpectedly high performance penalty but the ui designers refused to let it go.

bee_rider 3y ago
Is it possible that they are just the well know representative example? I vaguely suspect they that is the case, but I can’t think of the broader class they are an example of, haha. The play button they show seems to be a good one, though. It is really nice to have it overlaid on the video.
- DonHopkins 3y ago
  Fortunately the Play button disappears when the video starts playing, so it has no effect on the frame rate! Or instead of a triangular Play button, you could draw a big funny nose in some position and orientation, and the game would be to pause the video on a frame with somebody's face in it, with the nose in just the right spot. I don't know why the vlc project is ignoring my prs.
solarkraft 3y ago
It's something I as a user would also refuse to let go, given that the performance penalty is reasonably small (I think it is).
- torginus 3y ago
  I think the point is that it's not - rather than just copying a rectangular area to the screen, you have to go through the intermediate step of rendering everything to a temporary buffer, and compositing the results via a shader.
  - chris_wot 3y ago
    But… the example given shows that they place the video frame behind the window an make the front window transparent except for the round play button. This apparently offloads the frame… so why not just do the same for rounded corners? What am I missing?

sylware 3y ago

steam deck wayland compositor is built on drm(dmabufs)/vulkan. (Sad it is written in c++) But what did surprise me even more: I was expecting GTK, and moreover the 4, to be on par with valve software. On linux, I would not event think to code a modern system hardware accelerated GFX component which is not drm(dmabuf)/vulkan.

charcircuit 3y ago
This blog post is not talking about Mutter, GNOME's compositor. GTK's hardware acceleration had already been using dmabufs before adding this graphics offload feature.
- sylware 3y ago
  But as the article states, with GL, not vulkan. Unless the article is obsolete itself?
  - charcircuit 3y ago
    Sure, but OpenGL itself is still useful and used in modern software.
    - sylware 3y ago
      It is legacy and started to be retired. Not to mention GL is a massive and gigantic kludge/bloat compared to vulkan (mostly due to the glsl compiler). So this is good to let it go.
      - tristan957 3y ago
        If you have spare time, the GTK maintainers want people to work on the Vulkan renderer. Benjamin Otte and Georges Stavracas Neto have put in a bit of effort to make the Vulkan renderer better. GL is only deprecated on Mac from what I understand.

ori_b 3y ago

The thing that's always felt slow to me in GTK was resizing windows, not getting pixels to the screen. I'm wondering if adding all these composited surfaces adds a cost when resizing the windows and their associated out of process surfaces.

rollcat 3y ago
More likely it removes costs. This is very specifically an optimization.

matheusmoreira 3y ago

> A dmabuf is a memory buffer in kernel space that is identified by a file descriptor. > The idea is that you don’t have to copy lots of pixel data around, and instead just pass a file descriptor between kernel subsystems. So like sendfile for graphics? https://www.man7.org/linux/man-pages/man2/sendfile.2.html https://www.man7.org/linux/man-pages/man2/sendfile.2.html It's a pretty awesome system call. We should have more of those.