7 ms·

I’ve never understood why the GPU isn’t leveraged for 2D/UI stuff. Isn’t it all fundamentally the same? Vectors and pixels?

chrisseaton 5y ago
The GPU is used to render UI stuff in any modern system.
- danielvaughn 5y ago
  What I'd really love to see is a UI system that gives developers access to the GPU somehow, but I'm not sure what that would look like.
  - ben-schaaf 5y ago
    Like Direct2D, QPainter (QT) and cairo (GTK)?
  - kllrnohj 5y ago
    Like https://developer.android.com/reference/android/graphics/RuntimeShader https://developer.android.com/reference/android/graphics/Run... https://developer.android.com/about/versions/13/features#shaders https://developer.android.com/about/versions/13/features#sha... ?
  - deleted 5y ago
    [deleted]
resonious 5y ago
I think it is. I see all kinds of 2D projects claiming to be "GPU accelerated" - for example GTK, KDE, web browsers. I'm not sure how much of the actual processing is done on the GPU, but it's enough to call it "accelerated"!
ben-schaaf 5y ago
Browsers largely make use of the GPU for UI rendering. Direct2D, Cocoa, QT and GTK(4) are all hardware accelerated as well. So not really sure what you mean?
Jasper_ 5y ago
The linked article covers the major challenges that makes it difficult to adapt the GPU to 2D vector graphics rendering. tl;dr 2D cares about shapes and curves and proper antialiased coverage of super small shapes, the traditional rasterization GPU pipeline is very good at triangles and textures and have limited coverage options. Direct2D, Qt and GTK+ still do a good portion of the graphics work on the CPU, and only use GPU for composition. Some limited graphics can be done on the GPU, usually with quality tradeoffs. Font rasterization is still done the CPU, and uploaded to the GPU as a texture. Newer libraries like Pathfinder, Spinel, piet-gpu all work by not using the triangle rasterization parts but instead treating the GPU as a general-purpose parallel processor with compute shaders.
pengaru 5y ago
Programming graphics, especially 2D stuff, is far more ergonomic and convenient in your native language already executing on the CPU. If you can get away with it performance wise, there's really no incentive to incur the myriad obnoxious bullshit inherent in GPU programming. But with the advent of high dpi displays it's become problematic to do even simple 2D/UI rendering on the CPU just because of the enormous quantity of pixels. When you pull in GPU support, now you're stuck having to pick a backend (gl/vulkan/d3d/metal) or some compatibility layer to make some/all of them work. You have to write shaders, you have to constantly move state in/out of the GPU across this GPU:CPU API boundary. It's just a total clusterfuck best avoided if possible.
- copperx 5y ago
  Naive question: Can't you leverage a game engine for this? Why do you have to do all the low level work from scratch?
  - pengaru 5y ago
    I'm not familiar with modern game engines, but I'd be very surprised if any of them managed to eliminate the utterly unnatural reality of writing shaders vs. writing classical 2D rendering algorithms operating on a linear buffer of pixels in memory. For concurrency reasons shaders logically run on a single pixel. Gone are your longstanding algorithms for doing simple things like bresenham line-drawing, or something as simple as drawing a filled box like this: for (int y = box.y; y < box.y + box.h; y++) for (int x = box.x; x < box.x + box.w; x++) FB[y * FB_STRIDE + x] = box.color; Nope, not happening in a shader. Every shader basically executes in isolation on a pixel and you have to operate from a sort of dead-reckoning perspective. No more sequential loops iterating rows and lines, a fashion which we have literally decades of graphics programming publications explaining how to do things. Not to mention how natural it is to think about things that way, since it closely resembles drawing on paper. In shaders you often end up doing things that feel utterly absurd in terms of overhead because of this "you run on an arbitrary pixel" perspective. Oftentimes you're writing some kind of distance function, where previously you would have written a loop iterating across lines and rows advancing some state as you step through the pixels. In a shader it's like the paper is covered with thousands of pencils that don't move, and the shader program just determines what color the pencil should be based on its location. GPU programming is plain annoying, even without the GPU API fragmentation clusterfuck. Especially if you've been writing 2D stuff on the CPU for decades.
kllrnohj 5y ago
It is. There's a terminology problem in play here. Throughout this article 2D does not mean 2D, it means "arbitrary complexity paths, eg bezier curves". This is a small subset of 2D as makes up a UI. It'd be like saying 3D exclusively means infinitely detailed tesellated shapes with path traced rendering. That's definitely an area of 3D that exists, but of course is not at all the entirety of 3D in practice in eg games or movies. Rather it's more like the holy grail. Same thing here with eg. SVGs. GPU accelerating SVGs is stupidly complicated because it's an inherently serial algorithm, and GPUs are poop at that. But how much of your 2D UI is made up of that? Text is in that same category but how much else? Typically very little. Maybe a few icons, but that tends to be about it. Instead you have higher level shapes, like round rectangles. And those you can do with a GPU quite easily. Similarly images are usually just a textured quad. Again trivial for a GPU. You could describe them as paths if you had a fully generic, fully accelerated path rendering system. But nobody has that, so nobody actually describes them like that. So very nearly all 2D/UI systems are GPU accelerated. It'd be perhaps more accurate to call them hybrid renderers. Things like fonts are just CPU rendered because CPUs are better at it, but the GPU is doing all the fills, gradients, "simple" shapes, texturing, etc...
- Jasper_ 5y ago
  I don't know if I'd go that far -- icons and text are vector paths. Strokes and drop shadows (aka blurs) are all things that GPUs aren't great particularly great at. Simple shapes like rounded rectangles, GPUs can be OK at, but you'd have overdraw problems if done naively. I've worked on 2D rendering engines, so I've seen the content thrown at it in the wild. Very rarely do you have a simple case. GitHub's buttons are maybe the simplest example I can think of, and they have strokes (GPUs: ugh) on a filled rounded border (GPUs: ugh), with text inside (GPUs: ugh), sometimes with a text shadow (GPUs: ugh). It can be done, but you basically have to get away from triangles and move into research methods which are exceptionally more tricky, aka the stuff in Pathfinder and piet-gpu.
  - teknopaul 5y ago
    Showing my ignorance here: Why we don't have dedicated graphics hardware for 2D? Feels like moving 2D jobs to a fast 3D engine is wrong approach at the top level.
kmeisthax 5y ago
GPUs want to draw triangles, and in fact only know how to draw triangles[0]. Pretty much all graphics API innovation has been around either feeding more triangles to the GPU faster, letting the GPU create more triangles after they've been sent, or finding cool new ways to draw things on the surface of those triangles. 2D/UI breaks down into drawing curves, either as filled shapes or strokes. The preferred representation of such is a Bezier spline, which is a series of degree-three[1] polynomials that GPUs have zero support for rasterizing. Furthermore, strokes on the basis functions of those splines are not polynomials, but an even more bizarre curve type called an algebraic curve. You cannot just offset the control points to derive a stroke curve; you either have to approximate the stroke itself with Beziers, or actually draw a line sequentially in a way that GPUs are really not capable of doing. The four things you can do to render 2D/UI on a GPU is: - Tesselate the Bezier spline with a series of triangles. Lyon does this. Bezier curves make this rather cheap to do, but this requires foreknowledge of what scale the Bezier will be rendered at, and you cannot adjust stroke sizes at all without retessellating. - Send the control points to the GPU and use hardware tessellation to do the above per-frame. No clue if anyone does this. - Don't tessellate at all, but send the control points to the GPU as a polygonal mesh, and draw the actual Beziers in the fragment shaders for each polygon. For degree-two/quadratics there are a series of coordinate transforms that you can do which conveniently map all curves to one UV coordinate space; degree-three/cubics require a lot more attention in order to render correctly. If I remember correctly Mozilla Pathfinder does this[2]. - Send a signed distance field and have the GPU march it to render curves. I don't know much about this but I remember hearing about this a while back. All of these approaches have downsides. Tessellation is the approach I'm most familiar with because it's used heavily in Ruffle; so I'll just explain it's downsides to give you a good idea of why this is a huge problem: - We can't support some of Flash's weirder rendering hacks, like hairline strokes. Once we have a tessellated stroke, it will always be that width regardless of how we scale the shape. But hairlines require that the stroke get proportionally bigger as the shape gets smaller. In Flash, they were rendering on CPU, so it was just a matter of saying "strokes are always at least 1px". - We have to sort of guess what scale we want to render at and hope we have enough detail that the curves look like curves. There's one particular Flash optimization trick that consistently breaks our detail estimation and causes us to generate really lo-fi polygons. - Tessellation requires the curve shape to actually make sense as a sealed hull. We've exposed numerous underlying bugs in lyon purely by throwing really complicated or badly-specified Flash art at it. - All of this is expensive, especially for complicated shapes. For example, pretty much any Homestuck SWF will lock up your browser for multiple minutes as lyon tries to make sense of all of Hussie's art. This also precludes varying strokes by retessellating per-frame, which would otherwise fix the hairline stroke problem I mentioned above. [0] AFAIK quads are emulated on most modern GPUs, but they are just as useless for 2D/UI as triangles are. [1] Some earlier 2D systems used degree-two Bezier splines, including most of Adobe Flash. [2] We have half a PR to use this in Ruffle, but it was abandoned a while back.
eru 5y ago
That's exactly what eg the game Shovel Knight is doing. See eg https://www.youtube.com/watch?v=vjENktnbCaE https://www.youtube.com/watch?v=vjENktnbCaE

chrisseaton 5y ago

The GPU is used to render UI stuff in any modern system.

danielvaughn 5y ago
What I'd really love to see is a UI system that gives developers access to the GPU somehow, but I'm not sure what that would look like.
- ben-schaaf 5y ago
  Like Direct2D, QPainter (QT) and cairo (GTK)?
- kllrnohj 5y ago
  Like https://developer.android.com/reference/android/graphics/RuntimeShader https://developer.android.com/reference/android/graphics/Run... https://developer.android.com/about/versions/13/features#shaders https://developer.android.com/about/versions/13/features#sha... ?
- deleted 5y ago
  [deleted]

resonious 5y ago

I think it is. I see all kinds of 2D projects claiming to be "GPU accelerated" - for example GTK, KDE, web browsers. I'm not sure how much of the actual processing is done on the GPU, but it's enough to call it "accelerated"!

ben-schaaf 5y ago

Browsers largely make use of the GPU for UI rendering. Direct2D, Cocoa, QT and GTK(4) are all hardware accelerated as well. So not really sure what you mean?

Jasper_ 5y ago

The linked article covers the major challenges that makes it difficult to adapt the GPU to 2D vector graphics rendering. tl;dr 2D cares about shapes and curves and proper antialiased coverage of super small shapes, the traditional rasterization GPU pipeline is very good at triangles and textures and have limited coverage options. Direct2D, Qt and GTK+ still do a good portion of the graphics work on the CPU, and only use GPU for composition. Some limited graphics can be done on the GPU, usually with quality tradeoffs. Font rasterization is still done the CPU, and uploaded to the GPU as a texture. Newer libraries like Pathfinder, Spinel, piet-gpu all work by not using the triangle rasterization parts but instead treating the GPU as a general-purpose parallel processor with compute shaders.

pengaru 5y ago

Programming graphics, especially 2D stuff, is far more ergonomic and convenient in your native language already executing on the CPU. If you can get away with it performance wise, there's really no incentive to incur the myriad obnoxious bullshit inherent in GPU programming. But with the advent of high dpi displays it's become problematic to do even simple 2D/UI rendering on the CPU just because of the enormous quantity of pixels. When you pull in GPU support, now you're stuck having to pick a backend (gl/vulkan/d3d/metal) or some compatibility layer to make some/all of them work. You have to write shaders, you have to constantly move state in/out of the GPU across this GPU:CPU API boundary. It's just a total clusterfuck best avoided if possible.

copperx 5y ago
Naive question: Can't you leverage a game engine for this? Why do you have to do all the low level work from scratch?
- pengaru 5y ago
  I'm not familiar with modern game engines, but I'd be very surprised if any of them managed to eliminate the utterly unnatural reality of writing shaders vs. writing classical 2D rendering algorithms operating on a linear buffer of pixels in memory. For concurrency reasons shaders logically run on a single pixel. Gone are your longstanding algorithms for doing simple things like bresenham line-drawing, or something as simple as drawing a filled box like this: for (int y = box.y; y < box.y + box.h; y++) for (int x = box.x; x < box.x + box.w; x++) FB[y * FB_STRIDE + x] = box.color; Nope, not happening in a shader. Every shader basically executes in isolation on a pixel and you have to operate from a sort of dead-reckoning perspective. No more sequential loops iterating rows and lines, a fashion which we have literally decades of graphics programming publications explaining how to do things. Not to mention how natural it is to think about things that way, since it closely resembles drawing on paper. In shaders you often end up doing things that feel utterly absurd in terms of overhead because of this "you run on an arbitrary pixel" perspective. Oftentimes you're writing some kind of distance function, where previously you would have written a loop iterating across lines and rows advancing some state as you step through the pixels. In a shader it's like the paper is covered with thousands of pencils that don't move, and the shader program just determines what color the pencil should be based on its location. GPU programming is plain annoying, even without the GPU API fragmentation clusterfuck. Especially if you've been writing 2D stuff on the CPU for decades.

kllrnohj 5y ago

It is. There's a terminology problem in play here. Throughout this article 2D does not mean 2D, it means "arbitrary complexity paths, eg bezier curves". This is a small subset of 2D as makes up a UI. It'd be like saying 3D exclusively means infinitely detailed tesellated shapes with path traced rendering. That's definitely an area of 3D that exists, but of course is not at all the entirety of 3D in practice in eg games or movies. Rather it's more like the holy grail. Same thing here with eg. SVGs. GPU accelerating SVGs is stupidly complicated because it's an inherently serial algorithm, and GPUs are poop at that. But how much of your 2D UI is made up of that? Text is in that same category but how much else? Typically very little. Maybe a few icons, but that tends to be about it. Instead you have higher level shapes, like round rectangles. And those you can do with a GPU quite easily. Similarly images are usually just a textured quad. Again trivial for a GPU. You could describe them as paths if you had a fully generic, fully accelerated path rendering system. But nobody has that, so nobody actually describes them like that. So very nearly all 2D/UI systems are GPU accelerated. It'd be perhaps more accurate to call them hybrid renderers. Things like fonts are just CPU rendered because CPUs are better at it, but the GPU is doing all the fills, gradients, "simple" shapes, texturing, etc...

Jasper_ 5y ago
I don't know if I'd go that far -- icons and text are vector paths. Strokes and drop shadows (aka blurs) are all things that GPUs aren't great particularly great at. Simple shapes like rounded rectangles, GPUs can be OK at, but you'd have overdraw problems if done naively. I've worked on 2D rendering engines, so I've seen the content thrown at it in the wild. Very rarely do you have a simple case. GitHub's buttons are maybe the simplest example I can think of, and they have strokes (GPUs: ugh) on a filled rounded border (GPUs: ugh), with text inside (GPUs: ugh), sometimes with a text shadow (GPUs: ugh). It can be done, but you basically have to get away from triangles and move into research methods which are exceptionally more tricky, aka the stuff in Pathfinder and piet-gpu.
- teknopaul 5y ago
  Showing my ignorance here: Why we don't have dedicated graphics hardware for 2D? Feels like moving 2D jobs to a fast 3D engine is wrong approach at the top level.

kmeisthax 5y ago

GPUs want to draw triangles, and in fact only know how to draw triangles[0]. Pretty much all graphics API innovation has been around either feeding more triangles to the GPU faster, letting the GPU create more triangles after they've been sent, or finding cool new ways to draw things on the surface of those triangles. 2D/UI breaks down into drawing curves, either as filled shapes or strokes. The preferred representation of such is a Bezier spline, which is a series of degree-three[1] polynomials that GPUs have zero support for rasterizing. Furthermore, strokes on the basis functions of those splines are not polynomials, but an even more bizarre curve type called an algebraic curve. You cannot just offset the control points to derive a stroke curve; you either have to approximate the stroke itself with Beziers, or actually draw a line sequentially in a way that GPUs are really not capable of doing. The four things you can do to render 2D/UI on a GPU is: - Tesselate the Bezier spline with a series of triangles. Lyon does this. Bezier curves make this rather cheap to do, but this requires foreknowledge of what scale the Bezier will be rendered at, and you cannot adjust stroke sizes at all without retessellating. - Send the control points to the GPU and use hardware tessellation to do the above per-frame. No clue if anyone does this. - Don't tessellate at all, but send the control points to the GPU as a polygonal mesh, and draw the actual Beziers in the fragment shaders for each polygon. For degree-two/quadratics there are a series of coordinate transforms that you can do which conveniently map all curves to one UV coordinate space; degree-three/cubics require a lot more attention in order to render correctly. If I remember correctly Mozilla Pathfinder does this[2]. - Send a signed distance field and have the GPU march it to render curves. I don't know much about this but I remember hearing about this a while back. All of these approaches have downsides. Tessellation is the approach I'm most familiar with because it's used heavily in Ruffle; so I'll just explain it's downsides to give you a good idea of why this is a huge problem: - We can't support some of Flash's weirder rendering hacks, like hairline strokes. Once we have a tessellated stroke, it will always be that width regardless of how we scale the shape. But hairlines require that the stroke get proportionally bigger as the shape gets smaller. In Flash, they were rendering on CPU, so it was just a matter of saying "strokes are always at least 1px". - We have to sort of guess what scale we want to render at and hope we have enough detail that the curves look like curves. There's one particular Flash optimization trick that consistently breaks our detail estimation and causes us to generate really lo-fi polygons. - Tessellation requires the curve shape to actually make sense as a sealed hull. We've exposed numerous underlying bugs in lyon purely by throwing really complicated or badly-specified Flash art at it. - All of this is expensive, especially for complicated shapes. For example, pretty much any Homestuck SWF will lock up your browser for multiple minutes as lyon tries to make sense of all of Hussie's art. This also precludes varying strokes by retessellating per-frame, which would otherwise fix the hairline stroke problem I mentioned above. [0] AFAIK quads are emulated on most modern GPUs, but they are just as useless for 2D/UI as triangles are. [1] Some earlier 2D systems used degree-two Bezier splines, including most of Adobe Flash. [2] We have half a PR to use this in Ruffle, but it was abandoned a while back.

eru 5y ago

That's exactly what eg the game Shovel Knight is doing. See eg https://www.youtube.com/watch?v=vjENktnbCaE https://www.youtube.com/watch?v=vjENktnbCaE