10 ms·

2D Graphics on Modern GPU (2019)

ink_13 6y ago
This would have been a lot better with examples in the form of rendered images or perhaps even a video. Maybe it's just my lack of background in graphics, but I had a lot of trouble grasping what the author was attempting to communicate without a concrete example.
- eternalban 6y ago
  It's about taking known 2D graphics and UI approaches which were developed for CPUs and looking at effective rendering engine architectures doing the same using GPUs. Terms such as "scene graph", "retained mode UI", etc. are those existing 2d graphics matter. So the approach, afaiu, is a data layout for the scene graph that basically is the more domain general concern of mapping graph e.g. Linked List, datastructures (that are CPU friendly) to array forms (GPU friendly) suitable for parallel treatment. Other GPU concerns, such as minimizing global traffic by local caching, and mapping thread groups to tiles. I found the idea of having the scene graph resident in GPU to be interesting. (note to author: "serialization" comes from networking roots of serializing a data structure for transmision over the net. So, definitely serial. /g)
- Agentlien 6y ago
  I understand where you are coming from. There is a lot of jargon and it assumes familiarity with many concepts. I think any explanatory images which would help someone unfamiliar with the field would need to be accompanied by quite a bit of explanation. One thing which I think made reading this a bit more work than necessary is that it feels like it's prattling on about a lot of tangential details and never quite gets to the point. edit: a prime example of an unnecessary aside is mentioning the warp/wavefront/subgroup terminology. I feel anyone in the target audience should know this already and it's not really relevant to what's being explained.
  - skohan 6y ago
    It doesn't seem to be a finished work. I guess this is more of a journal entry on the author's initial takeaways from a week-long coding retreat.
- MattRix 6y ago
  It wouldn’t have been better, it just would have been more high level and generalized, but I don’t think that’s what the author was going for. I found the amount of detail refreshing, and as someone about to make a GPU based 2D display tree renderer, it was written at just the right level to be quite useful.
skohan 6y ago
Is this approach novel? For instance is Apple's approach to native UI rendering doing UI rendering on the CPU, or using a 3D renderer?
- stephen_g 6y ago
  The author mentions PathFinder, the GPU font renderer from Servo a lot, so there do seem to be existing systems that do things in that way. I'm not 100% sure though about Apple and other's approach though - definitely when compositing desktop environments were new, the way it was done was to software render UI elements into textures, and then use the GPU to composite it all together. I assume more is being done on the GPU now but it may not actually be all that performance critical for regular UIs (he talks about things like CAD which are more performance sensitive).
- Animats 6y ago
  He's describing roughly the feature set of Flash, which is a system for efficiently putting 2D objects on top of other 2D objects.
  - skohan 6y ago
    No I mean is the approach of doing this on the GPU actually novel
    - Animats 6y ago
      Games often use the GPU for their 2D elements. It's inefficient to do a window system that way, because you have to update on every frame, but if you're updating the whole window on every frame anyway, it doesn't add cost. As the original poster points out, it does run down the battery vs. a "window damage" approach.
      - skohan 6y ago
        Yes but games typically use standard rasterization to render 2D elements. My question is whether using compute to "simulate cpu rendering" here is a novel approach.
danybittel 6y ago
It would be fantastic if something like this were part of the modern apis. Vulkan, Metal DX12. But I guess it's not as sexy as raytracing.
- moonchild 6y ago
  Nvidia tried to make it happen[0]. Sadly, it didn't catch on. 0. https://developer.nvidia.com/nv-path-rendering https://developer.nvidia.com/nv-path-rendering
  - slimsag 6y ago
    It's still there on all NVIDIA GPUs as an extension, just nobody uses it. IMO it didn't catch on because all three of these points: 1. It only works on NVIDIA GPUs, and is riddled with joint patents from NVIDIA and Microsoft forbidding anyone like AMD or Intel from supporting it. 2. It's hard to use: you need to get your data into a format it can consume, usage is non-trivial, and often video game artists are already working with rasterized textures anyway so it's easy to omit. 3. Vector graphics suck for artists. The number of graphic designers I have met (who are the most likely subset of artists to work with vector graphics) that simply hate or do not understand the cubic bezier curve control points in Adobe, Inkscape, and other tools is incredible.
    - derefr 6y ago
      > It only works on NVIDIA GPUs, and is riddled with joint patents from NVIDIA and Microsoft forbidding anyone like AMD or Intel from supporting it. Why do companies do this? What do they expect to get out of creating an API that is proprietary to their specific non-monopoly device, and that therefore very obviously nobody will ever actually use?
      - numlock86 6y ago
        Well, you could always implement a fallback for unsupported devices and advertise for better performance on supported devices. Implementation will usually be "sponsored" by the one providing the technology. If the feature then gets adopted enough others will have to come up with something similar or support it native, too. This happens all the time with GPU and CPU vendors, both in gaming and industry.
moonchild 6y ago
Something else worth looking at: the slug font renderer[0]. Sadly it's patented, but the paper[1] is there for those of you in the EU. 0. http://sluglibrary.com/ http://sluglibrary.com/ 1. http://jcgt.org/published/0006/02/02/paper.pdf http://jcgt.org/published/0006/02/02/paper.pdf
- Jasper_ 6y ago
  Slug isn't great for lots of shapes since it does the winding order scanning per-pixel on the pixel shader. It does have a novel quadratic root-finder. Put simply, it's better suited to fonts than large vector graphics.
  - vg_head 6y ago
    I've once implemented the basic idea behind the algorithm used in slug(described in the paper [1], though without the 'band' optimization, I just wanted to see how it works), and I agree with you, the real innovation is in that quadratic root-finder. It can tell you whether you are inside or outside just by manipulating the three control points of a curve, it's very fast, what remains to be done is to use an acceleration data structure so that you don't have to check for every curve. That works very well for quadratic Bézier curves, in the paper it says that it can be easily extended to cubics, though no example is provided(and I doubt it's trivial). What I think would be hard with Slug's method is extending it to draw gradients, shadows, basically general vector graphics like you say. Eric Lengyel on his twitter showed a demo [2] using Slug to render general vector graphics, but I'm not sure of how many features it supports, but it definitely supports cubic Bézier curves. I'd also like to add that the algorithm didn't impress me with how the text looks at small sizes, which I think is very important in general, though maybe not so much for games(maybe I just didn't implement it correctly). [1] "GPU-Centered Font Rendering Directly from Glyph Outlines" http://jcgt.org/published/0006/02/02/ http://jcgt.org/published/0006/02/02/ [2] https://twitter.com/EricLengyel/status/1190045334791057408 https://twitter.com/EricLengyel/status/1190045334791057408
- korijn 6y ago
  Any alternative solutions for the problem of GPU text rendering (that are not patent infringing)?
slmjkdbtl 6y ago
always appreciate raph's work on rendering and UI programming, but want to ask a somewhat unrelated question to this post: does anyone have a lot of experience doing 2d graphics on the cpu? i wonder if there'll be a day we're confident doing all 2d stuff on the cpu since cpus are much easier to work with and have much more control, i also read some old 3d games are also using software rendering and did well on old hardwares, that gave me a lot of confidence in software render every (lowrez) thing
- Jasper_ 6y ago
  Yes? We know how to write scanline renderers for 2D graphics. They're not that hard, a simple one can be done in ~100 lines of code or so, see my article here https://magcius.github.io/xplain/article/rast1.html https://magcius.github.io/xplain/article/rast1.html
  - slmjkdbtl 6y ago
    Tanks for the amazing article! I wonder if you met any performance annoyances / bottlenecks when doing actual GUI / game dev with this?
    - iainmerrick 6y ago
      You just need to render your font on the CPU once, and upload mipmaps to the GPU (either signed distance fields, or just sharpened textures, that works absolutely fine too). I think all this GPU font rendering stuff is a bit of a red herring. Are any popular apps or games actually making heavy use of it?
  - olau 6y ago
    Enjoyable article, thanks!
- bumbada 6y ago
  I have lots of experience, created several font renderers in the CPU and GPU. No, doing CPU drawing is too inefficient. Without too much trouble you get 50x more efficiency in the GPU, that is, you can draw 50frames per every frame in the CPU, using the same amount of energy and time. If you go deeper, low level with Vulkan-Metal and specially control the GPU memory, you can get 200x, being way harder to program. CPU drawing is very useful for testing: You create the reference you compare the GPU stuff with. CPU drawing is the past, not the future.
Const-me 6y ago
The quality is not good, and the performance is not even mentioned. I have used different approach: https://github.com/Const-me/Vrmac#vector-graphics-engine https://github.com/Const-me/Vrmac#vector-graphics-engine My version is cross-platform, tested with Direct3D 12 and GLES 3.1. My version does not view GPUs as SIMD CPU, it actually views them as a GPU. When rendering a square without anti-aliasing, the library will render 2 triangles. When rendering a filled square with anti-aliasing, the library will render about 10 triangles, large opaque square in the center, and thin border about 1 pixel thick around it for AA. It uses hardware Z buffer with early Z rejection to save pixel shaders and fill rate. It uses screen-space derivatives in the pixel shader for anti-aliasing. It renders arbitrarily complex 2D scenes with only two draw calls, one front to back with opaque stuff, another back to front with translucent stuff. It does not have quality issues with stroked lines much thinner than 1px.
- skohan 6y ago
  Does your approach allow for quality optimizations like sub pixel rendering for arbitrary curves? It seems like this is what is interesting about this approach. Also in terms of “two draw calls”, does that include drawing text as part of your transparent pass, or are you assuming that your window contents are already rendered to textures?
  - Const-me 6y ago
    > Does your approach allow for quality optimizations like sub pixel rendering for arbitrary curves? The library only does grayscale AA for vector graphics. Subpixel rendering implemented for text but comes with limitations: only works when text is not transformed (or transformed in specific ways, like rotated 180°, or horizontally flipped), and you need to pass background color behind the text to the API. Will only look good if the text is on solid color background, or on slow-changing gradient. Sub pixel AA is hard for arbitrary backgrounds. I’m not sure many GPUs support the required blend states, and workarounds are slow. > does that include drawing text as part of your transparent pass Yes, that includes text and bitmaps. Here’s two HLSL shaders who do that, the GPU abstraction library I’ve picked recompiles the HLSL into GLSL or others on the fly: https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Shaders/drawVS.hlsl https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Sha... https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Shaders/drawPS.hlsl https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Sha... These shaders are compiled multiple times with different set of preprocessor macros, but I did test with all of them enabled at once.
raphlinus 6y ago
Hi again! This post was an early exploration into GPU rendering. The work continues as piet-gpu, and, while it's not yet a complete 2D renderer, there's good progress, and also an active open source community including people from the Gio UI project. I've not long ago implemented nested clipping (which can be generalized to blend modes) and have a half-finished blog post draft. I'm also working on this as my work as a researcher on the Google Fonts team. Feel free to ask questions in this thread - I probably won't follow up with everything, as the discussion is pretty sprawling.
- lame88 6y ago
  One small error (I think) - I noticed your link to pathfinder linked to someone's 2020 fork of the repository rather than the upstream servo repository.
  - littlestymaar 6y ago
    > someone's 2020 fork pcwalton was the developer behind pathfinder at Mozilla (but was part of past summer's layoff).
klaussilveira 6y ago
Loosely related: Blend2D has been innovating a lot on this space. https://blend2d.com/ https://blend2d.com/
choxi 6y ago
I spent a couple years learning graphics programming to build an iPad app for creating subtle animated effects. The idea was kind of like Procreate but if you had "animated brushes" that produced glimmer or other types of looping animated effects. From what I've read, the technique behind most digital brushes is to render overlapping "stamps" over the stroke. They're spaced closely enough that you can't actually see the stamps. But if you want to animate the stamps, you either have to store the stroke data as a very large sequence of stamp meshes or you can only work with the data in a raster format. The former is way too many meshes even with instancing, and the latter loses a lot of useful information about the stroke. Imagine you wanted to create a brush where the edges of the stroke kind of pulsate like a laser beam, you ideally want to store that stroke data in a vector format to make it easier to identify e.g. centers and edges. But it turned out to be too challenging for me to figure out how to 1) build a vector representation of a stroke/path without losing some of the control over detail you get with the stamping technique and 2) efficiently render those vectors on the GPU. I'm not sure if this would help with the issues I ran into, but I'm definitely excited to see some focus on 2D rendering improvements!

This would have been a lot better with examples in the form of rendered images or perhaps even a video. Maybe it's just my lack of background in graphics, but I had a lot of trouble grasping what the author was attempting to communicate without a concrete example.

eternalban 6y ago
It's about taking known 2D graphics and UI approaches which were developed for CPUs and looking at effective rendering engine architectures doing the same using GPUs. Terms such as "scene graph", "retained mode UI", etc. are those existing 2d graphics matter. So the approach, afaiu, is a data layout for the scene graph that basically is the more domain general concern of mapping graph e.g. Linked List, datastructures (that are CPU friendly) to array forms (GPU friendly) suitable for parallel treatment. Other GPU concerns, such as minimizing global traffic by local caching, and mapping thread groups to tiles. I found the idea of having the scene graph resident in GPU to be interesting. (note to author: "serialization" comes from networking roots of serializing a data structure for transmision over the net. So, definitely serial. /g)
Agentlien 6y ago
I understand where you are coming from. There is a lot of jargon and it assumes familiarity with many concepts. I think any explanatory images which would help someone unfamiliar with the field would need to be accompanied by quite a bit of explanation. One thing which I think made reading this a bit more work than necessary is that it feels like it's prattling on about a lot of tangential details and never quite gets to the point. edit: a prime example of an unnecessary aside is mentioning the warp/wavefront/subgroup terminology. I feel anyone in the target audience should know this already and it's not really relevant to what's being explained.
- skohan 6y ago
  It doesn't seem to be a finished work. I guess this is more of a journal entry on the author's initial takeaways from a week-long coding retreat.
MattRix 6y ago
It wouldn’t have been better, it just would have been more high level and generalized, but I don’t think that’s what the author was going for. I found the amount of detail refreshing, and as someone about to make a GPU based 2D display tree renderer, it was written at just the right level to be quite useful.

skohan 6y ago

Is this approach novel? For instance is Apple's approach to native UI rendering doing UI rendering on the CPU, or using a 3D renderer?

stephen_g 6y ago
The author mentions PathFinder, the GPU font renderer from Servo a lot, so there do seem to be existing systems that do things in that way. I'm not 100% sure though about Apple and other's approach though - definitely when compositing desktop environments were new, the way it was done was to software render UI elements into textures, and then use the GPU to composite it all together. I assume more is being done on the GPU now but it may not actually be all that performance critical for regular UIs (he talks about things like CAD which are more performance sensitive).
Animats 6y ago
He's describing roughly the feature set of Flash, which is a system for efficiently putting 2D objects on top of other 2D objects.
- skohan 6y ago
  No I mean is the approach of doing this on the GPU actually novel
  - Animats 6y ago
    Games often use the GPU for their 2D elements. It's inefficient to do a window system that way, because you have to update on every frame, but if you're updating the whole window on every frame anyway, it doesn't add cost. As the original poster points out, it does run down the battery vs. a "window damage" approach.
    - skohan 6y ago
      Yes but games typically use standard rasterization to render 2D elements. My question is whether using compute to "simulate cpu rendering" here is a novel approach.

danybittel 6y ago

It would be fantastic if something like this were part of the modern apis. Vulkan, Metal DX12. But I guess it's not as sexy as raytracing.

moonchild 6y ago
Nvidia tried to make it happen[0]. Sadly, it didn't catch on. 0. https://developer.nvidia.com/nv-path-rendering https://developer.nvidia.com/nv-path-rendering
- slimsag 6y ago
  It's still there on all NVIDIA GPUs as an extension, just nobody uses it. IMO it didn't catch on because all three of these points: 1. It only works on NVIDIA GPUs, and is riddled with joint patents from NVIDIA and Microsoft forbidding anyone like AMD or Intel from supporting it. 2. It's hard to use: you need to get your data into a format it can consume, usage is non-trivial, and often video game artists are already working with rasterized textures anyway so it's easy to omit. 3. Vector graphics suck for artists. The number of graphic designers I have met (who are the most likely subset of artists to work with vector graphics) that simply hate or do not understand the cubic bezier curve control points in Adobe, Inkscape, and other tools is incredible.
  - derefr 6y ago
    > It only works on NVIDIA GPUs, and is riddled with joint patents from NVIDIA and Microsoft forbidding anyone like AMD or Intel from supporting it. Why do companies do this? What do they expect to get out of creating an API that is proprietary to their specific non-monopoly device, and that therefore very obviously nobody will ever actually use?
    - numlock86 6y ago
      Well, you could always implement a fallback for unsupported devices and advertise for better performance on supported devices. Implementation will usually be "sponsored" by the one providing the technology. If the feature then gets adopted enough others will have to come up with something similar or support it native, too. This happens all the time with GPU and CPU vendors, both in gaming and industry.

moonchild 6y ago

Something else worth looking at: the slug font renderer[0]. Sadly it's patented, but the paper[1] is there for those of you in the EU. 0. http://sluglibrary.com/ http://sluglibrary.com/ 1. http://jcgt.org/published/0006/02/02/paper.pdf http://jcgt.org/published/0006/02/02/paper.pdf

Jasper_ 6y ago
Slug isn't great for lots of shapes since it does the winding order scanning per-pixel on the pixel shader. It does have a novel quadratic root-finder. Put simply, it's better suited to fonts than large vector graphics.
- vg_head 6y ago
  I've once implemented the basic idea behind the algorithm used in slug(described in the paper [1], though without the 'band' optimization, I just wanted to see how it works), and I agree with you, the real innovation is in that quadratic root-finder. It can tell you whether you are inside or outside just by manipulating the three control points of a curve, it's very fast, what remains to be done is to use an acceleration data structure so that you don't have to check for every curve. That works very well for quadratic Bézier curves, in the paper it says that it can be easily extended to cubics, though no example is provided(and I doubt it's trivial). What I think would be hard with Slug's method is extending it to draw gradients, shadows, basically general vector graphics like you say. Eric Lengyel on his twitter showed a demo [2] using Slug to render general vector graphics, but I'm not sure of how many features it supports, but it definitely supports cubic Bézier curves. I'd also like to add that the algorithm didn't impress me with how the text looks at small sizes, which I think is very important in general, though maybe not so much for games(maybe I just didn't implement it correctly). [1] "GPU-Centered Font Rendering Directly from Glyph Outlines" http://jcgt.org/published/0006/02/02/ http://jcgt.org/published/0006/02/02/ [2] https://twitter.com/EricLengyel/status/1190045334791057408 https://twitter.com/EricLengyel/status/1190045334791057408
korijn 6y ago
Any alternative solutions for the problem of GPU text rendering (that are not patent infringing)?

slmjkdbtl 6y ago

always appreciate raph's work on rendering and UI programming, but want to ask a somewhat unrelated question to this post: does anyone have a lot of experience doing 2d graphics on the cpu? i wonder if there'll be a day we're confident doing all 2d stuff on the cpu since cpus are much easier to work with and have much more control, i also read some old 3d games are also using software rendering and did well on old hardwares, that gave me a lot of confidence in software render every (lowrez) thing

Jasper_ 6y ago
Yes? We know how to write scanline renderers for 2D graphics. They're not that hard, a simple one can be done in ~100 lines of code or so, see my article here https://magcius.github.io/xplain/article/rast1.html https://magcius.github.io/xplain/article/rast1.html
- slmjkdbtl 6y ago
  Tanks for the amazing article! I wonder if you met any performance annoyances / bottlenecks when doing actual GUI / game dev with this?
  - iainmerrick 6y ago
    You just need to render your font on the CPU once, and upload mipmaps to the GPU (either signed distance fields, or just sharpened textures, that works absolutely fine too). I think all this GPU font rendering stuff is a bit of a red herring. Are any popular apps or games actually making heavy use of it?
- olau 6y ago
  Enjoyable article, thanks!
bumbada 6y ago
I have lots of experience, created several font renderers in the CPU and GPU. No, doing CPU drawing is too inefficient. Without too much trouble you get 50x more efficiency in the GPU, that is, you can draw 50frames per every frame in the CPU, using the same amount of energy and time. If you go deeper, low level with Vulkan-Metal and specially control the GPU memory, you can get 200x, being way harder to program. CPU drawing is very useful for testing: You create the reference you compare the GPU stuff with. CPU drawing is the past, not the future.

Const-me 6y ago

The quality is not good, and the performance is not even mentioned. I have used different approach: https://github.com/Const-me/Vrmac#vector-graphics-engine https://github.com/Const-me/Vrmac#vector-graphics-engine My version is cross-platform, tested with Direct3D 12 and GLES 3.1. My version does not view GPUs as SIMD CPU, it actually views them as a GPU. When rendering a square without anti-aliasing, the library will render 2 triangles. When rendering a filled square with anti-aliasing, the library will render about 10 triangles, large opaque square in the center, and thin border about 1 pixel thick around it for AA. It uses hardware Z buffer with early Z rejection to save pixel shaders and fill rate. It uses screen-space derivatives in the pixel shader for anti-aliasing. It renders arbitrarily complex 2D scenes with only two draw calls, one front to back with opaque stuff, another back to front with translucent stuff. It does not have quality issues with stroked lines much thinner than 1px.

skohan 6y ago
Does your approach allow for quality optimizations like sub pixel rendering for arbitrary curves? It seems like this is what is interesting about this approach. Also in terms of “two draw calls”, does that include drawing text as part of your transparent pass, or are you assuming that your window contents are already rendered to textures?
- Const-me 6y ago
  > Does your approach allow for quality optimizations like sub pixel rendering for arbitrary curves? The library only does grayscale AA for vector graphics. Subpixel rendering implemented for text but comes with limitations: only works when text is not transformed (or transformed in specific ways, like rotated 180°, or horizontally flipped), and you need to pass background color behind the text to the API. Will only look good if the text is on solid color background, or on slow-changing gradient. Sub pixel AA is hard for arbitrary backgrounds. I’m not sure many GPUs support the required blend states, and workarounds are slow. > does that include drawing text as part of your transparent pass Yes, that includes text and bitmaps. Here’s two HLSL shaders who do that, the GPU abstraction library I’ve picked recompiles the HLSL into GLSL or others on the fly: https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Shaders/drawVS.hlsl https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Sha... https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Shaders/drawPS.hlsl https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Sha... These shaders are compiled multiple times with different set of preprocessor macros, but I did test with all of them enabled at once.

raphlinus 6y ago

Hi again! This post was an early exploration into GPU rendering. The work continues as piet-gpu, and, while it's not yet a complete 2D renderer, there's good progress, and also an active open source community including people from the Gio UI project. I've not long ago implemented nested clipping (which can be generalized to blend modes) and have a half-finished blog post draft. I'm also working on this as my work as a researcher on the Google Fonts team. Feel free to ask questions in this thread - I probably won't follow up with everything, as the discussion is pretty sprawling.

lame88 6y ago
One small error (I think) - I noticed your link to pathfinder linked to someone's 2020 fork of the repository rather than the upstream servo repository.
- littlestymaar 6y ago
  > someone's 2020 fork pcwalton was the developer behind pathfinder at Mozilla (but was part of past summer's layoff).

klaussilveira 6y ago

Loosely related: Blend2D has been innovating a lot on this space. https://blend2d.com/ https://blend2d.com/

choxi 6y ago

I spent a couple years learning graphics programming to build an iPad app for creating subtle animated effects. The idea was kind of like Procreate but if you had "animated brushes" that produced glimmer or other types of looping animated effects. From what I've read, the technique behind most digital brushes is to render overlapping "stamps" over the stroke. They're spaced closely enough that you can't actually see the stamps. But if you want to animate the stamps, you either have to store the stroke data as a very large sequence of stamp meshes or you can only work with the data in a raster format. The former is way too many meshes even with instancing, and the latter loses a lot of useful information about the stroke. Imagine you wanted to create a brush where the edges of the stroke kind of pulsate like a laser beam, you ideally want to store that stroke data in a vector format to make it easier to identify e.g. centers and edges. But it turned out to be too challenging for me to figure out how to 1) build a vector representation of a stroke/path without losing some of the control over detail you get with the stamping technique and 2) efficiently render those vectors on the GPU. I'm not sure if this would help with the issues I ran into, but I'm definitely excited to see some focus on 2D rendering improvements!