4 ms·

As someone who knows literally nothing about the GPU, the notion that the GPU might not be good for UI graphics seems...odd? Like, it can handle 3D graphics rig

As someone who knows literally nothing about the GPU, the notion that the GPU might not be good for UI graphics seems...odd? Like, it can handle 3D graphics right? Aren't those waaaay more computationally expensive than 2D graphics? I know that I'm missing something important, but I don't know what it is.

raphlinus 7y ago
It's complicated. 2D graphics aren't inherently harder or require more computation than 3D, but they require very different optimizations. I think it boils down to the fact that 2D graphics is ultimately data structure heavy, whereas 3D graphics can often be represented as a huge array of triangles. So there's a lot of literature (some of which I linked, then those papers have a deep literature review), but no one obviously best solution. One thing I think I contributed is a relatively simple solution, especially for doing almost all the work on GPU rather than relying on CPU for things.
- incompatible 7y ago
  Is there any particular reason why recent / future GPUs would be worse at 2D graphics than older versions? I.e., what needs to change, when current methods for 2D graphics have worked for years? Also not knowing much about the field, GPUs seem to have a lot of unpleasantness in the form of incompatible and/or proprietary APIs, so implementing anything that's supposed to be cross platform (also to old and new GPUs, and future GPUs as they are released) may be hard work.
  - raphlinus 7y ago
    What makes you say that 2D graphics has worked well? Scrolling often janks, UI designers constantly have to work around limitations in the imaging model and performance quirks (for a long time, clipping to a rounded rect would kill performance), and there were lots of shortcuts with things like gamma for blending. I'm hoping to build something way better, where the UI always updates smoothly, and the UI designer doesn't have to be conscious of the limitations in the renderer.
    - tenaciousDaniel 7y ago
      > I'm hoping to build something way better Are you talking about a specific project you're working on? I'm currently building a UI tool for designers, and I'm extremely interested to hear about any work in that area.
      - raphlinus 7y ago
        Yes. I'll have more to say soon, but I'm basically building what I think is next-gen UI infrastructure in Rust. That's a project with perhaps overly ambitious scope, so I'm starting with one specific app, namely a font editor. That will be a great showcase for beautifully antialiased vector paths and smooth-as-silk scrolling and interaction, I think.
  - kllrnohj 7y ago
    > I.e., what needs to change, when current methods for 2D graphics have worked for years? Very nearly every 2D graphics system uses software rasterization for handling generic paths. Many 2D graphics renderers are even just mostly software, using limited GPU acceleration for just basic blitting & blending of textures. What changed is mobile happened, radically shifting the resolution vs. CPU performance balance, along with decent GPUs being suddenly a basic feature. Getting generic vector path performance up to par in this world to be a basic building block is now interesting.
Jasper_ 7y ago
This is a blog post I've been meaning to write for a while now, because it's definitely surprising to learn. But here's the key points: GPUs are good at triangles. Triangles are very simple -- three planar equations tells you if you are inside or outside the shape. 2D graphics is typically specified in overlapping curves. Determining whether a pixel should be filled or not is a lot trickier -- even determining which side of a Bezier curve you are on is a non-obvious and numerically unstable problem (nitpickers corner: I know about Citardauq and Slug's root finding approximations). Part of this is a tooling and culture problem. There's a tradition and expectation in 3D to take higher-order implicit surfaces like Catmull-Clark and "bake" them down to triangles, and part of the skillset of a 3D artist is to manage this transition. 2D has no such expectation -- the API is "curveTo" and it's required realtime. And this API is not only assumed in platforms, but also TrueType fonts, and SVG graphics. There's also some other differences -- in 3D graphics, you can be imprecise with your antialiasing because the viewport constantly shifting with perspective can mask a lot of ugliness, and typical scenes will have a lot of detail and texture. In 2D, the standard is clean silhouettes (which most post-processing AA techniques do not work well on), and stability of small shapes and details (you don't want your letters to bubble and jitter around as you scroll a page or type, like you might see from a temporal antialiasing algorithm). Ultimately, 3D scenes are constructed mostly with materials and textures -- fancy things inside the shapes, but 2D is the opposite -- very simple solid color or gradient fills on complex shapes.
bmer 7y ago
This post is a good introduction to the challenges of rendering high quality vector graphics with good performance: https://nical.github.io/posts/rustfest-paris-01.html https://nical.github.io/posts/rustfest-paris-01.html

raphlinus 7y ago

It's complicated. 2D graphics aren't inherently harder or require more computation than 3D, but they require very different optimizations. I think it boils down to the fact that 2D graphics is ultimately data structure heavy, whereas 3D graphics can often be represented as a huge array of triangles. So there's a lot of literature (some of which I linked, then those papers have a deep literature review), but no one obviously best solution. One thing I think I contributed is a relatively simple solution, especially for doing almost all the work on GPU rather than relying on CPU for things.

incompatible 7y ago
Is there any particular reason why recent / future GPUs would be worse at 2D graphics than older versions? I.e., what needs to change, when current methods for 2D graphics have worked for years? Also not knowing much about the field, GPUs seem to have a lot of unpleasantness in the form of incompatible and/or proprietary APIs, so implementing anything that's supposed to be cross platform (also to old and new GPUs, and future GPUs as they are released) may be hard work.
- raphlinus 7y ago
  What makes you say that 2D graphics has worked well? Scrolling often janks, UI designers constantly have to work around limitations in the imaging model and performance quirks (for a long time, clipping to a rounded rect would kill performance), and there were lots of shortcuts with things like gamma for blending. I'm hoping to build something way better, where the UI always updates smoothly, and the UI designer doesn't have to be conscious of the limitations in the renderer.
  - tenaciousDaniel 7y ago
    > I'm hoping to build something way better Are you talking about a specific project you're working on? I'm currently building a UI tool for designers, and I'm extremely interested to hear about any work in that area.
    - raphlinus 7y ago
      Yes. I'll have more to say soon, but I'm basically building what I think is next-gen UI infrastructure in Rust. That's a project with perhaps overly ambitious scope, so I'm starting with one specific app, namely a font editor. That will be a great showcase for beautifully antialiased vector paths and smooth-as-silk scrolling and interaction, I think.
- kllrnohj 7y ago
  > I.e., what needs to change, when current methods for 2D graphics have worked for years? Very nearly every 2D graphics system uses software rasterization for handling generic paths. Many 2D graphics renderers are even just mostly software, using limited GPU acceleration for just basic blitting & blending of textures. What changed is mobile happened, radically shifting the resolution vs. CPU performance balance, along with decent GPUs being suddenly a basic feature. Getting generic vector path performance up to par in this world to be a basic building block is now interesting.

Jasper_ 7y ago

This is a blog post I've been meaning to write for a while now, because it's definitely surprising to learn. But here's the key points: GPUs are good at triangles. Triangles are very simple -- three planar equations tells you if you are inside or outside the shape. 2D graphics is typically specified in overlapping curves. Determining whether a pixel should be filled or not is a lot trickier -- even determining which side of a Bezier curve you are on is a non-obvious and numerically unstable problem (nitpickers corner: I know about Citardauq and Slug's root finding approximations). Part of this is a tooling and culture problem. There's a tradition and expectation in 3D to take higher-order implicit surfaces like Catmull-Clark and "bake" them down to triangles, and part of the skillset of a 3D artist is to manage this transition. 2D has no such expectation -- the API is "curveTo" and it's required realtime. And this API is not only assumed in platforms, but also TrueType fonts, and SVG graphics. There's also some other differences -- in 3D graphics, you can be imprecise with your antialiasing because the viewport constantly shifting with perspective can mask a lot of ugliness, and typical scenes will have a lot of detail and texture. In 2D, the standard is clean silhouettes (which most post-processing AA techniques do not work well on), and stability of small shapes and details (you don't want your letters to bubble and jitter around as you scroll a page or type, like you might see from a temporal antialiasing algorithm). Ultimately, 3D scenes are constructed mostly with materials and textures -- fancy things inside the shapes, but 2D is the opposite -- very simple solid color or gradient fills on complex shapes.

bmer 7y ago

This post is a good introduction to the challenges of rendering high quality vector graphics with good performance: https://nical.github.io/posts/rustfest-paris-01.html https://nical.github.io/posts/rustfest-paris-01.html