9 ms·

Show HN: 2D Graphics on Modern GPU

306 points by raphlinus 7y ago

adamnemecek 7y ago
I've been also working on a 2D UI on the GPU. It's quite amazing actually, compared with the code running on the CPU, the code is very terse. The setup can be somewhat verbose, but the shaders themselves are short. Like a 20 line shader can do miracles.
- dman 7y ago
  Any github links that are freely available?
- irascible 7y ago
  yeah shaders really force you to invert your thinking in a good way. Normally our brains spit out cluttered logic with a bunch of for loops and hash tables but shaders cleave all that away.
- mpfundstein 7y ago
  Do you have an example?
blogspam 7y ago
Honestly, HN is nothing but bike shedding by now. There's near zero hacker subculture to it anymore. Don't show HN anything. The ranking seems to be near arbitrary at this point, with nothing but media outlet vote brigades forcing their spyware.js ad-tech bloat trash megaphones to the front page. Fuck it, dude. Fuck this aggregator.
Waterluvian 7y ago
Something I've been really surprised and disappointed by is the lack of 2d vector graphics support in modern gaming engines. I'm an amateur enthusiast but I spent hours and hours unsuccessfully trying to find any decent engine where I could make games using svg or other vector formats.
- paulryanrogers 7y ago
  Godot has a skeletal animator built-in which is a nice middle ground for rotating sprites. I imagine the lack of dedicated vector art support could be that 3D engines can simulate that vectorised style.
- pcwalton 7y ago
  I would love to see someone step up and add Pathfinder support to Unity, Godot, and other popular engines. I'd be very interested in taking pull requests to this effect :)
- jra101 7y ago
  Defold might be an option. https://www.defold.com/ https://www.defold.com/
  - speps 7y ago
    Doesn't support vector graphics, only SVG import into a texture if I recall.
- DonHopkins 7y ago
  I know what you mean! Unity3D has no way to simply draw a circle or pie chart into a texture. It would be great to be able to use the canvas api (and any JavaScript library like d3 that uses canvas) to dynamically draw Unity3D textures. To address that problem (and others), I'm been developing a Unity3D extension called UnityJS, which integrates a web browser and JavaScript with Unity3D, so you can use the canvas api to draw images for use as 2D user interface overlays and 3D mesh textures in Unity. And of course the other (main) point of UnityJS is to use JavaScript for scripting and debugging Unity apps, and integrate Unity apps with off-the-shelf JavaScript web libraries (d3, socket.io, whatever). https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy.txt https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy... Drawing on canvases actually works pretty well in the WebGL build, because the JavaScript runs in the same browser tab and address space that's running the Unity app, so you can efficiently blit the canvas's pixels right into Unity's texture memory. (Although I don't believe there's a way to share the canvas texture directly through GPU memory, or even directly share Unity's ArrayBuffer memory, but it's fast enough for interactive user interface stuff, and millions of times better than the "portable" technique of writing out a data uri with a base 64 encoded compressed png file, which is the "standard" way of getting pixels out of the web browser! Web tech sure sucks!) https://github.com/SimHacker/UnityJS/blob/master/Libraries/UnityJS/StreamingAssets/UnityJS/bridge.jss#L640 https://github.com/SimHacker/UnityJS/blob/master/Libraries/U...
Jasper_ 7y ago
Impressive work, Raph! As I'm sure everyone knows by now, the PostScript imaging model is hard to make work on the GPU, and its traditional geometry pipeline isn't suited well for curves or analytic coverage-based AA. Since I come from the games space, a lot of my thoughts have been about baking as much data as possible: if we had infinite build time, or artist tooling akin to Maya, how would that help us develop better UIs? If we leave PostScript behind and think about new graphics models, what can we do? So a lot of my own research has been studying really old graphics history, since there's always fun ways of looking at the problem before we settled on one solution. I'll have to collect some of my research and demos and publish it sometime soon. Although, one thing I've noticed during my time in games is that frame time is the standard. If we speed up our rasterizer, we'll just add more junk to the scene until it hits our 60fps frame target. More blurs! More effects! :)
- raphlinus 7y ago
  These are all interesting questions. I've been following some of the stuff on Shadertoy, and I think that points the way. Most is 3d, but not all, and in particular there's a lot to be done with distance fields. I've been thinking about blur (and mentioned some approximation techniques), but it's expensive in any context. Maybe if we think in shaders instead of the blur tool, we'll find a visual palette that is both fresh and efficient.
  - DonHopkins 7y ago
    Distance fields are great! What do you think an alternative to PostScript's imaging model (Porter/Duff) formulated in terms of signed distance fields would be like? Much of Photoshop's power comes from representing the selection as first class alpha channels that you can use all of Photoshop's editing and filtering tools on, so you can select something with any of the the selection tools, then take a step back into "quick mask" mode and edit your selection in the "channel" domain, with any of the painting tools or filters, and store selections in extra channels for later use. I wonder how an image editor like Photoshop (and a rendering model in general) could directly and conveniently support signed distance fields as shapes, selections, and channels, as nicely as it supports alpha channels? Back in 1993 (5 years after Photoshop was created in 1988), it sounded revolutionary to use texture mapping as a fundamental drawing primitive, because nobody suspected some day everybody'd be carrying around more computrons than an SGI workstation in their phone. http://www.graficaobscura.com/texmap/index.html http://www.graficaobscura.com/texmap/index.html
amelius 7y ago
> I’m particularly interested in the rendering quality. Shouldn't the rendering be done using oversampling? E.g. instead of 100x100, you render on 400x400, then scale back down to 100x100 to get the best result. One of the problems is that anti-aliasing creates pixels which are half-on, e.g. at the edge of a line or polygon. Now if multiple edges meet near a certain pixel, then rendering the polygon(s) will touch a "half-on" pixel several times, and as a result it may receive the wrong opacity.
- raphlinus 7y ago
  It is true that is more correct (see https://github.com/pcwalton/pathfinder/issues/142 https://github.com/pcwalton/pathfinder/issues/142). But my motivation is much more user interface than document rendering, so it's quite reasonable to tell the UI designers, "don't do that," especially if the flip side is butter-smooth performance.
  - amelius 7y ago
    But UI designers are not always in complete control, e.g. if screen resolutions are different for different users.
    - pcwalton 7y ago
      In practice, if you avoid coincident edges, you're fine. Font rasterizers have been using analytic AA for a long time without problems. In theory, the lack of knowledge of render resolution makes that a concern, but in practice it's not too much of an issue.
    - raphlinus 7y ago
      Agreed, and there are some deep issues, for example the rendering of very thin strokes which is quite dependent on resolution. Also keep in mind that the 4x4 oversampling you mentioned does not give nice smooth edges. I'm working on font editing, so the quality and smoothness of the line, to most accurately represent the shape without artifacts, is paramount.
- 7y ago
pavlov 7y ago
What an interesting rabbit hole. Thanks for sharing your notes too! Re: > "As a digression, I find it amusing that the word for packing a data structure into a byte buffer is “serialization” even when it’s designed to be accessed in parallel. Maybe we should come up with a better term, as “parallel-friendly serialization” is an oxymoron." NeXTStep/Cocoa calls this simply "encoding" which always made sense to me.
kllrnohj 7y ago
> and many people believe that damage regions are obsolete Old-style damage regions are obsolete. But new-style ones leveraging buffer age & swap-with-damage are not. Vulkan changes this, but these older EGL extensions are good introductions to the concepts: https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_buffer_age.txt https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_... https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_partial_update.txt https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_... and https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_swap_buffers_with_damage.txt https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_... > Flutter is a good modern approach to this, and its “layers” are one of the keys to its performance. Flutter actually does it badly. Layers are terrible for performance because they are a source of jitter (and memory usage), and jitter often results in jank. They are necessary in some cases, like for correct alpha blending, but in general should be avoided if your goal is to be smooth & fast. You want to optimize for your slowest frames, not your fastest ones. Flutter's excessive layering can also rapidly be a net-loss in performance due to the overhead of render target switching, increase in memory bandwidth required, and fewer opportunities for overdraw avoidance & disabled blending.
- raphlinus 7y ago
  Agree on both fronts! (Also, your username seems familiar somehow - have we had discussions around this before? :) I didn't talk about this in the post because I haven't actually implemented fancy compositing, but based on what I've seen you never want to render to texture (which Flutter does sometimes) as it uses up scarce global GPU memory bandwidth, and if you're doing that using heuristics, then performance gets very unpredictable. I should express this more clearly, but the goal in piet-metal is not to do the fastest renderer (I'm sure it isn't), but the one with the most consistent and predictable performance.
  - kllrnohj 7y ago
    > but based on what I've seen you never want to render to texture (which Flutter does sometimes) Flutter's "layers" are rasterized to texture if they are unchanged for N frames (I think N = 3?). Which happens kind of a lot, and is (one of) the reasons Flutter has such a big RAM footprint. Flutter's layers remind me a lot of the hacky, terrible will-change CSS property. Other than that it's just a rather crude way to hack up the rendering commands into smaller display lists. Android has been doing this for ages with RenderNode with both display lists & transform properties on those display lists.
Retr0spectrum 7y ago
Is there a demo/source anywhere? I admit I only skimmed the article, for now.
- deleted 7y ago
  [deleted]
- raphlinus 7y ago
  Yes, https://github.com/linebender/piet-metal https://github.com/linebender/piet-metal , it's mac-only (for now).
tenaciousDaniel 7y ago
As someone who knows literally nothing about the GPU, the notion that the GPU might not be good for UI graphics seems...odd? Like, it can handle 3D graphics right? Aren't those waaaay more computationally expensive than 2D graphics? I know that I'm missing something important, but I don't know what it is.
- raphlinus 7y ago
  It's complicated. 2D graphics aren't inherently harder or require more computation than 3D, but they require very different optimizations. I think it boils down to the fact that 2D graphics is ultimately data structure heavy, whereas 3D graphics can often be represented as a huge array of triangles. So there's a lot of literature (some of which I linked, then those papers have a deep literature review), but no one obviously best solution. One thing I think I contributed is a relatively simple solution, especially for doing almost all the work on GPU rather than relying on CPU for things.
  - incompatible 7y ago
    Is there any particular reason why recent / future GPUs would be worse at 2D graphics than older versions? I.e., what needs to change, when current methods for 2D graphics have worked for years? Also not knowing much about the field, GPUs seem to have a lot of unpleasantness in the form of incompatible and/or proprietary APIs, so implementing anything that's supposed to be cross platform (also to old and new GPUs, and future GPUs as they are released) may be hard work.
    - raphlinus 7y ago
      What makes you say that 2D graphics has worked well? Scrolling often janks, UI designers constantly have to work around limitations in the imaging model and performance quirks (for a long time, clipping to a rounded rect would kill performance), and there were lots of shortcuts with things like gamma for blending. I'm hoping to build something way better, where the UI always updates smoothly, and the UI designer doesn't have to be conscious of the limitations in the renderer.
dragontamer 7y ago
> It’s often said that GPU is bad at data structures, but I’d turn that around. Most, but not all, data structures are bad at GPU. An extreme example is a linked list, which is still considered reasonable on CPU, and is the backbone of many popular data structures. Not only does it force sequential access, but it also doesn’t hide the latency of global memory access, which can be as high as 1029 cycles on a modern GPU such as Volta. You're almost correct here, from my understanding. But... I still feel like its important to note that Pointer-jumping exists, and linked-list like structures (linked lists, trees, and graphs) can be traversed in parallel in many cases. https://en.wikipedia.org/wiki/Pointer_jumping https://en.wikipedia.org/wiki/Pointer_jumping For pointer-jumping in GPUs / SIMD systems, I know that "Data Parallel Algorithms" discusses the technique as applied to a "reduce" operation aka a "scan": https://dl.acm.org/citation.cfm?id=7903 https://dl.acm.org/citation.cfm?id=7903 The requirement (not fully discussed in the 'Data Parallel Algorithms' article) is that you still need an array of nodes to be listed off by the SIMD system. But this array of nodes does NOT need to be sorted. As such, you can accomplish a "GPU Reduce" over a linked list in O(lg(n)) steps (assuming an infinite-core machine). This is way slower than the O(lg(n)) array traversal, but you're at least within the same asymptotic class.
DonHopkins 7y ago
>Other than that, the serialization format is not that exotic, broadly similar to FlatBuffers or Cap’n Proto. As a digression, I find it amusing that the word for packing a data structure into a byte buffer is “serialization” even when it’s designed to be accessed in parallel. Maybe we should come up with a better term, as “parallel-friendly serialization” is an oxymoron. Great observation! I get the feeling there must be a great punny term for the enigmatic oxymoron "parallel-friendly serialization", but I can't quite put my finger on it. (Reminds me of "de-optimizer" => "pessimizer", or "spectral frequency analysis and filtering" => "cepstral quefrency alanysis and liftering".) https://en.wikipedia.org/wiki/Cepstrum https://en.wikipedia.org/wiki/Cepstrum
- strmpnk 7y ago
  Marhsalling has a precedent in some languages. It makes a lot of sense. "serde[s]" always reminds me of sequencing of transmission more than a format concern of what goes where in a buffer.
iamleppert 7y ago
Very interesting work. I’m wondering if you’ve ever worked with vector tiles? Mostly it’s used for drawing large numbers of vectors on maps, but has a similar architecture and approach for 2D drawing on the GPU via an efficient command-based language based on the extent of the tile.
aasasd 7y ago
> Performant UI must use GPU effectively, and it’s increasingly common to write UI directly in terms of GPU rendering, without a 2D graphics API as in the intermediate layer. Y tho? I'm just a passerby in this topic, but I see this sentiment now and again these days, and I don't understand what happened. I don't remember hearing complaints of DirectDraw being too slow, for example.
- Jasper_ 7y ago
  DirectDraw wasn't a 2D graphics package; you probably mean GDI/GDI+. These libraries were used at at much smaller resolutions (1024x768, but fullscreen was pushing it, your window was smaller). GDI could roughly be used at 60Hz but it didn't support antialiasing or transparency, you needed GDI+ for that. And GDI+ was much slower, even at the smaller resolutions, so you tended to draw into a big bitmap ahead of time and scroll that. Animations in GDI+ were unheard of.
  - roel_v 7y ago
    I'm not sure what you mean by '2D graphics package', so it may be a semantics thing here, but DirectDraw was a 2D graphics API on top of DirectX. I only used it a bit and didn't find that for my uses, the added complexity was worth it over just using GDI, but DirectDraw did do GPU accelerated 2D rendering. How fast it was in light of today's requirements and hardware, I don't know.
    - Jasper_ 7y ago
      Are you confusing it with Direct2D? DirectDraw was an old API from the D3D5-ish era and basically allocated a front buffer you could draw to with GDI. It was not GPU accelerated.
      - roel_v 7y ago
        Well you got me wondering now on what I worked with, but https://en.wikipedia.org/wiki/DirectDraw https://en.wikipedia.org/wiki/DirectDraw seems to say DirectDraw was hardware-accelerated too. I guess the point is that there are and have been for a long time GPU accelerated 2D API's in DirectX land.
dahart 7y ago
> Well known to game developers, what is efficient on GPU is a structure-of-arrays approach. In particular, the tiling phase spends a lot of time looking at bounding boxes, to decide what belongs in each tile. [...] Now we get to the heart of the algorithm: going through an array of bounding boxes, looking for those that intersect a subset of tiles. Raph, you're describing ray tracing! I haven't thought a lot about this, but maybe the ray tracing hardware trend really could very much be utilized for 2D UI. With ray tracing support now announced all over the place, are you already thinking along those lines? The main difference between rasterizing and ray tracing is at it's most basic SoA vs AoS, i.e., whether the outer loop is over triangles or over pixels. And the pixel operation is going through an array of bounding boxes to see what overlaps. The somewhat amazing thing about the ray tracing hardware is you get in effect your entire search query through all the bounding boxes in a single "instruction". It's such a departure from all the predictable bite-sized hardware operations we're used to. It reminds me of stories from college of the DEC VAX and it's single instruction memcpy & strcmp. > While I mostly focused on parallel read access, I’m also intrigued by the possibility of generating the scene graph in parallel, which obviously means doing allocations in a multithread-friendly way. FWIW, this is definitely doable and already happening on the 3d GPU ray tracing side of the world...! Parallel scene graph construction, parallel BVH builds, etc.
- deleted 7y ago
  [deleted]
rock_artist 7y ago
> it seems quite viable to implement 2D rendering directly on GPU, with very promising quality and performance. I guess all agree that GPU rendering is the future. Yet, I'm still in favor for libraries that abstracts the actual renderer so it could fallback if needed. Especially when future and past conflicts. I come from audio development where people use plug-ins (dynamic libs) that loads on a host that provides native OS window you compose. Cross-platform is pretty vital so there are frameworks to target macOS, Windows, iOS (and sometimes Android & Linux). Here is a real life scenario that many audio developers are currently at: - you had a product with UI that needed to show 'real-time' visualization of signal - Analyzer (let's say FFT). - eventually you went the OpenGL way (because it was the right way in 2010...) you've written shaders. - it worked great on macOS and ok on most Windows machines (some old machines had limited or broken OpenGL within their drivers). - Apple deprecates OpenGL in favor of Metal... So now what was "modern" 9 years ago requires compete re-write. if there was an abstraction layer it might be minor changes and switching to new Metal renderer instead of the OpenGL one. In comparison, The native/cpu C++ code for the same product proven to be much more future proof and only required simple maintenance. It is indeed important work. but I guess targeting Vulkan is better especially when there are libs like MoltenVK. I guess the question is how much portability is important factor vs optimal performance.
- Const-me 7y ago
  > what was "modern" 9 years ago requires compete re-write. Have you thought about porting to GLES (shouldn't be that hard if you already have GL shaders), and using MoltenGL (apple) and ANGLE (windows)? AFAIK they both work reasonably well in practice. > I guess targeting Vulkan is better Vulkan support is improving but I’m not sure it’s ready yet. For Intel GPUs, only supported on Windows starting from Skylake (2015). > the question is how much portability is important factor vs optimal performance I think portable and fast cross-platform GUI is possible, just expensive to implement and especially support. Commercial game engines like Unity or Unreal optimize both at the same time, and they support low-spec platforms like mobile phones. When Microsoft wanted fast rich GUI portable across their platforms, they’ve built XAML. When Google wanted the same, they acquired Skia and built Chromium and now Flutter on top.
- 7y ago
malms 7y ago
"graphics" and then no video :(
- raphlinus 7y ago
  I know. I was eager to get this out. I'll be working on more visuals for a Libre Graphics Meeting presentation, where I'll include some of this in the Rust 2D graphics talk.
vbuwivbiu 7y ago
I don't get it. 2D is a subset of 3D which the GPU already does. Why the need for tiles and whatnot why not render 2D text and other bezier curves as 3D curves with a z coordinate of 0 projected in orthographic perspective ?
- pavlov 7y ago
  The short answer is that 3D is all about triangles, and that's not a good fit for the PostScript-based 2D rasterization model used by most APIs and tools (SVG, Illustrator, Apple CoreGraphics, Sketch...) See the answers to this question in the same thread: https://news.ycombinator.com/item?id=19864066 https://news.ycombinator.com/item?id=19864066

306 points by raphlinus 7y ago

I've been also working on a 2D UI on the GPU. It's quite amazing actually, compared with the code running on the CPU, the code is very terse. The setup can be somewhat verbose, but the shaders themselves are short. Like a 20 line shader can do miracles.

dman 7y ago
Any github links that are freely available?
irascible 7y ago
yeah shaders really force you to invert your thinking in a good way. Normally our brains spit out cluttered logic with a bunch of for loops and hash tables but shaders cleave all that away.
mpfundstein 7y ago
Do you have an example?

blogspam 7y ago

Honestly, HN is nothing but bike shedding by now. There's near zero hacker subculture to it anymore. Don't show HN anything. The ranking seems to be near arbitrary at this point, with nothing but media outlet vote brigades forcing their spyware.js ad-tech bloat trash megaphones to the front page. Fuck it, dude. Fuck this aggregator.

Waterluvian 7y ago

Something I've been really surprised and disappointed by is the lack of 2d vector graphics support in modern gaming engines. I'm an amateur enthusiast but I spent hours and hours unsuccessfully trying to find any decent engine where I could make games using svg or other vector formats.

paulryanrogers 7y ago
Godot has a skeletal animator built-in which is a nice middle ground for rotating sprites. I imagine the lack of dedicated vector art support could be that 3D engines can simulate that vectorised style.
pcwalton 7y ago
I would love to see someone step up and add Pathfinder support to Unity, Godot, and other popular engines. I'd be very interested in taking pull requests to this effect :)
jra101 7y ago
Defold might be an option. https://www.defold.com/ https://www.defold.com/
- speps 7y ago
  Doesn't support vector graphics, only SVG import into a texture if I recall.
DonHopkins 7y ago
I know what you mean! Unity3D has no way to simply draw a circle or pie chart into a texture. It would be great to be able to use the canvas api (and any JavaScript library like d3 that uses canvas) to dynamically draw Unity3D textures. To address that problem (and others), I'm been developing a Unity3D extension called UnityJS, which integrates a web browser and JavaScript with Unity3D, so you can use the canvas api to draw images for use as 2D user interface overlays and 3D mesh textures in Unity. And of course the other (main) point of UnityJS is to use JavaScript for scripting and debugging Unity apps, and integrate Unity apps with off-the-shelf JavaScript web libraries (d3, socket.io, whatever). https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy.txt https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy... Drawing on canvases actually works pretty well in the WebGL build, because the JavaScript runs in the same browser tab and address space that's running the Unity app, so you can efficiently blit the canvas's pixels right into Unity's texture memory. (Although I don't believe there's a way to share the canvas texture directly through GPU memory, or even directly share Unity's ArrayBuffer memory, but it's fast enough for interactive user interface stuff, and millions of times better than the "portable" technique of writing out a data uri with a base 64 encoded compressed png file, which is the "standard" way of getting pixels out of the web browser! Web tech sure sucks!) https://github.com/SimHacker/UnityJS/blob/master/Libraries/UnityJS/StreamingAssets/UnityJS/bridge.jss#L640 https://github.com/SimHacker/UnityJS/blob/master/Libraries/U...

Jasper_ 7y ago

Impressive work, Raph! As I'm sure everyone knows by now, the PostScript imaging model is hard to make work on the GPU, and its traditional geometry pipeline isn't suited well for curves or analytic coverage-based AA. Since I come from the games space, a lot of my thoughts have been about baking as much data as possible: if we had infinite build time, or artist tooling akin to Maya, how would that help us develop better UIs? If we leave PostScript behind and think about new graphics models, what can we do? So a lot of my own research has been studying really old graphics history, since there's always fun ways of looking at the problem before we settled on one solution. I'll have to collect some of my research and demos and publish it sometime soon. Although, one thing I've noticed during my time in games is that frame time is the standard. If we speed up our rasterizer, we'll just add more junk to the scene until it hits our 60fps frame target. More blurs! More effects! :)

raphlinus 7y ago
These are all interesting questions. I've been following some of the stuff on Shadertoy, and I think that points the way. Most is 3d, but not all, and in particular there's a lot to be done with distance fields. I've been thinking about blur (and mentioned some approximation techniques), but it's expensive in any context. Maybe if we think in shaders instead of the blur tool, we'll find a visual palette that is both fresh and efficient.
- DonHopkins 7y ago
  Distance fields are great! What do you think an alternative to PostScript's imaging model (Porter/Duff) formulated in terms of signed distance fields would be like? Much of Photoshop's power comes from representing the selection as first class alpha channels that you can use all of Photoshop's editing and filtering tools on, so you can select something with any of the the selection tools, then take a step back into "quick mask" mode and edit your selection in the "channel" domain, with any of the painting tools or filters, and store selections in extra channels for later use. I wonder how an image editor like Photoshop (and a rendering model in general) could directly and conveniently support signed distance fields as shapes, selections, and channels, as nicely as it supports alpha channels? Back in 1993 (5 years after Photoshop was created in 1988), it sounded revolutionary to use texture mapping as a fundamental drawing primitive, because nobody suspected some day everybody'd be carrying around more computrons than an SGI workstation in their phone. http://www.graficaobscura.com/texmap/index.html http://www.graficaobscura.com/texmap/index.html

amelius 7y ago

> I’m particularly interested in the rendering quality. Shouldn't the rendering be done using oversampling? E.g. instead of 100x100, you render on 400x400, then scale back down to 100x100 to get the best result. One of the problems is that anti-aliasing creates pixels which are half-on, e.g. at the edge of a line or polygon. Now if multiple edges meet near a certain pixel, then rendering the polygon(s) will touch a "half-on" pixel several times, and as a result it may receive the wrong opacity.

raphlinus 7y ago
It is true that is more correct (see https://github.com/pcwalton/pathfinder/issues/142 https://github.com/pcwalton/pathfinder/issues/142). But my motivation is much more user interface than document rendering, so it's quite reasonable to tell the UI designers, "don't do that," especially if the flip side is butter-smooth performance.
- amelius 7y ago
  But UI designers are not always in complete control, e.g. if screen resolutions are different for different users.
  - pcwalton 7y ago
    In practice, if you avoid coincident edges, you're fine. Font rasterizers have been using analytic AA for a long time without problems. In theory, the lack of knowledge of render resolution makes that a concern, but in practice it's not too much of an issue.
  - raphlinus 7y ago
    Agreed, and there are some deep issues, for example the rendering of very thin strokes which is quite dependent on resolution. Also keep in mind that the 4x4 oversampling you mentioned does not give nice smooth edges. I'm working on font editing, so the quality and smoothness of the line, to most accurately represent the shape without artifacts, is paramount.
7y ago

pavlov 7y ago

What an interesting rabbit hole. Thanks for sharing your notes too! Re: > "As a digression, I find it amusing that the word for packing a data structure into a byte buffer is “serialization” even when it’s designed to be accessed in parallel. Maybe we should come up with a better term, as “parallel-friendly serialization” is an oxymoron." NeXTStep/Cocoa calls this simply "encoding" which always made sense to me.

kllrnohj 7y ago

> and many people believe that damage regions are obsolete Old-style damage regions are obsolete. But new-style ones leveraging buffer age & swap-with-damage are not. Vulkan changes this, but these older EGL extensions are good introductions to the concepts: https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_buffer_age.txt https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_... https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_partial_update.txt https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_... and https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_swap_buffers_with_damage.txt https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_... > Flutter is a good modern approach to this, and its “layers” are one of the keys to its performance. Flutter actually does it badly. Layers are terrible for performance because they are a source of jitter (and memory usage), and jitter often results in jank. They are necessary in some cases, like for correct alpha blending, but in general should be avoided if your goal is to be smooth & fast. You want to optimize for your slowest frames, not your fastest ones. Flutter's excessive layering can also rapidly be a net-loss in performance due to the overhead of render target switching, increase in memory bandwidth required, and fewer opportunities for overdraw avoidance & disabled blending.

raphlinus 7y ago
Agree on both fronts! (Also, your username seems familiar somehow - have we had discussions around this before? :) I didn't talk about this in the post because I haven't actually implemented fancy compositing, but based on what I've seen you never want to render to texture (which Flutter does sometimes) as it uses up scarce global GPU memory bandwidth, and if you're doing that using heuristics, then performance gets very unpredictable. I should express this more clearly, but the goal in piet-metal is not to do the fastest renderer (I'm sure it isn't), but the one with the most consistent and predictable performance.
- kllrnohj 7y ago
  > but based on what I've seen you never want to render to texture (which Flutter does sometimes) Flutter's "layers" are rasterized to texture if they are unchanged for N frames (I think N = 3?). Which happens kind of a lot, and is (one of) the reasons Flutter has such a big RAM footprint. Flutter's layers remind me a lot of the hacky, terrible will-change CSS property. Other than that it's just a rather crude way to hack up the rendering commands into smaller display lists. Android has been doing this for ages with RenderNode with both display lists & transform properties on those display lists.

Retr0spectrum 7y ago

Is there a demo/source anywhere? I admit I only skimmed the article, for now.

deleted 7y ago
[deleted]
raphlinus 7y ago
Yes, https://github.com/linebender/piet-metal https://github.com/linebender/piet-metal , it's mac-only (for now).

tenaciousDaniel 7y ago

As someone who knows literally nothing about the GPU, the notion that the GPU might not be good for UI graphics seems...odd? Like, it can handle 3D graphics right? Aren't those waaaay more computationally expensive than 2D graphics? I know that I'm missing something important, but I don't know what it is.

raphlinus 7y ago
It's complicated. 2D graphics aren't inherently harder or require more computation than 3D, but they require very different optimizations. I think it boils down to the fact that 2D graphics is ultimately data structure heavy, whereas 3D graphics can often be represented as a huge array of triangles. So there's a lot of literature (some of which I linked, then those papers have a deep literature review), but no one obviously best solution. One thing I think I contributed is a relatively simple solution, especially for doing almost all the work on GPU rather than relying on CPU for things.
- incompatible 7y ago
  Is there any particular reason why recent / future GPUs would be worse at 2D graphics than older versions? I.e., what needs to change, when current methods for 2D graphics have worked for years? Also not knowing much about the field, GPUs seem to have a lot of unpleasantness in the form of incompatible and/or proprietary APIs, so implementing anything that's supposed to be cross platform (also to old and new GPUs, and future GPUs as they are released) may be hard work.
  - raphlinus 7y ago
    What makes you say that 2D graphics has worked well? Scrolling often janks, UI designers constantly have to work around limitations in the imaging model and performance quirks (for a long time, clipping to a rounded rect would kill performance), and there were lots of shortcuts with things like gamma for blending. I'm hoping to build something way better, where the UI always updates smoothly, and the UI designer doesn't have to be conscious of the limitations in the renderer.

dragontamer 7y ago

> It’s often said that GPU is bad at data structures, but I’d turn that around. Most, but not all, data structures are bad at GPU. An extreme example is a linked list, which is still considered reasonable on CPU, and is the backbone of many popular data structures. Not only does it force sequential access, but it also doesn’t hide the latency of global memory access, which can be as high as 1029 cycles on a modern GPU such as Volta. You're almost correct here, from my understanding. But... I still feel like its important to note that Pointer-jumping exists, and linked-list like structures (linked lists, trees, and graphs) can be traversed in parallel in many cases. https://en.wikipedia.org/wiki/Pointer_jumping https://en.wikipedia.org/wiki/Pointer_jumping For pointer-jumping in GPUs / SIMD systems, I know that "Data Parallel Algorithms" discusses the technique as applied to a "reduce" operation aka a "scan": https://dl.acm.org/citation.cfm?id=7903 https://dl.acm.org/citation.cfm?id=7903 The requirement (not fully discussed in the 'Data Parallel Algorithms' article) is that you still need an array of nodes to be listed off by the SIMD system. But this array of nodes does NOT need to be sorted. As such, you can accomplish a "GPU Reduce" over a linked list in O(lg(n)) steps (assuming an infinite-core machine). This is way slower than the O(lg(n)) array traversal, but you're at least within the same asymptotic class.

DonHopkins 7y ago

>Other than that, the serialization format is not that exotic, broadly similar to FlatBuffers or Cap’n Proto. As a digression, I find it amusing that the word for packing a data structure into a byte buffer is “serialization” even when it’s designed to be accessed in parallel. Maybe we should come up with a better term, as “parallel-friendly serialization” is an oxymoron. Great observation! I get the feeling there must be a great punny term for the enigmatic oxymoron "parallel-friendly serialization", but I can't quite put my finger on it. (Reminds me of "de-optimizer" => "pessimizer", or "spectral frequency analysis and filtering" => "cepstral quefrency alanysis and liftering".) https://en.wikipedia.org/wiki/Cepstrum https://en.wikipedia.org/wiki/Cepstrum

strmpnk 7y ago
Marhsalling has a precedent in some languages. It makes a lot of sense. "serde[s]" always reminds me of sequencing of transmission more than a format concern of what goes where in a buffer.

iamleppert 7y ago

Very interesting work. I’m wondering if you’ve ever worked with vector tiles? Mostly it’s used for drawing large numbers of vectors on maps, but has a similar architecture and approach for 2D drawing on the GPU via an efficient command-based language based on the extent of the tile.

aasasd 7y ago

> Performant UI must use GPU effectively, and it’s increasingly common to write UI directly in terms of GPU rendering, without a 2D graphics API as in the intermediate layer. Y tho? I'm just a passerby in this topic, but I see this sentiment now and again these days, and I don't understand what happened. I don't remember hearing complaints of DirectDraw being too slow, for example.

Jasper_ 7y ago
DirectDraw wasn't a 2D graphics package; you probably mean GDI/GDI+. These libraries were used at at much smaller resolutions (1024x768, but fullscreen was pushing it, your window was smaller). GDI could roughly be used at 60Hz but it didn't support antialiasing or transparency, you needed GDI+ for that. And GDI+ was much slower, even at the smaller resolutions, so you tended to draw into a big bitmap ahead of time and scroll that. Animations in GDI+ were unheard of.
- roel_v 7y ago
  I'm not sure what you mean by '2D graphics package', so it may be a semantics thing here, but DirectDraw was a 2D graphics API on top of DirectX. I only used it a bit and didn't find that for my uses, the added complexity was worth it over just using GDI, but DirectDraw did do GPU accelerated 2D rendering. How fast it was in light of today's requirements and hardware, I don't know.
  - Jasper_ 7y ago
    Are you confusing it with Direct2D? DirectDraw was an old API from the D3D5-ish era and basically allocated a front buffer you could draw to with GDI. It was not GPU accelerated.
    - roel_v 7y ago
      Well you got me wondering now on what I worked with, but https://en.wikipedia.org/wiki/DirectDraw https://en.wikipedia.org/wiki/DirectDraw seems to say DirectDraw was hardware-accelerated too. I guess the point is that there are and have been for a long time GPU accelerated 2D API's in DirectX land.

dahart 7y ago

> Well known to game developers, what is efficient on GPU is a structure-of-arrays approach. In particular, the tiling phase spends a lot of time looking at bounding boxes, to decide what belongs in each tile. [...] Now we get to the heart of the algorithm: going through an array of bounding boxes, looking for those that intersect a subset of tiles. Raph, you're describing ray tracing! I haven't thought a lot about this, but maybe the ray tracing hardware trend really could very much be utilized for 2D UI. With ray tracing support now announced all over the place, are you already thinking along those lines? The main difference between rasterizing and ray tracing is at it's most basic SoA vs AoS, i.e., whether the outer loop is over triangles or over pixels. And the pixel operation is going through an array of bounding boxes to see what overlaps. The somewhat amazing thing about the ray tracing hardware is you get in effect your entire search query through all the bounding boxes in a single "instruction". It's such a departure from all the predictable bite-sized hardware operations we're used to. It reminds me of stories from college of the DEC VAX and it's single instruction memcpy & strcmp. > While I mostly focused on parallel read access, I’m also intrigued by the possibility of generating the scene graph in parallel, which obviously means doing allocations in a multithread-friendly way. FWIW, this is definitely doable and already happening on the 3d GPU ray tracing side of the world...! Parallel scene graph construction, parallel BVH builds, etc.

deleted 7y ago
[deleted]

rock_artist 7y ago

> it seems quite viable to implement 2D rendering directly on GPU, with very promising quality and performance. I guess all agree that GPU rendering is the future. Yet, I'm still in favor for libraries that abstracts the actual renderer so it could fallback if needed. Especially when future and past conflicts. I come from audio development where people use plug-ins (dynamic libs) that loads on a host that provides native OS window you compose. Cross-platform is pretty vital so there are frameworks to target macOS, Windows, iOS (and sometimes Android & Linux). Here is a real life scenario that many audio developers are currently at: - you had a product with UI that needed to show 'real-time' visualization of signal - Analyzer (let's say FFT). - eventually you went the OpenGL way (because it was the right way in 2010...) you've written shaders. - it worked great on macOS and ok on most Windows machines (some old machines had limited or broken OpenGL within their drivers). - Apple deprecates OpenGL in favor of Metal... So now what was "modern" 9 years ago requires compete re-write. if there was an abstraction layer it might be minor changes and switching to new Metal renderer instead of the OpenGL one. In comparison, The native/cpu C++ code for the same product proven to be much more future proof and only required simple maintenance. It is indeed important work. but I guess targeting Vulkan is better especially when there are libs like MoltenVK. I guess the question is how much portability is important factor vs optimal performance.

Const-me 7y ago
> what was "modern" 9 years ago requires compete re-write. Have you thought about porting to GLES (shouldn't be that hard if you already have GL shaders), and using MoltenGL (apple) and ANGLE (windows)? AFAIK they both work reasonably well in practice. > I guess targeting Vulkan is better Vulkan support is improving but I’m not sure it’s ready yet. For Intel GPUs, only supported on Windows starting from Skylake (2015). > the question is how much portability is important factor vs optimal performance I think portable and fast cross-platform GUI is possible, just expensive to implement and especially support. Commercial game engines like Unity or Unreal optimize both at the same time, and they support low-spec platforms like mobile phones. When Microsoft wanted fast rich GUI portable across their platforms, they’ve built XAML. When Google wanted the same, they acquired Skia and built Chromium and now Flutter on top.
7y ago

malms 7y ago

"graphics" and then no video :(

raphlinus 7y ago
I know. I was eager to get this out. I'll be working on more visuals for a Libre Graphics Meeting presentation, where I'll include some of this in the Rust 2D graphics talk.

vbuwivbiu 7y ago

I don't get it. 2D is a subset of 3D which the GPU already does. Why the need for tiles and whatnot why not render 2D text and other bezier curves as 3D curves with a z coordinate of 0 projected in orthographic perspective ?

pavlov 7y ago
The short answer is that 3D is all about triangles, and that's not a good fit for the PostScript-based 2D rasterization model used by most APIs and tools (SVG, Illustrator, Apple CoreGraphics, Sketch...) See the answers to this question in the same thread: https://news.ycombinator.com/item?id=19864066 https://news.ycombinator.com/item?id=19864066