10 ms·

Fast 2D Rendering on GPU

286 points by raphlinus 6y ago

raphlinus 6y ago
Fast 2D rendering on GPU has represented the last few months of concentrated work for me, and I'm happy to present the results now. It's required going pretty deep into various aspects of GPU compute, so feel free to ask me about that, 2D vector graphics, or anything related.
- tenaciousDaniel 6y ago
  Hi! This looks interesting, although I confess I'm very new to this area. I'm writing a programming language specifically for UI designers, and currently I'm building a naive implementation in html5 canvas. I know very little about low-level rendering - just really know how to use drawing API's like canvas and Quartz 2D. With that being said, I'm looking to delve deeper into this subject. Do you have any recommendations on where to start? Whenever I look beyond simple drawing API's, the focus seems to be entirely on 3D rendering (which is interesting but not my main focus right now).
  - raphlinus 6y ago
    It's a good question, as honestly the knowledge for 2D graphics is pretty arcane, as opposed to 3D being so widely taught. I actually started a github repo for a book on 2d graphics but have no idea whether I'll actually finish it. In the meantime, antigrain.com is one good (if old) source. The original PostScript "red book" was extremely influential in its time (it's where I learned a lot of this stuff) but is quite dated now. Best of luck, and I'm also happy to field requests for more specific areas. For example, for color theory (an important aspect of 2D graphics!), handprint.com is quite a remarkable resource.
    - tenaciousDaniel 6y ago
      Thanks!
    - blondin 6y ago
      > knowledge for 2D graphics is pretty arcane such an unfortunate state of affairs! i am currently learning how to render graphics using the GPU on my mac using apple metal. what i am getting is that the GPU has been optimized for 3D rendering?! GPUs make no provision or easy way for rendering 2D graphics? it makes no sense to me... that's where you start...
      - dungdang 6y ago
        now this may be a dimb question, but why would you start there? as far as i understand, 2d and 3d have literally zero to do with each other in how they are rendered. one is a bunch of triangles. the other is lines, curves, thickness, gradients, and fonts (which are essentially little programs)
        blondin 6y ago
        that's not a dumb question! it's my own fault. i only have knowledge of 2D graphics.
        chrisseaton 6y ago
        > the other is lines, curves, thickness, gradients, and fonts You can reduce all these to drawing triangles.
        deleted 6y ago
        [deleted]
        Jasper_ 6y ago
        Doing that reduction is surprisingly difficult, and usually a serial algorithm that runs on the CPU, and a naive approach is a resulting triangle set is not efficient to a GPU, but that's what toolkits like cairo, Direct2D, nanovg do. Raph is describing an architecture where path evaluation happens on the GPU, without being baked to triangles.
        raphlinus 6y ago
        > You can reduce all these to drawing triangles. You can, people have tried this, and it sucks. The main problem is that the conversion of Bézier paths to triangles is a hard problem with lots of conditional branching. Even when you do it, there is the other problem of rendering triangles with really good antialiasing, MSAA forces a compromise between performance and quality. By contrast, piet-gpu does an exact-area calculation for antialiasing. So it's not a question of whether you can do it, but whether it works well, and approaches like piet-gpu absolutely stomp triangles.
    - smallstepforman 6y ago
      Haiku OS AppServer (the screen rendering component, similar to Unix X11) is a full GUI system implemented with AntiGrain Geometry as the renderer.
    - gct 6y ago
      I think that blend2d (https://blend2d.com/ https://blend2d.com/) is a worthy successor to AGG, and it's under active development.
      - longstation 6y ago
        Thank you for the link. It was a great reading. But I just want to point out that Blender2D is a software renderer.
        gct 6y ago
        So is/was AGG
  - syspec 6y ago
    I recommend the blog of the OP, so many gems
- awfulaxolotl 6y ago
  How well does this approach work for something like 2D data visualization where most of the visual elements are the same -- i.e. can be instanced in OpenGL/etc? Thanks for publishing this, it's awesome work! I'm looking forward to progression to wgpu hinted in the Github README.
  - raphlinus 6y ago
    I absolutely have data visualization in mind for this, as I think it can benefit greatly from the scale. But the pipeline I've built is very agile, it will easily handle a diverse mix of items. It's not like OpenGL etc where there's a certain amount of overhead for a draw call so there are significant gains to be had from instancing and batching. It is likely that CPU-side encoding can be made more efficient, though, by just filling in quantities to a template, rather than encoding from scratch.
- leeoniya 6y ago
  any relation to the work currently being done here by pcwalton? https://github.com/servo/pathfinder/pull/350 https://github.com/servo/pathfinder/pull/350 hopefully you guys aren't doing identical work in parallel (no pun intended) :)
  - pcwalton 6y ago
    Yes, there's been a lot of influence in both directions. The approaches have a lot in common, but also have significant differences. I plan to write up a description of the Pathfinder compute work soonish.
    - paufernandez 6y ago
      I would love that!
  - raphlinus 6y ago
    As Patrick indicated, there's a lot of cross-fertilization of ideas, and one of the best outcomes of this work would be for high performance compute rendering to ship in Pathfinder.
- RivieraKid 6y ago
  - Shouldn't 2D rendering be a solved problem given that it's basically a subset of 3D rendering? - Don't libraries like Skia, Qt, Cairo use GPU rendering? I've always assumed so. I mean, this is 2020, GPUs have been around for decades.
  - pcwalton 6y ago
    > - Shouldn't 2D rendering be a solved problem given that it's basically a subset of 3D rendering? The problem is that primitives artists use are different. 3D rendering tends to all consist of polygon meshes, which are relatively easy to render. 2D rendering (basically) consists of Bezier paths, which are harder. The equivalent in 3D, which is adaptive subdivision, is not really a solved problem in real-time either. Additionally, 2D rendering quality tends to be more important than 3D rendering quality. Whereas you can get away with 4xMSAA or hacks like FXAA in 3D, true 16xAA (without hacks) is the absolute minimum for 2D rendering quality nowadays, and even it isn't considered great for some tasks like font rendering (Pathfinder and piet-gpu both use analytic AA which is effectively 256xAA). > - Don't libraries like Skia, Qt, Cairo use GPU rendering? I've always assumed so. I mean, this is 2020, GPUs have been around for decades. There's a difference between renderers with GPU support and renderers that are oriented around using the GPU efficiently. In many cases this results in an order-of-magnitude speedup. On the GPU, state changes are expensive, and many such renderers that have GPU support don't really go out of their way to avoid them. There are also occlusion culling optimizations that most renderers don't do, but piet-gpu and Pathfinder do.
  - stanfordkid 6y ago
    Things like multi-resolution font-rendering are actually more complicated than one might imagine. The easy way is to just tesselate the font into polygons -- but this tesselation often depends on the zoom level. The same thing can be said about implicit curves etc. Most libraries do use GPU's for basic draw operations (e.g rendering a gradient), but to build something like Photoshop -- you need much more complexity.
  - raphlinus 6y ago
    Others have spoken to this, but as a general introduction I highly recommend Jasper's post "Why are 2D vector graphics so much harder than 3D?" In short, no, it's not a solved problem. Also, I see a lot of variations of this question, but I should state this more clearly. There's been accelerated graphics in one form or another for a long time, but what I'm doing is a completely different type of thing. In my world, on the CPU you just encode the scene into a binary representation that's optimized for GPU but in many ways is like flatbuffers, and then the GPU runs a highly parallel program to render the whole thing. In previous approaches, the CPU is deeply involved in taking the scene apart and putting it back together in a form that's well suited to relatively dumb pixel pipes. Now that GPUs are really fast, that approach runs into limitations. It also depends what you're trying to do. I'm focusing here on dynamic paths (and thus font rendering), while most of the libraries optimized for UI put text into texture atlases and then use the GPU to composite quads to the final surface, something they can do well. https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-harder-than-3d-graphics/ https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-harde...
- moonchild 6y ago
  It looks like the main fruit of this is piet-gpu[1]. How close would you say it is to stable? And how complete a realisation of the concepts put forth in the OP? I'm currently working on a ui library, and evaluating alternatives for hw acceleration. The whole library wants to be in c, so I would want to rewrite the library by release-time to use it, but don't want to waste time doing that now if it's still really volatile. 1. https://github.com/linebender/piet-gpu https://github.com/linebender/piet-gpu
  - raphlinus 6y ago
    Not very, and not very. This is research, it depends on compute capabilities that have really only gone mainstream in GPU lately, and the current codebase is about trying out the ideas. Best of luck in your project!
- MrBuddyCasino 6y ago
  Wow, those numbers look very impressive on an absolute basis! Do you have an idea on how piet-gpu compares to Skia? I'm not a graphics guy, my impression was that it is currently the dominant vector graphics renderer.
- structural 6y ago
  What's been your overall experience with using Vulkan shaders for compute? Are there basic primitives that are missing from shading languages and/or have you found any impedence mismatches between writing shaders vs. how you might describe the same algorithms in other languages?
  - NotCamelCase 6y ago
    I pondered on the same subject recently as I was implementing the same algorithm (i.e. Mandelbrot set) on the CPU (scalar vs SIMD) and GPU compute using fixed-point and floating-point for comparisons (if interested: https://tayfunkayhan.wordpress.com/2020/06/03/mandelbrot-in-floating-point-vs-fixed-point https://tayfunkayhan.wordpress.com/2020/06/03/mandelbrot-in-...). It bothers me how little progress has been done on "shading" languages front compared to overall many-core computation models and capabilities over the years. And, that is despite the fact that shaders are very often where the most time is spent in modern workloads. Compute with Vulkan is another story. It offers some nice abstractions, but it shows that it's mostly intended for async-compute/work-offloading for rendering, IMO. Too much fruction.
  - raphlinus 6y ago
    That's a big topic. I've been able to work around the missing primitives (for example, I autogenerate code for Rust-style structs and enums), but have had much bigger struggles around two issues: tools, which are still quite primitive, and understanding performance, which is extremely difficult. These two problems intersect because I can imagine a lot better tools for digging into performance issues. One that I would have paid good money for is an instruction-level simulator that would highlight the source code to tell me where the stalls, bank conflicts, divergence problems, etc. are in the source code. Such a thing is possible (there are academic papers like [1]), but not as far as I know usable in daily development. The "impedance mismatch" is that you (generally) have to write in a style to extract lots of parallelism. This tends to be very different than the way you'd write scalar CPU code, but not completely alien to me as it has a lot of similarity with the way you'd write SIMD. I've pretty much gotten the hang of it now. I'm thinking of a blog post of redoing path_coarse.comp from its current basically scalar style to a more parallel version, as that would I think illuminate the issues. [1] http://comparch.gatech.edu/hparch/papers/gera_ispass18.pdf http://comparch.gatech.edu/hparch/papers/gera_ispass18.pdf
- deleted 6y ago
  [deleted]
chadcmulligan 6y ago
This is very impressive. Have you benchmarked it against core graphics on a Mac? I believe they've done a similar thing - performing the render on Metal
- pcwalton 6y ago
  Core Graphics is generally slower than Cairo/Pixman.
- c-smile 6y ago
  CoreGraphics is pure CPU rasterizer, similar to GDI+. Sciter (https://sciter.com https://sciter.com) on MacOS uses Skia/OpenGL by default with fallback to CoreGraphics. It is possible to configure Sciter to use CoreGraphics on MacOS to compare these two on the same UI by using SciterSetOption(NULL, SCITER_SET_GFX_LAYER, GFX_LAYER_CG); I think it is safe to say that Skia/OpenGL is 5-10 times more performant than CG on typical UI tasks.
  - chadcmulligan 6y ago
    Interesting, thanks - I remember hearing at one of the WWDC that Coregraphics got a 10x speed improvement using metal. I just read the fine print - it seems draw calls only have a 10x speed improvement, I assume because they render through a layer of some sort. The CA* libraries may use metal - animation and layers, which I guess is where the 10x draw call improvement comes in maybe. Some discussion here https://arstechnica.com/civis/viewtopic.php?t=1285571 https://arstechnica.com/civis/viewtopic.php?t=1285571 , though a lot of guessing.
    - pcwalton 6y ago
      And before that, I'm pretty sure Core Graphics used OpenGL in some situations. I remember seeing CG::OGL in stack traces for WebKit.
    - c-smile 6y ago
      I believe that the most HW optimizations are made in MacOS on DWM level - window composition and animation. Window surface is just a bitmap in RAM that needs to be populated by CoreGraphics/CPU. In any case Acrylic/Vibrancy effects or "blur-behind background" (like on screenshots here: https://sciter.com/sciter-4-2-support-of-acrylic-theming/ https://sciter.com/sciter-4-2-support-of-acrylic-theming/) are achievable only on GPU and on DWM level.
c-smile 6y ago
Question to the author: is this only implementation of GPU rasterizer? What about anything like WARP (https://en.wikipedia.org/wiki/Windows_Advanced_Rasterization_Platform https://en.wikipedia.org/wiki/Windows_Advanced_Rasterization...) - fallback rendering when GPU is not available ?
- raphlinus 6y ago
  It's something I'd like to explore at some point, also Swiftshader, which might be easier to explore, as it's already Vulkan. I expect performance to be pretty good, but there are already really advanced CPU renderers such as Blend2D. Doing serious performance evaluation is hard work, so I actually hope it's something others take up, as I have pretty limited time for it myself.
  - c-smile 6y ago
    I understand. Problem is that any practical 2D rendering solution shall support as GPU as CPU rendering unfortunately. It would be interesting to see any GPU equivalent of something like AGG by Max Shemanarev, RIP.
    - throwaway9087 6y ago
      Practical 2D renderers implement an abstraction layer that lets them easily redirect their output to a number of low level libraries, which can be either CPU or GPU-based (or a mix). I worked on a few such 2D stacks, including OpenGL, DirectX, WebGL, and AGG based, and it took no more than a few days to add new 2D backends to an existing pipeline. Most 2D rendering is based on concepts from PostScript so it's usually easy to do such ports -- except for AGG, that one was a bit like a library from outer space. Maxim himself worked on a hardware accelerated 2D renderer for Scaleform, and it looked nothing like AGG, mostly because AGG is practically impossible to move to a GPU implementation.
      - c-smile 6y ago
        I know, AGG is just a set of primitives but not an abstraction like class Graphics {...}. But that one can be assembled from them. Did it once for early versions of Sciter. Ideally GPU and CPU rendering backends should have pixel perfect match that makes "adding new 2D backend" tricky at best.
aasasd 6y ago
Noob question. Isn't some GPU acceleration for 2D used since forever? I keep being perplexed as to why browsers still have problems with it (at least on my older Macbook with a shitty Intel's embedded GPU). I thought some stuff, like scrolling, was offloaded at least in early/mid-2000s, making for a distinctly meh experience when proper drivers weren't installed.
- thechao 6y ago
  GPUs derive most of their parallelism from emitting rows of related “quads” (the smallest unit of the screen that supports a difference operator). 2D graphics are “chatty” in window sizes that are more like 5–10 units in diameter. It’s hell on HW perf. To make things worse, 2D applications usually want submillisecond latency. A GPU driver/hw stack will struggle to get latency below 1–2ms. When there’s lots of multipass rendering (which is a thing 2D also wants a lot of), latency can to 10+ ms.
  - raphlinus 6y ago
    Well said, thanks. It's also a goal of this work to do compositing inside the "fine rasterizer" compute kernel, to avoid multipass as much as possible. Of course, in the general case for stuff like blur that's hard, but that's one reason why I've been working on special cases like blurred rounded rectangles, which I can render pretty much at lightspeed. (It's not integrated into the current code but would be quite easy)
  - rasz 6y ago
    What does latency have to do with 2D? Are you suggesting UI toolkits are writing GPU commands synchronously? You should fill display list, fire it and forget about GPU until the next update.
    - thechao 6y ago
      Pen drawing for UIs. The ideal case is to update a small portion of the screen at about 240hz, to provide a good simulation of pen/pencil feedback. Really, your latency envelope should be on the order of the propagation of sound through the barrel of the marking device, but screens don’t update that fast.
deleted 6y ago
[deleted]
amelius 6y ago
Is GPU here synonymous with Nvidia?
- adamnemecek 6y ago
  It uses metal and apple uses amd do no.
- raphlinus 6y ago
  No, it's designed to be portable to all GPU hardware that can support compute, which these days is a pretty good chunk of the fleet. I tested it on Linux on Intel HD 4000, and the master branch seems to run just fine, though previous versions. A lot of the academic literature (Massively Parallel Vector Graphics, the Li et al scanline work) is dependent on CUDA, but that's just because tools for doing compute on general purpose graphics APIs are so primitive. I have a talk and a bunch of blog posts on exactly this topic, as I had to explore deeply to figure it out. See https://news.ycombinator.com/item?id=22880502 https://news.ycombinator.com/item?id=22880502 and https://raphlinus.github.io/gpu/2020/04/30/prefix-sum.html https://raphlinus.github.io/gpu/2020/04/30/prefix-sum.html for more breadcrumbs on that.
stephc_int13 6y ago
Something I am not sure to completely understand, after reading the article, is: in which way is it better/ more desirable than the classical approach of GPU rendering? (with vertex, triangles, and shaders)
- pcwalton 6y ago
  The basic problem is that, without compute, you either have to encode the vector scene into GPU primitives (triangles) as quickly as possible, and all the ways I know of to do that either involve (a) an expensive CPU process or (b) a cheap CPU process but way too much overdraw on GPU side. Compute gives you the best of both worlds, allowing you to upload outlines directly to GPU and do the processing necessary to lower them to primitives on the GPU itself. Pathfinder in there is actually using regular old GPU rasterization with triangles and so forth, and I'm fairly confident it's about as fast as you can go at D3D10 level (i.e. no compute shaders) without sacrificing quality. Note that the numbers can vary wildly depending on hardware. On my MacBook Pro, with a powerful CPU and limiting myself to the Intel integrated GPU only, Pathfinder is actually about equal to the GPU compute approach on a lot of scenes like the tiger, though it uses a lot of CPU.
  - stephc_int13 6y ago
    Is it true for all 2D rendering? From my intuition, this seems pretty specialized for vector-like rendering, with a lot of small bezier shapes.
    - pcwalton 6y ago
      Yeah, when I say 2D I mean vector art. There are a lot of things under the heading of 2D rendering, such as blitting raster sprites, that are much closer to being solved problems. (Though you might be surprised--power concerns, coupled with greatly increased pixel density, have brought renewed attention to performance of blitting lately...)
  - Const-me 6y ago
    > all the ways I know of to do that either involve (a) an expensive CPU process or (b) a cheap CPU process but way too much overdraw on GPU side That's fixable, take a look: https://github.com/Const-me/Vrmac#vector-graphics-engine https://github.com/Const-me/Vrmac#vector-graphics-engine My CPU process is moderately expensive, and I use quite a few tricks to reduce overdraw, e.g. both draw calls (the complete vector image, however complex, takes 2 draw calls to render) use hardware depth buffer and early Z rejection.
The_rationalist 6y ago
How would it compare with https://github.com/intel/fastuidraw https://github.com/intel/fastuidraw And https://github.com/blend2d/blend2d https://github.com/blend2d/blend2d And the skia Vulkan backend?
Abishek_Muthian 6y ago
iTerm's usage of metal on macOS is good example of benefits in implementing 2D rendering on GPU[1]. [1]https://gitlab.com/gnachman/iterm2/-/wikis/Metal-Renderer https://gitlab.com/gnachman/iterm2/-/wikis/Metal-Renderer
- bori5 6y ago
  That links to iTerm2 which is what I’m guessing you meant to say ? Kitty is also a GPU accelerated terminal emulator and one that I enjoy using https://sw.kovidgoyal.net/kitty/ https://sw.kovidgoyal.net/kitty/ Not sure if it uses Metal on Mac though.
The_rationalist 6y ago
The benchmarcks do not compare to skia (Vulkan) so it is unclear if the state of the art has been beaten on a specific example.
chrismorgan 6y ago
I would like to get an idea of the capabilities of these things. Suppose I was to make an ebook reader designed to be modelled more closely after paper, on a device like the Surface Book’s 13″ 3000×2000 display, showing two pages side-by-side. Each page might contain something like 2,000–2,500 letters. I want to be able to flip through pages like I might with a paper book, so that I might be roughly completely rendering several pages at once, and perhaps parts of several more pages; ideally it might render the page like a real 3D page, but if that’s too troublesome I’d settle for an affine transformation while flipping. Assume that the layout of the pages, with all the shaping, is all done ahead of time and is in memory. I’ve never seen anyone attempt anything like this before. In the old way of doing things, I think anyone attempting this would render each page to a bitmap and use that as a GPU texture, and I think that could provide acceptable performance (unless you were flipping rapidly through hundreds of pages, because that’d take a lot of GPU memory to keep), but I imagine that the quality of the rendering mid-turn would be fairly atrocious—it could be the sort of thing where the appearance of the text subtly changes half a second after you finish turning the page, as it switches from one renderer to a slightly different one. Would the performance of this new approach be sufficient to render what I describe at 60fps, while rendering each frame perfectly? I was talking about the Surface Book; its Intel Core i7-6600U has Intel HD Graphics 520 as its GPU; probably not too far off your 630’s results. (And I’m interested in what integrated graphics can do, more than a dedicated GPU.) My guess based upon your paper-1 results is that this is probably just barely possible with integrated graphics, so long as you employ a few tricks to reduce the amount of work required.
- raphlinus 6y ago
  Short answer, yes, I believe it could work, though not with much margin to spare. It might require some dedicated optimization work to fit the frame budget, especially when you know things about the scene (it's mostly small glyphs and with basically no overdraw) as opposed to being completely general purpose. Doing a warp transformation in the element processing kernel would probably work just fine, and give you realistic movement and razor-sharp rendering. I'd love to see such a thing and would very much like to encourage people to build it based on my results :)
eco 6y ago
How does this (and Pathfinder for that matter) compare to NanoVG? I've recently been experimenting with swapping out Cairo for NanoVG and it seems much faster. The lack of dashed lines may kill my experiment though unless I can think of a decent workaround.
- raphlinus 6y ago
  I expect it to be faster, but doing performance evaluation is really hard work. Performance depends on so many variables, including the workloads you're trying to render and a ton of factors about the GPU and system. I know people are really curious about competitive performance against other renderers, and it's something I should probably do, but honestly I'd love it if somebody else took up that mantle.
  - eco 6y ago
    I kind of meant more in terms of how they work and what they can do but I suppose I could do the work in answering that question myself by figuring out how nanovg works then comparing it to what you've written. I've been finding Patrick's pathfinder work fascinating so it's nice to have yet another project to follow.
    - raphlinus 6y ago
      Got it. So far my research prototype really just does filled paths and limited strokes, but the architecture should extend to a richer set of graphics primitives. Pathfinder also started out that way but recent work is building out a good chunk of the SVG imaging model. I definitely recommend checking it out, there's a good chance it'll do what you need.
- pcwalton 6y ago
  I have to say I'm consistently impressed by NanoVG. Not because it does anything fancy, but because it doesn't. It's not the fastest or the most featureful vector renderer out there, but it resides in a very nice sweet spot that balances performance, features, and simplicity. In general I've found NanoVG to have comparable performance to the master branch of Pathfinder. NanoVG currently performs better at text (PF's implementation is designed to get international text right first and optimization hasn't really been done yet), while PF generally does better at complex vector workloads like the tiger.
marcosscriven 6y ago
Naive question - but does this change to make use of DX11/shaders in newer GPUs make it more likely to have a good cross-platform UI for app development? I’ve looked before and although OpenGL is good for windowing, widgets etc, it’s not great for sub pixel rendered/anti aliased text.
- raphlinus 6y ago
  Yes, the motivation and long term goal for this work is to provide a performant foundation for cross-platform UI. There's a lot to be done though!
Const-me 6y ago
I wonder how it compares performance-wise against my solution of the same problem: https://github.com/Const-me/Vrmac#vector-graphics-engine https://github.com/Const-me/Vrmac#vector-graphics-engine I don’t use compute shaders, tessellating input splines into polylines, and building triangular meshes from these.

286 points by raphlinus 6y ago

Fast 2D rendering on GPU has represented the last few months of concentrated work for me, and I'm happy to present the results now. It's required going pretty deep into various aspects of GPU compute, so feel free to ask me about that, 2D vector graphics, or anything related.

tenaciousDaniel 6y ago
Hi! This looks interesting, although I confess I'm very new to this area. I'm writing a programming language specifically for UI designers, and currently I'm building a naive implementation in html5 canvas. I know very little about low-level rendering - just really know how to use drawing API's like canvas and Quartz 2D. With that being said, I'm looking to delve deeper into this subject. Do you have any recommendations on where to start? Whenever I look beyond simple drawing API's, the focus seems to be entirely on 3D rendering (which is interesting but not my main focus right now).
- raphlinus 6y ago
  It's a good question, as honestly the knowledge for 2D graphics is pretty arcane, as opposed to 3D being so widely taught. I actually started a github repo for a book on 2d graphics but have no idea whether I'll actually finish it. In the meantime, antigrain.com is one good (if old) source. The original PostScript "red book" was extremely influential in its time (it's where I learned a lot of this stuff) but is quite dated now. Best of luck, and I'm also happy to field requests for more specific areas. For example, for color theory (an important aspect of 2D graphics!), handprint.com is quite a remarkable resource.
  - tenaciousDaniel 6y ago
    Thanks!
  - blondin 6y ago
    > knowledge for 2D graphics is pretty arcane such an unfortunate state of affairs! i am currently learning how to render graphics using the GPU on my mac using apple metal. what i am getting is that the GPU has been optimized for 3D rendering?! GPUs make no provision or easy way for rendering 2D graphics? it makes no sense to me... that's where you start...
    - dungdang 6y ago
      now this may be a dimb question, but why would you start there? as far as i understand, 2d and 3d have literally zero to do with each other in how they are rendered. one is a bunch of triangles. the other is lines, curves, thickness, gradients, and fonts (which are essentially little programs)
      - blondin 6y ago
        that's not a dumb question! it's my own fault. i only have knowledge of 2D graphics.
      - chrisseaton 6y ago
        > the other is lines, curves, thickness, gradients, and fonts You can reduce all these to drawing triangles.
        deleted 6y ago
        [deleted]
        Jasper_ 6y ago
        Doing that reduction is surprisingly difficult, and usually a serial algorithm that runs on the CPU, and a naive approach is a resulting triangle set is not efficient to a GPU, but that's what toolkits like cairo, Direct2D, nanovg do. Raph is describing an architecture where path evaluation happens on the GPU, without being baked to triangles.
        raphlinus 6y ago
        > You can reduce all these to drawing triangles. You can, people have tried this, and it sucks. The main problem is that the conversion of Bézier paths to triangles is a hard problem with lots of conditional branching. Even when you do it, there is the other problem of rendering triangles with really good antialiasing, MSAA forces a compromise between performance and quality. By contrast, piet-gpu does an exact-area calculation for antialiasing. So it's not a question of whether you can do it, but whether it works well, and approaches like piet-gpu absolutely stomp triangles.
  - smallstepforman 6y ago
    Haiku OS AppServer (the screen rendering component, similar to Unix X11) is a full GUI system implemented with AntiGrain Geometry as the renderer.
  - gct 6y ago
    I think that blend2d (https://blend2d.com/ https://blend2d.com/) is a worthy successor to AGG, and it's under active development.
    - longstation 6y ago
      Thank you for the link. It was a great reading. But I just want to point out that Blender2D is a software renderer.
      - gct 6y ago
        So is/was AGG
- syspec 6y ago
  I recommend the blog of the OP, so many gems
awfulaxolotl 6y ago
How well does this approach work for something like 2D data visualization where most of the visual elements are the same -- i.e. can be instanced in OpenGL/etc? Thanks for publishing this, it's awesome work! I'm looking forward to progression to wgpu hinted in the Github README.
- raphlinus 6y ago
  I absolutely have data visualization in mind for this, as I think it can benefit greatly from the scale. But the pipeline I've built is very agile, it will easily handle a diverse mix of items. It's not like OpenGL etc where there's a certain amount of overhead for a draw call so there are significant gains to be had from instancing and batching. It is likely that CPU-side encoding can be made more efficient, though, by just filling in quantities to a template, rather than encoding from scratch.
leeoniya 6y ago
any relation to the work currently being done here by pcwalton? https://github.com/servo/pathfinder/pull/350 https://github.com/servo/pathfinder/pull/350 hopefully you guys aren't doing identical work in parallel (no pun intended) :)
- pcwalton 6y ago
  Yes, there's been a lot of influence in both directions. The approaches have a lot in common, but also have significant differences. I plan to write up a description of the Pathfinder compute work soonish.
  - paufernandez 6y ago
    I would love that!
- raphlinus 6y ago
  As Patrick indicated, there's a lot of cross-fertilization of ideas, and one of the best outcomes of this work would be for high performance compute rendering to ship in Pathfinder.
RivieraKid 6y ago
- Shouldn't 2D rendering be a solved problem given that it's basically a subset of 3D rendering? - Don't libraries like Skia, Qt, Cairo use GPU rendering? I've always assumed so. I mean, this is 2020, GPUs have been around for decades.
- pcwalton 6y ago
  > - Shouldn't 2D rendering be a solved problem given that it's basically a subset of 3D rendering? The problem is that primitives artists use are different. 3D rendering tends to all consist of polygon meshes, which are relatively easy to render. 2D rendering (basically) consists of Bezier paths, which are harder. The equivalent in 3D, which is adaptive subdivision, is not really a solved problem in real-time either. Additionally, 2D rendering quality tends to be more important than 3D rendering quality. Whereas you can get away with 4xMSAA or hacks like FXAA in 3D, true 16xAA (without hacks) is the absolute minimum for 2D rendering quality nowadays, and even it isn't considered great for some tasks like font rendering (Pathfinder and piet-gpu both use analytic AA which is effectively 256xAA). > - Don't libraries like Skia, Qt, Cairo use GPU rendering? I've always assumed so. I mean, this is 2020, GPUs have been around for decades. There's a difference between renderers with GPU support and renderers that are oriented around using the GPU efficiently. In many cases this results in an order-of-magnitude speedup. On the GPU, state changes are expensive, and many such renderers that have GPU support don't really go out of their way to avoid them. There are also occlusion culling optimizations that most renderers don't do, but piet-gpu and Pathfinder do.
- stanfordkid 6y ago
  Things like multi-resolution font-rendering are actually more complicated than one might imagine. The easy way is to just tesselate the font into polygons -- but this tesselation often depends on the zoom level. The same thing can be said about implicit curves etc. Most libraries do use GPU's for basic draw operations (e.g rendering a gradient), but to build something like Photoshop -- you need much more complexity.
- raphlinus 6y ago
  Others have spoken to this, but as a general introduction I highly recommend Jasper's post "Why are 2D vector graphics so much harder than 3D?" In short, no, it's not a solved problem. Also, I see a lot of variations of this question, but I should state this more clearly. There's been accelerated graphics in one form or another for a long time, but what I'm doing is a completely different type of thing. In my world, on the CPU you just encode the scene into a binary representation that's optimized for GPU but in many ways is like flatbuffers, and then the GPU runs a highly parallel program to render the whole thing. In previous approaches, the CPU is deeply involved in taking the scene apart and putting it back together in a form that's well suited to relatively dumb pixel pipes. Now that GPUs are really fast, that approach runs into limitations. It also depends what you're trying to do. I'm focusing here on dynamic paths (and thus font rendering), while most of the libraries optimized for UI put text into texture atlases and then use the GPU to composite quads to the final surface, something they can do well. https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-harder-than-3d-graphics/ https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-harde...
moonchild 6y ago
It looks like the main fruit of this is piet-gpu[1]. How close would you say it is to stable? And how complete a realisation of the concepts put forth in the OP? I'm currently working on a ui library, and evaluating alternatives for hw acceleration. The whole library wants to be in c, so I would want to rewrite the library by release-time to use it, but don't want to waste time doing that now if it's still really volatile. 1. https://github.com/linebender/piet-gpu https://github.com/linebender/piet-gpu
- raphlinus 6y ago
  Not very, and not very. This is research, it depends on compute capabilities that have really only gone mainstream in GPU lately, and the current codebase is about trying out the ideas. Best of luck in your project!
MrBuddyCasino 6y ago
Wow, those numbers look very impressive on an absolute basis! Do you have an idea on how piet-gpu compares to Skia? I'm not a graphics guy, my impression was that it is currently the dominant vector graphics renderer.
structural 6y ago
What's been your overall experience with using Vulkan shaders for compute? Are there basic primitives that are missing from shading languages and/or have you found any impedence mismatches between writing shaders vs. how you might describe the same algorithms in other languages?
- NotCamelCase 6y ago
  I pondered on the same subject recently as I was implementing the same algorithm (i.e. Mandelbrot set) on the CPU (scalar vs SIMD) and GPU compute using fixed-point and floating-point for comparisons (if interested: https://tayfunkayhan.wordpress.com/2020/06/03/mandelbrot-in-floating-point-vs-fixed-point https://tayfunkayhan.wordpress.com/2020/06/03/mandelbrot-in-...). It bothers me how little progress has been done on "shading" languages front compared to overall many-core computation models and capabilities over the years. And, that is despite the fact that shaders are very often where the most time is spent in modern workloads. Compute with Vulkan is another story. It offers some nice abstractions, but it shows that it's mostly intended for async-compute/work-offloading for rendering, IMO. Too much fruction.
- raphlinus 6y ago
  That's a big topic. I've been able to work around the missing primitives (for example, I autogenerate code for Rust-style structs and enums), but have had much bigger struggles around two issues: tools, which are still quite primitive, and understanding performance, which is extremely difficult. These two problems intersect because I can imagine a lot better tools for digging into performance issues. One that I would have paid good money for is an instruction-level simulator that would highlight the source code to tell me where the stalls, bank conflicts, divergence problems, etc. are in the source code. Such a thing is possible (there are academic papers like [1]), but not as far as I know usable in daily development. The "impedance mismatch" is that you (generally) have to write in a style to extract lots of parallelism. This tends to be very different than the way you'd write scalar CPU code, but not completely alien to me as it has a lot of similarity with the way you'd write SIMD. I've pretty much gotten the hang of it now. I'm thinking of a blog post of redoing path_coarse.comp from its current basically scalar style to a more parallel version, as that would I think illuminate the issues. [1] http://comparch.gatech.edu/hparch/papers/gera_ispass18.pdf http://comparch.gatech.edu/hparch/papers/gera_ispass18.pdf
deleted 6y ago
[deleted]

chadcmulligan 6y ago

This is very impressive. Have you benchmarked it against core graphics on a Mac? I believe they've done a similar thing - performing the render on Metal

pcwalton 6y ago
Core Graphics is generally slower than Cairo/Pixman.
c-smile 6y ago
CoreGraphics is pure CPU rasterizer, similar to GDI+. Sciter (https://sciter.com https://sciter.com) on MacOS uses Skia/OpenGL by default with fallback to CoreGraphics. It is possible to configure Sciter to use CoreGraphics on MacOS to compare these two on the same UI by using SciterSetOption(NULL, SCITER_SET_GFX_LAYER, GFX_LAYER_CG); I think it is safe to say that Skia/OpenGL is 5-10 times more performant than CG on typical UI tasks.
- chadcmulligan 6y ago
  Interesting, thanks - I remember hearing at one of the WWDC that Coregraphics got a 10x speed improvement using metal. I just read the fine print - it seems draw calls only have a 10x speed improvement, I assume because they render through a layer of some sort. The CA* libraries may use metal - animation and layers, which I guess is where the 10x draw call improvement comes in maybe. Some discussion here https://arstechnica.com/civis/viewtopic.php?t=1285571 https://arstechnica.com/civis/viewtopic.php?t=1285571 , though a lot of guessing.
  - pcwalton 6y ago
    And before that, I'm pretty sure Core Graphics used OpenGL in some situations. I remember seeing CG::OGL in stack traces for WebKit.
  - c-smile 6y ago
    I believe that the most HW optimizations are made in MacOS on DWM level - window composition and animation. Window surface is just a bitmap in RAM that needs to be populated by CoreGraphics/CPU. In any case Acrylic/Vibrancy effects or "blur-behind background" (like on screenshots here: https://sciter.com/sciter-4-2-support-of-acrylic-theming/ https://sciter.com/sciter-4-2-support-of-acrylic-theming/) are achievable only on GPU and on DWM level.

c-smile 6y ago

Question to the author: is this only implementation of GPU rasterizer? What about anything like WARP (https://en.wikipedia.org/wiki/Windows_Advanced_Rasterization_Platform https://en.wikipedia.org/wiki/Windows_Advanced_Rasterization...) - fallback rendering when GPU is not available ?

raphlinus 6y ago
It's something I'd like to explore at some point, also Swiftshader, which might be easier to explore, as it's already Vulkan. I expect performance to be pretty good, but there are already really advanced CPU renderers such as Blend2D. Doing serious performance evaluation is hard work, so I actually hope it's something others take up, as I have pretty limited time for it myself.
- c-smile 6y ago
  I understand. Problem is that any practical 2D rendering solution shall support as GPU as CPU rendering unfortunately. It would be interesting to see any GPU equivalent of something like AGG by Max Shemanarev, RIP.
  - throwaway9087 6y ago
    Practical 2D renderers implement an abstraction layer that lets them easily redirect their output to a number of low level libraries, which can be either CPU or GPU-based (or a mix). I worked on a few such 2D stacks, including OpenGL, DirectX, WebGL, and AGG based, and it took no more than a few days to add new 2D backends to an existing pipeline. Most 2D rendering is based on concepts from PostScript so it's usually easy to do such ports -- except for AGG, that one was a bit like a library from outer space. Maxim himself worked on a hardware accelerated 2D renderer for Scaleform, and it looked nothing like AGG, mostly because AGG is practically impossible to move to a GPU implementation.
    - c-smile 6y ago
      I know, AGG is just a set of primitives but not an abstraction like class Graphics {...}. But that one can be assembled from them. Did it once for early versions of Sciter. Ideally GPU and CPU rendering backends should have pixel perfect match that makes "adding new 2D backend" tricky at best.

aasasd 6y ago

Noob question. Isn't some GPU acceleration for 2D used since forever? I keep being perplexed as to why browsers still have problems with it (at least on my older Macbook with a shitty Intel's embedded GPU). I thought some stuff, like scrolling, was offloaded at least in early/mid-2000s, making for a distinctly meh experience when proper drivers weren't installed.

thechao 6y ago
GPUs derive most of their parallelism from emitting rows of related “quads” (the smallest unit of the screen that supports a difference operator). 2D graphics are “chatty” in window sizes that are more like 5–10 units in diameter. It’s hell on HW perf. To make things worse, 2D applications usually want submillisecond latency. A GPU driver/hw stack will struggle to get latency below 1–2ms. When there’s lots of multipass rendering (which is a thing 2D also wants a lot of), latency can to 10+ ms.
- raphlinus 6y ago
  Well said, thanks. It's also a goal of this work to do compositing inside the "fine rasterizer" compute kernel, to avoid multipass as much as possible. Of course, in the general case for stuff like blur that's hard, but that's one reason why I've been working on special cases like blurred rounded rectangles, which I can render pretty much at lightspeed. (It's not integrated into the current code but would be quite easy)
- rasz 6y ago
  What does latency have to do with 2D? Are you suggesting UI toolkits are writing GPU commands synchronously? You should fill display list, fire it and forget about GPU until the next update.
  - thechao 6y ago
    Pen drawing for UIs. The ideal case is to update a small portion of the screen at about 240hz, to provide a good simulation of pen/pencil feedback. Really, your latency envelope should be on the order of the propagation of sound through the barrel of the marking device, but screens don’t update that fast.

deleted 6y ago

[deleted]

amelius 6y ago

Is GPU here synonymous with Nvidia?

adamnemecek 6y ago
It uses metal and apple uses amd do no.
raphlinus 6y ago
No, it's designed to be portable to all GPU hardware that can support compute, which these days is a pretty good chunk of the fleet. I tested it on Linux on Intel HD 4000, and the master branch seems to run just fine, though previous versions. A lot of the academic literature (Massively Parallel Vector Graphics, the Li et al scanline work) is dependent on CUDA, but that's just because tools for doing compute on general purpose graphics APIs are so primitive. I have a talk and a bunch of blog posts on exactly this topic, as I had to explore deeply to figure it out. See https://news.ycombinator.com/item?id=22880502 https://news.ycombinator.com/item?id=22880502 and https://raphlinus.github.io/gpu/2020/04/30/prefix-sum.html https://raphlinus.github.io/gpu/2020/04/30/prefix-sum.html for more breadcrumbs on that.

stephc_int13 6y ago

Something I am not sure to completely understand, after reading the article, is: in which way is it better/ more desirable than the classical approach of GPU rendering? (with vertex, triangles, and shaders)

pcwalton 6y ago
The basic problem is that, without compute, you either have to encode the vector scene into GPU primitives (triangles) as quickly as possible, and all the ways I know of to do that either involve (a) an expensive CPU process or (b) a cheap CPU process but way too much overdraw on GPU side. Compute gives you the best of both worlds, allowing you to upload outlines directly to GPU and do the processing necessary to lower them to primitives on the GPU itself. Pathfinder in there is actually using regular old GPU rasterization with triangles and so forth, and I'm fairly confident it's about as fast as you can go at D3D10 level (i.e. no compute shaders) without sacrificing quality. Note that the numbers can vary wildly depending on hardware. On my MacBook Pro, with a powerful CPU and limiting myself to the Intel integrated GPU only, Pathfinder is actually about equal to the GPU compute approach on a lot of scenes like the tiger, though it uses a lot of CPU.
- stephc_int13 6y ago
  Is it true for all 2D rendering? From my intuition, this seems pretty specialized for vector-like rendering, with a lot of small bezier shapes.
  - pcwalton 6y ago
    Yeah, when I say 2D I mean vector art. There are a lot of things under the heading of 2D rendering, such as blitting raster sprites, that are much closer to being solved problems. (Though you might be surprised--power concerns, coupled with greatly increased pixel density, have brought renewed attention to performance of blitting lately...)
- Const-me 6y ago
  > all the ways I know of to do that either involve (a) an expensive CPU process or (b) a cheap CPU process but way too much overdraw on GPU side That's fixable, take a look: https://github.com/Const-me/Vrmac#vector-graphics-engine https://github.com/Const-me/Vrmac#vector-graphics-engine My CPU process is moderately expensive, and I use quite a few tricks to reduce overdraw, e.g. both draw calls (the complete vector image, however complex, takes 2 draw calls to render) use hardware depth buffer and early Z rejection.

The_rationalist 6y ago

How would it compare with https://github.com/intel/fastuidraw https://github.com/intel/fastuidraw And https://github.com/blend2d/blend2d https://github.com/blend2d/blend2d And the skia Vulkan backend?

Abishek_Muthian 6y ago

iTerm's usage of metal on macOS is good example of benefits in implementing 2D rendering on GPU[1]. [1]https://gitlab.com/gnachman/iterm2/-/wikis/Metal-Renderer https://gitlab.com/gnachman/iterm2/-/wikis/Metal-Renderer

bori5 6y ago
That links to iTerm2 which is what I’m guessing you meant to say ? Kitty is also a GPU accelerated terminal emulator and one that I enjoy using https://sw.kovidgoyal.net/kitty/ https://sw.kovidgoyal.net/kitty/ Not sure if it uses Metal on Mac though.

The_rationalist 6y ago

The benchmarcks do not compare to skia (Vulkan) so it is unclear if the state of the art has been beaten on a specific example.

chrismorgan 6y ago

I would like to get an idea of the capabilities of these things. Suppose I was to make an ebook reader designed to be modelled more closely after paper, on a device like the Surface Book’s 13″ 3000×2000 display, showing two pages side-by-side. Each page might contain something like 2,000–2,500 letters. I want to be able to flip through pages like I might with a paper book, so that I might be roughly completely rendering several pages at once, and perhaps parts of several more pages; ideally it might render the page like a real 3D page, but if that’s too troublesome I’d settle for an affine transformation while flipping. Assume that the layout of the pages, with all the shaping, is all done ahead of time and is in memory. I’ve never seen anyone attempt anything like this before. In the old way of doing things, I think anyone attempting this would render each page to a bitmap and use that as a GPU texture, and I think that could provide acceptable performance (unless you were flipping rapidly through hundreds of pages, because that’d take a lot of GPU memory to keep), but I imagine that the quality of the rendering mid-turn would be fairly atrocious—it could be the sort of thing where the appearance of the text subtly changes half a second after you finish turning the page, as it switches from one renderer to a slightly different one. Would the performance of this new approach be sufficient to render what I describe at 60fps, while rendering each frame perfectly? I was talking about the Surface Book; its Intel Core i7-6600U has Intel HD Graphics 520 as its GPU; probably not too far off your 630’s results. (And I’m interested in what integrated graphics can do, more than a dedicated GPU.) My guess based upon your paper-1 results is that this is probably just barely possible with integrated graphics, so long as you employ a few tricks to reduce the amount of work required.

raphlinus 6y ago
Short answer, yes, I believe it could work, though not with much margin to spare. It might require some dedicated optimization work to fit the frame budget, especially when you know things about the scene (it's mostly small glyphs and with basically no overdraw) as opposed to being completely general purpose. Doing a warp transformation in the element processing kernel would probably work just fine, and give you realistic movement and razor-sharp rendering. I'd love to see such a thing and would very much like to encourage people to build it based on my results :)

eco 6y ago

How does this (and Pathfinder for that matter) compare to NanoVG? I've recently been experimenting with swapping out Cairo for NanoVG and it seems much faster. The lack of dashed lines may kill my experiment though unless I can think of a decent workaround.

raphlinus 6y ago
I expect it to be faster, but doing performance evaluation is really hard work. Performance depends on so many variables, including the workloads you're trying to render and a ton of factors about the GPU and system. I know people are really curious about competitive performance against other renderers, and it's something I should probably do, but honestly I'd love it if somebody else took up that mantle.
- eco 6y ago
  I kind of meant more in terms of how they work and what they can do but I suppose I could do the work in answering that question myself by figuring out how nanovg works then comparing it to what you've written. I've been finding Patrick's pathfinder work fascinating so it's nice to have yet another project to follow.
  - raphlinus 6y ago
    Got it. So far my research prototype really just does filled paths and limited strokes, but the architecture should extend to a richer set of graphics primitives. Pathfinder also started out that way but recent work is building out a good chunk of the SVG imaging model. I definitely recommend checking it out, there's a good chance it'll do what you need.
pcwalton 6y ago
I have to say I'm consistently impressed by NanoVG. Not because it does anything fancy, but because it doesn't. It's not the fastest or the most featureful vector renderer out there, but it resides in a very nice sweet spot that balances performance, features, and simplicity. In general I've found NanoVG to have comparable performance to the master branch of Pathfinder. NanoVG currently performs better at text (PF's implementation is designed to get international text right first and optimization hasn't really been done yet), while PF generally does better at complex vector workloads like the tiger.

marcosscriven 6y ago

Naive question - but does this change to make use of DX11/shaders in newer GPUs make it more likely to have a good cross-platform UI for app development? I’ve looked before and although OpenGL is good for windowing, widgets etc, it’s not great for sub pixel rendered/anti aliased text.

raphlinus 6y ago
Yes, the motivation and long term goal for this work is to provide a performant foundation for cross-platform UI. There's a lot to be done though!

Const-me 6y ago

I wonder how it compares performance-wise against my solution of the same problem: https://github.com/Const-me/Vrmac#vector-graphics-engine https://github.com/Const-me/Vrmac#vector-graphics-engine I don’t use compute shaders, tessellating input splines into polylines, and building triangular meshes from these.