4 ms·

> and many people believe that damage regions are obsolete Old-style damage regions are obsolete. But new-style ones leveraging buffer age & swap-with-damage a

> and many people believe that damage regions are obsolete Old-style damage regions are obsolete. But new-style ones leveraging buffer age & swap-with-damage are not. Vulkan changes this, but these older EGL extensions are good introductions to the concepts: https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_buffer_age.txt https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_... https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_partial_update.txt https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_... and https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_swap_buffers_with_damage.txt https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_... > Flutter is a good modern approach to this, and its “layers” are one of the keys to its performance. Flutter actually does it badly. Layers are terrible for performance because they are a source of jitter (and memory usage), and jitter often results in jank. They are necessary in some cases, like for correct alpha blending, but in general should be avoided if your goal is to be smooth & fast. You want to optimize for your slowest frames, not your fastest ones. Flutter's excessive layering can also rapidly be a net-loss in performance due to the overhead of render target switching, increase in memory bandwidth required, and fewer opportunities for overdraw avoidance & disabled blending.

raphlinus 8y ago
Agree on both fronts! (Also, your username seems familiar somehow - have we had discussions around this before? :) I didn't talk about this in the post because I haven't actually implemented fancy compositing, but based on what I've seen you never want to render to texture (which Flutter does sometimes) as it uses up scarce global GPU memory bandwidth, and if you're doing that using heuristics, then performance gets very unpredictable. I should express this more clearly, but the goal in piet-metal is not to do the fastest renderer (I'm sure it isn't), but the one with the most consistent and predictable performance.
- kllrnohj 8y ago
  > but based on what I've seen you never want to render to texture (which Flutter does sometimes) Flutter's "layers" are rasterized to texture if they are unchanged for N frames (I think N = 3?). Which happens kind of a lot, and is (one of) the reasons Flutter has such a big RAM footprint. Flutter's layers remind me a lot of the hacky, terrible will-change CSS property. Other than that it's just a rather crude way to hack up the rendering commands into smaller display lists. Android has been doing this for ages with RenderNode with both display lists & transform properties on those display lists.
  - nine_k 8y ago
    Doesn't macOS do a similar thing?
    - pcwalton 8y ago
      Well, in macOS (and Windows) the layers are explicitly created by the programmer: they aren't implicitly created based on heuristics. That's a significant difference. The Web has historically followed Flutter here in having a complex set of "layerization" heuristics. One of the main motivations of WebRender is to change that.
      - raphlinus 8y ago
        Patrick is of course right here, macOS isn't quite like Flutter because it basically always texturizes, although ultimately this is under control of the programmers. And perhaps it would be more accurate to say Flutter follows the Web, as many of the team are former Chrome engineers, though I think they've tried to clean up a lot of the warty stuff. And absolutely, the idea of relying less on the compositor is not original to me, WebRender has similar goals, though I think I can make things performant, rich, and fairly simple by leveraging compute.
      - kvark 8y ago
        Ironically, with picture caching WebRender converges to the very same heuristics that drive Flutter layerization...
    - raphlinus 8y ago
      Yes, macOS relies very heavily on CALayer to accomplish smooth scrolling, largely because much of the content of those layers is software-rendered. One thing that I find interesting is the subtle effect this has on the aesthetics of macOS (and iOS also) - lots of semitransparent planes sliding over each other and fading, very few other things that would be expensive in that model. I think this is a bad approach on modern hardware, for the reasons kllrnohj stated.
Jasper_ 8y ago
In Vulkan, buffer age is implicit with the FIFO swap-chain model. Agreed that there's currently no SwapWithDamage equivalent to tell the compositor about the damaged region in Vulkan, but if it's wanted, it shouldn't be too difficult of an extension to draft or support. I'm open to throwing it in the queue if wanted. What swapchain platforms were you thinking of it for?
- raphlinus 8y ago
  If you're talking about improving support for incremental present in Vulkan implementations, let's definitely connect. Certainly I can see how to do it well on Windows/DirectX, using IDXGISwapChain1::Present1. On macOS I think it's impossible (though Chrome sometimes fakes it by pasting update layers on top of the base window in the compositor).
  - kllrnohj 8y ago
    For incremental buffer updates you just know the content of the new VkImage you're going to draw into - it's the same thing it was the last time you present. So you keep a ringbuffer of dirt rects for the last N frames, and then when you acquire the next frame you just intersect the last N frames of dirty and that's the area of the buffer to update. This is "standard" vulkan, there's no platform aspect to this. It has very widespread support in EGL/GL as well with either egl_ext_buffer_age extension or, more commonly, the EGL_KHR_partial_update extension. If you then want to limit the area of the screen that's re-composited THAT'S when swapchain extensions or platform support enters the picture. That would be eglSwapBuffersWithDamage or VkPresentRegionKHR. But this optimization has very little performance impact, and typically minimal battery life improvement as well. Worth doing, but the partial re-render of the buffer is far more significant. > though Chrome sometimes fakes it by pasting update layers on top of the base window in the compositor For a long time that was also just how chrome handled display list updates. They just had a stack of them, and re-rendering a part of the screen just created an entirely new displaylist and plopped it on top: https://www.chromium.org/developers/design-documents/impl-side-painting https://www.chromium.org/developers/design-documents/impl-si... (see "PicturePile") I wouldn't necessarily follow Chrome as an example of what to do - there's a lot of legacy in Chrome's rendering stack. Also web pages are huge, so they do things like put rendering commands in an rtree because quick rejects happen 100x more often than actual draws.

raphlinus 8y ago

Agree on both fronts! (Also, your username seems familiar somehow - have we had discussions around this before? :) I didn't talk about this in the post because I haven't actually implemented fancy compositing, but based on what I've seen you never want to render to texture (which Flutter does sometimes) as it uses up scarce global GPU memory bandwidth, and if you're doing that using heuristics, then performance gets very unpredictable. I should express this more clearly, but the goal in piet-metal is not to do the fastest renderer (I'm sure it isn't), but the one with the most consistent and predictable performance.

kllrnohj 8y ago
> but based on what I've seen you never want to render to texture (which Flutter does sometimes) Flutter's "layers" are rasterized to texture if they are unchanged for N frames (I think N = 3?). Which happens kind of a lot, and is (one of) the reasons Flutter has such a big RAM footprint. Flutter's layers remind me a lot of the hacky, terrible will-change CSS property. Other than that it's just a rather crude way to hack up the rendering commands into smaller display lists. Android has been doing this for ages with RenderNode with both display lists & transform properties on those display lists.
- nine_k 8y ago
  Doesn't macOS do a similar thing?
  - pcwalton 8y ago
    Well, in macOS (and Windows) the layers are explicitly created by the programmer: they aren't implicitly created based on heuristics. That's a significant difference. The Web has historically followed Flutter here in having a complex set of "layerization" heuristics. One of the main motivations of WebRender is to change that.
    - raphlinus 8y ago
      Patrick is of course right here, macOS isn't quite like Flutter because it basically always texturizes, although ultimately this is under control of the programmers. And perhaps it would be more accurate to say Flutter follows the Web, as many of the team are former Chrome engineers, though I think they've tried to clean up a lot of the warty stuff. And absolutely, the idea of relying less on the compositor is not original to me, WebRender has similar goals, though I think I can make things performant, rich, and fairly simple by leveraging compute.
    - kvark 8y ago
      Ironically, with picture caching WebRender converges to the very same heuristics that drive Flutter layerization...
  - raphlinus 8y ago
    Yes, macOS relies very heavily on CALayer to accomplish smooth scrolling, largely because much of the content of those layers is software-rendered. One thing that I find interesting is the subtle effect this has on the aesthetics of macOS (and iOS also) - lots of semitransparent planes sliding over each other and fading, very few other things that would be expensive in that model. I think this is a bad approach on modern hardware, for the reasons kllrnohj stated.

Jasper_ 8y ago

In Vulkan, buffer age is implicit with the FIFO swap-chain model. Agreed that there's currently no SwapWithDamage equivalent to tell the compositor about the damaged region in Vulkan, but if it's wanted, it shouldn't be too difficult of an extension to draft or support. I'm open to throwing it in the queue if wanted. What swapchain platforms were you thinking of it for?

raphlinus 8y ago
If you're talking about improving support for incremental present in Vulkan implementations, let's definitely connect. Certainly I can see how to do it well on Windows/DirectX, using IDXGISwapChain1::Present1. On macOS I think it's impossible (though Chrome sometimes fakes it by pasting update layers on top of the base window in the compositor).
- kllrnohj 8y ago
  For incremental buffer updates you just know the content of the new VkImage you're going to draw into - it's the same thing it was the last time you present. So you keep a ringbuffer of dirt rects for the last N frames, and then when you acquire the next frame you just intersect the last N frames of dirty and that's the area of the buffer to update. This is "standard" vulkan, there's no platform aspect to this. It has very widespread support in EGL/GL as well with either egl_ext_buffer_age extension or, more commonly, the EGL_KHR_partial_update extension. If you then want to limit the area of the screen that's re-composited THAT'S when swapchain extensions or platform support enters the picture. That would be eglSwapBuffersWithDamage or VkPresentRegionKHR. But this optimization has very little performance impact, and typically minimal battery life improvement as well. Worth doing, but the partial re-render of the buffer is far more significant. > though Chrome sometimes fakes it by pasting update layers on top of the base window in the compositor For a long time that was also just how chrome handled display list updates. They just had a stack of them, and re-rendering a part of the screen just created an entirely new displaylist and plopped it on top: https://www.chromium.org/developers/design-documents/impl-side-painting https://www.chromium.org/developers/design-documents/impl-si... (see "PicturePile") I wouldn't necessarily follow Chrome as an example of what to do - there's a lot of legacy in Chrome's rendering stack. Also web pages are huge, so they do things like put rendering commands in an rtree because quick rejects happen 100x more often than actual draws.