6 ms·

I've done a lot of work testing this type of latency in web browsers: https://google.github.io/latency-benchmark/ https://google.github.io/latency-benchmark/ O

I've done a lot of work testing this type of latency in web browsers: https://google.github.io/latency-benchmark/ https://google.github.io/latency-benchmark/ On Windows, DWM's display compositing adds one frame of latency to every window on screen. It's not possible to render a dragged object in any window that sticks to the mouse cursor without at least one frame of latency. But when you drag whole windows around they do stick to the mouse cursor with apparently zero frames of latency; how does DWM do it? Easy, they cheat by disabling the hardware mouse overlay during window dragging so that the mouse cursor gets that extra frame of latency too. You can prove this by enabling "Night light" in settings; watch the mouse cursor change colors as it transitions from hardware overlay to software rendering when you start dragging a window.

mehrdadn 7y ago
Do you know how DWM disables the hardware mouse overlay? Is it an IOCTL?
- gruez 7y ago
  https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-showcursor https://docs.microsoft.com/en-us/windows/win32/api/winuser/n... any application can do it, hence why you get laggy cursor in some games that opt to draw their own cursor.
  - mehrdadn 7y ago
    Wait are you saying DWM literally hides the cursor and then draws its own cursor manually? I thought it would change to a different type of fallback rendering in the kernel or something. Interesting, okay thanks.
gruez 7y ago
>On Windows, DWM's display compositing adds one frame of latency to every window on screen. It's not possible to render a dragged object in any window that sticks to the mouse cursor without at least one frame of latency. AFAIK you can bypass this by using dxgi flip model so no additional latency is incurred. There's still is going to be 1 frame of latency from the vsync though. >You can prove this by enabling "Night light" in settings; watch the mouse cursor change colors as it transitions from hardware overlay to software rendering when you start dragging a window. can't reproduce on my end. maybe the upped the night light implementation so the hardware cursor is tinted as well.
- modeless 7y ago
  > AFAIK you can bypass this by using dxgi flip model so no additional latency is incurred. Using the flip model only eliminates the latency if DWM promotes your window to a hardware overlay. On Nvidia systems this is simply not supported, so the latency is always there and it's impossible to get rid of it. Maybe DWM supports overlays on Intel or AMD, I'm not sure. It would be interesting for someone to test this. > There's still is going to be 1 frame of latency from the vsync though. Vsync does not inherently require any extra latency. You can render as close to vsync as you like to reduce the latency an arbitrary amount. That's what VR compositors do. All you need to do is ensure you can't flip during scanout and you can't get tearing.
  - raphlinus 7y ago
    My understanding is that promoting a window to hardware overlay is only supported on Kaby Lake and later Intel integrated graphics, and there it's a heuristic, so there's no way to guarantee getting it. You do have to be in flip mode, but in flip mode smooth resizing can't be done without artifacts. Currently druid downgrades to direct2d hwnd render targets during a live resize, but this feels hacky and is likely creating other problems. I've spent a fair amount of time investigating this and have a mostly written blog post on it, but right now I'm kind of sick of the topic - it's a good illustration of how easily software evolves into stuff that's complex and broken.
    - modeless 7y ago
      Great info, thanks. I'd love to see that blog post. I similarly got fed up with it after a bunch of investigation. I also have a speculative suspicion that the reason overlays aren't supported is that they were artificially omitted from the GeForce driver to support Nvidia's Quadro price discrimination. Ugh.
ReactiveJelly 7y ago
> It's not possible to render a dragged object in any window that sticks to the mouse cursor without at least one frame of latency. I thought this was a fact of all window managers? I'd noticed it when making games in SDL / SDL2 on Linux and just assumed it was because the X server couldn't possibly wait on me to paint a frame before updating its own cursor
DaiPlusPlus 7y ago
Could compositors be optimised to eliminate the extra frame of lag in the case where every window on-screen is being displayed “directly” by invisibly switching to a mode that maps each scanline and pixel column to a window’s framebuffer - and non-client areas to the window-manager’s UI buffer which is directly read by the monitor signal generator. While this would mean transparency effects wouldn’t work, it could supported with some special-casing. Basically a framebuffer-less hardware compositor. I think rendering windows to 3D deformable meshes [makes for cool demos](https://youtu.be/USedxVrU2Ko https://youtu.be/USedxVrU2Ko) but in practice we just don’t use it for anything besides window open/close animations. I had to use a monitor running at 30Hz for a while (4K over HDMI 1.4) and while that was bad enough, the compositor’s lag meant all window contents had an extra (unnecessary IMO) delay of 33ms. Add on to that normal monitor input lag. We’ll probably all shift to 120Hz w/ variable-rate refreshing as a new baseline standard over the next 10 years as Apple seems to be heading in that direction - at 120Hz the lag of the compositor would be acceptable - but I’m worried lazy that graphics devs are going to use that as an excuse to add another frame of latency...
- modeless 7y ago
  > Could compositors be optimised to eliminate the extra frame of lag in the case where every window on-screen is being displayed “directly” by invisibly switching to a mode that maps each scanline and pixel column to a window’s framebuffer Yes. This concept is called hardware overlays and there are varying levels of support for it in different GPUs and compositors. There are tradeoffs. Using multiple hardware overlays may cost extra power and/or memory bandwidth, the number of supported overlays may be very limited, alpha blending may not be supported, and the transforms that can be applied to overlays may be very limited. The extremely hardware specific nature of the restrictions and the lack of good APIs exposing overlays means they get much less use than they should.
  - heavenlyblue 7y ago
    Technically if your pixel operation is commutative and reversible (as could be with alpha blending), you could store a buffer of pixels at the current position, then undo the previous pixel by current window at that buffer and then re-apply the operation with the new pixel value, and then directly send this to the display? Am I missing something apart from the fact that alpha blending is not actually commutative and/or reversible; and the fact that nobody implemented this yet?

mehrdadn 7y ago

Do you know how DWM disables the hardware mouse overlay? Is it an IOCTL?

gruez 7y ago
https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-showcursor https://docs.microsoft.com/en-us/windows/win32/api/winuser/n... any application can do it, hence why you get laggy cursor in some games that opt to draw their own cursor.
- mehrdadn 7y ago
  Wait are you saying DWM literally hides the cursor and then draws its own cursor manually? I thought it would change to a different type of fallback rendering in the kernel or something. Interesting, okay thanks.

gruez 7y ago

>On Windows, DWM's display compositing adds one frame of latency to every window on screen. It's not possible to render a dragged object in any window that sticks to the mouse cursor without at least one frame of latency. AFAIK you can bypass this by using dxgi flip model so no additional latency is incurred. There's still is going to be 1 frame of latency from the vsync though. >You can prove this by enabling "Night light" in settings; watch the mouse cursor change colors as it transitions from hardware overlay to software rendering when you start dragging a window. can't reproduce on my end. maybe the upped the night light implementation so the hardware cursor is tinted as well.

modeless 7y ago
> AFAIK you can bypass this by using dxgi flip model so no additional latency is incurred. Using the flip model only eliminates the latency if DWM promotes your window to a hardware overlay. On Nvidia systems this is simply not supported, so the latency is always there and it's impossible to get rid of it. Maybe DWM supports overlays on Intel or AMD, I'm not sure. It would be interesting for someone to test this. > There's still is going to be 1 frame of latency from the vsync though. Vsync does not inherently require any extra latency. You can render as close to vsync as you like to reduce the latency an arbitrary amount. That's what VR compositors do. All you need to do is ensure you can't flip during scanout and you can't get tearing.
- raphlinus 7y ago
  My understanding is that promoting a window to hardware overlay is only supported on Kaby Lake and later Intel integrated graphics, and there it's a heuristic, so there's no way to guarantee getting it. You do have to be in flip mode, but in flip mode smooth resizing can't be done without artifacts. Currently druid downgrades to direct2d hwnd render targets during a live resize, but this feels hacky and is likely creating other problems. I've spent a fair amount of time investigating this and have a mostly written blog post on it, but right now I'm kind of sick of the topic - it's a good illustration of how easily software evolves into stuff that's complex and broken.
  - modeless 7y ago
    Great info, thanks. I'd love to see that blog post. I similarly got fed up with it after a bunch of investigation. I also have a speculative suspicion that the reason overlays aren't supported is that they were artificially omitted from the GeForce driver to support Nvidia's Quadro price discrimination. Ugh.

ReactiveJelly 7y ago

> It's not possible to render a dragged object in any window that sticks to the mouse cursor without at least one frame of latency. I thought this was a fact of all window managers? I'd noticed it when making games in SDL / SDL2 on Linux and just assumed it was because the X server couldn't possibly wait on me to paint a frame before updating its own cursor

DaiPlusPlus 7y ago

Could compositors be optimised to eliminate the extra frame of lag in the case where every window on-screen is being displayed “directly” by invisibly switching to a mode that maps each scanline and pixel column to a window’s framebuffer - and non-client areas to the window-manager’s UI buffer which is directly read by the monitor signal generator. While this would mean transparency effects wouldn’t work, it could supported with some special-casing. Basically a framebuffer-less hardware compositor. I think rendering windows to 3D deformable meshes [makes for cool demos](https://youtu.be/USedxVrU2Ko https://youtu.be/USedxVrU2Ko) but in practice we just don’t use it for anything besides window open/close animations. I had to use a monitor running at 30Hz for a while (4K over HDMI 1.4) and while that was bad enough, the compositor’s lag meant all window contents had an extra (unnecessary IMO) delay of 33ms. Add on to that normal monitor input lag. We’ll probably all shift to 120Hz w/ variable-rate refreshing as a new baseline standard over the next 10 years as Apple seems to be heading in that direction - at 120Hz the lag of the compositor would be acceptable - but I’m worried lazy that graphics devs are going to use that as an excuse to add another frame of latency...

modeless 7y ago
> Could compositors be optimised to eliminate the extra frame of lag in the case where every window on-screen is being displayed “directly” by invisibly switching to a mode that maps each scanline and pixel column to a window’s framebuffer Yes. This concept is called hardware overlays and there are varying levels of support for it in different GPUs and compositors. There are tradeoffs. Using multiple hardware overlays may cost extra power and/or memory bandwidth, the number of supported overlays may be very limited, alpha blending may not be supported, and the transforms that can be applied to overlays may be very limited. The extremely hardware specific nature of the restrictions and the lack of good APIs exposing overlays means they get much less use than they should.
- heavenlyblue 7y ago
  Technically if your pixel operation is commutative and reversible (as could be with alpha blending), you could store a buffer of pixels at the current position, then undo the previous pixel by current window at that buffer and then re-apply the operation with the new pixel value, and then directly send this to the display? Am I missing something apart from the fact that alpha blending is not actually commutative and/or reversible; and the fact that nobody implemented this yet?