6 ms·

Apple's CVDisplayLink Doesn't Link to Your Display

110 points by jordwalke 9y ago

deleted 9y ago
[deleted]
adamnemecek 9y ago
There’s quite a few approaches, it’s hard to figure out which one is appropriate for this one. I think that enabling setNeedsDisplay and dispatching commands asynchronously might do the trick? With asynchronous dispatch I’ve actually been getting 120 FPS. But it’s very much case by case basis and I might not be understanding the use case. IIRC this project Gets to 120 when you drag your mouse https://github.com/jtbandes/metalbrot-playground https://github.com/jtbandes/metalbrot-playground
- trishume 9y ago
  I have a lot of avenues to test but unfortunately testing each one requires setting up a sample app with signposts and recording a bunch of runs in Instruments, so there's a bit of work to do if I want to figure out the right way. And do you have a 120fps+ monitor? If not I'm not sure what you mean by 120fps unless it's broken or not vsynced at all and painting twice per frame. If you have a proper way of waiting on vsync 120fps is just a waste of power with no benefits and you don't want that.
  - adamnemecek 9y ago
    Haha welcome to gpu programming. You should hit up Warren Moore on Twitter (@warrenm), he’s a real life savior. Re: 120fps, I questioned it myself but the difference is perceivable. Idk if something is broken or not I was confused too. Re power consumption, that’s why you enable setneedsdisplay.
    - trishume 9y ago
      Maybe the reason that 120fps is perceivably better is that your draw calls don't line up with vsync, so doubling the number of frames eliminates the dropped frame effect in @jordwalke's diagram! And thanks for the Twitter recommendation, I followed him.
- jvz 9y ago
  > IIRC this project Gets to 120 when you drag your mouse What causes that kind of thing? I've noticed stuff like that while playing video in mpv sometimes (despite my Mac being easily fast enough to play the video) and it just baffles me.
  - Doxin 9y ago
    Moving the mouse has a tendency to send various events to windows. Sometimes just the mousemotion event is enough to trigger the program to redraw. I'm not sure how cursors are implemented on osX but it might just be marking the bit under the cursor dirty, triggering a redraw.
mrpippy 9y ago
Bring back VBL tasks. http://mirror.informatimago.com/next/developer.apple.com/documentation/mac/Processes/Processes-74.html http://mirror.informatimago.com/next/developer.apple.com/doc...
- revelation 9y ago
  I like the Linux DRM atomic interface, they recently added native fences so you can pass in a synchronization object from the GPU for when it's done rendering and the flip can proceed and you get back a fence that is signalled once the flip is done. Basically all asynchronous now but it's still synced to the hardware flip, no timer bullshit. IIRC it's somewhat inspired by Android SurfaceFlinger.
  - exikyut 9y ago
    Hmm. I wouldn't mind finding out how that actually works. One question though. Is this driver-dependent in such a way that it only works on newer hardware? For example on the 10+-year old machine I'm using right now, could I play with this?
    - simcop2387 9y ago
      Should work with anything that uses the open source drm drivers as I understand it.
      - exikyut 9y ago
        Interesting. Thanks.
- exikyut 9y ago
  This is totally tangential, but... that's a mildly interesting mirror, and I'm wondering if you happen to know anything about the old SPFTP.
  - mrpippy 9y ago
TwoBit 9y ago
You can synchronize to actual vsync on Windows.
- isatty 9y ago
  Very helpful, thank you.
jchb 9y ago
If you use OpenGL or Metal with triple buffering (the blog post mentions MTLView which is a Metal view) I believe this doesn't matter. You'll only be using the CVDisplayLink timer events to generate frames spaced at a even interval - which is equal to the display vsync interval, depending how you configure CVDisplayLink. Then, the actual presentation of those frames will happen in sync with the actual display vsync. See Metal Triple Buffering (https://developer.apple.com/library/content/documentation/3DDrawing/Conceptual/MTLBestPracticesGuide/TripleBuffering.html#//apple_ref/doc/uid/TP40016642-CH5-SW1 https://developer.apple.com/library/content/documentation/3D...) and MTLCommandBuffer presentDrawable. For OpenGL there are similar mechanisms, eg. NSOpenGLCPSwapInterval.
- wtallis 9y ago
  If you're aiming for the lowest latency possible, you will still want to have access to the actual display refresh timing information. But these days with variable refresh rate monitors growing in popularity, this gets pretty complicated.
  - izacus 9y ago
    Does Apple even support any of the variable refresh rate standards?
    - tvararu 9y ago
      Not sure about Apple, but on a hackintosh you can install Nvidia Geforce drivers that allow you to turn on GSync.
      - paxswill 9y ago
        Which is kinda buggy last I checked, crashing if more than one display was attached.
- seandougall 9y ago
  If you set the swap interval to 1, my recollection is that glFlush() (or -[NSOpenGLContext flushBuffers]) will block until the frame is presented. I’ve never relied on that behavior, but I’ve always wondered if the display link timer was even necessary, or if you could just use a while loop.
danra 9y ago
Wow, this is bad. If the author is reading this, is there any chance this is a regression in recent OSes? I remember a WWDC video from quite a few years ago showing exactly why it’s important to use CVDisplayLink which is linked to when your display needs new frames, showing a similar illustration to the one shown above about how otherwise you lose frames.
- lobster_johnson 9y ago
  It's not bad, it's by design: https://news.ycombinator.com/item?id=15890910 https://news.ycombinator.com/item?id=15890910.
Someone 9y ago
The documentation (https://developer.apple.com/library/content/documentation/GraphicsImaging/Conceptual/CoreVideo/CVProg_Concepts/CVProg_Concepts.html https://developer.apple.com/library/content/documentation/Gr...) doesn’t mention vertical refresh interrupts. It also says ”In the past, synchronizing your video frames with the display’s refresh rate was often a problem, especially if you also had audio. You could only make simple guesses for when to output a frame […] which didn’t take into account possible latency from user interactions, CPU loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies. […] If for some reason the processing takes longer than expected (that is, the display link’s estimate is off), the video graphics card can still drop frames or otherwise compensate for the timing error as necessary.”. So, disassembling the code wasn’t strictly necessary. Combining this with this article, it seems they try to call your program as late as possible, possibly to decrease latency between the time you have influence on what is displayed and the time it is displayed. Could that especially if you also had audio be the reason they do that, as it might make it easier to keep audio in sync with video?
- wtallis 9y ago
  Audio synchronization is a problem if the graphics pipeline has multi-frame latency, but when you're working to reduce sub-frame input latency by delaying the start of rendering, I think audio synchronization problems are by that point a minor concern that few people could notice.
- trishume 9y ago
  Yah I had noticed their docs never said it called you after vsync. It's not that it calls you as late as possible though, as my experiments show it just calls you at an arbitrary but consistent point in the frame like a timer.
- seandougall 9y ago
  The callback gets a timestamp as an argument, so audio sync is easy to compensate for if you’re doing it systematically. (Plus 1/60 second is well within human perception, when it comes to audio/video synchronization.) I think the bigger concern is latency between HID input and visual feedback, which people are more sensitive to, and which you can’t schedule in advance for.
trishume 9y ago
tfw you do a bunch of research you're proud of and write an article that reaches the HN front page, and then someone chimes in on Twitter that actually the research is flawed and only applies to multiple displays, and you have to update the post about how wrong you were. :/ I'm just sad about all the people that already read it and won't see the update so will go away with incorrect knowledge.
- pcwalton 9y ago
  Interesting. So if I'm reading this properly, CVDisplayLink is still a timer--it's just that it tries to synchronize itself to the "vblTime" if there's only one of those to synchronize to. If I'm not mistaken, then, it would seem you can do the same thing yourself by just blocking on CGLFlushDrawable() and setting up a timer on the refresh rate as soon as it returns. (I guess that's similar to what mstange suggested in the comments.) At least CVDisplayLink tweaks timer parameters in order to get the timer to fire more reliably.
- dannyobrien 9y ago
  Thank you for your correction. I appreciate your commitment to the truth: it raises not lowers my opinion of you!
stmw 9y ago
Interesting turnabout from classic MacOS, where the VSync interrupt was both synced and really important. It was the way you got lots of non-graphical processing done, too - lots of event timers & the like went in there.

110 points by jordwalke 9y ago

deleted 9y ago

[deleted]

There’s quite a few approaches, it’s hard to figure out which one is appropriate for this one. I think that enabling setNeedsDisplay and dispatching commands asynchronously might do the trick? With asynchronous dispatch I’ve actually been getting 120 FPS. But it’s very much case by case basis and I might not be understanding the use case. IIRC this project Gets to 120 when you drag your mouse https://github.com/jtbandes/metalbrot-playground https://github.com/jtbandes/metalbrot-playground

trishume 9y ago
I have a lot of avenues to test but unfortunately testing each one requires setting up a sample app with signposts and recording a bunch of runs in Instruments, so there's a bit of work to do if I want to figure out the right way. And do you have a 120fps+ monitor? If not I'm not sure what you mean by 120fps unless it's broken or not vsynced at all and painting twice per frame. If you have a proper way of waiting on vsync 120fps is just a waste of power with no benefits and you don't want that.
- adamnemecek 9y ago
  Haha welcome to gpu programming. You should hit up Warren Moore on Twitter (@warrenm), he’s a real life savior. Re: 120fps, I questioned it myself but the difference is perceivable. Idk if something is broken or not I was confused too. Re power consumption, that’s why you enable setneedsdisplay.
  - trishume 9y ago
    Maybe the reason that 120fps is perceivably better is that your draw calls don't line up with vsync, so doubling the number of frames eliminates the dropped frame effect in @jordwalke's diagram! And thanks for the Twitter recommendation, I followed him.
jvz 9y ago
> IIRC this project Gets to 120 when you drag your mouse What causes that kind of thing? I've noticed stuff like that while playing video in mpv sometimes (despite my Mac being easily fast enough to play the video) and it just baffles me.
- Doxin 9y ago
  Moving the mouse has a tendency to send various events to windows. Sometimes just the mousemotion event is enough to trigger the program to redraw. I'm not sure how cursors are implemented on osX but it might just be marking the bit under the cursor dirty, triggering a redraw.

mrpippy 9y ago

Bring back VBL tasks. http://mirror.informatimago.com/next/developer.apple.com/documentation/mac/Processes/Processes-74.html http://mirror.informatimago.com/next/developer.apple.com/doc...

revelation 9y ago
I like the Linux DRM atomic interface, they recently added native fences so you can pass in a synchronization object from the GPU for when it's done rendering and the flip can proceed and you get back a fence that is signalled once the flip is done. Basically all asynchronous now but it's still synced to the hardware flip, no timer bullshit. IIRC it's somewhat inspired by Android SurfaceFlinger.
- exikyut 9y ago
  Hmm. I wouldn't mind finding out how that actually works. One question though. Is this driver-dependent in such a way that it only works on newer hardware? For example on the 10+-year old machine I'm using right now, could I play with this?
  - simcop2387 9y ago
    Should work with anything that uses the open source drm drivers as I understand it.
    - exikyut 9y ago
      Interesting. Thanks.
exikyut 9y ago
This is totally tangential, but... that's a mildly interesting mirror, and I'm wondering if you happen to know anything about the old SPFTP.
- mrpippy 9y ago

TwoBit 9y ago

You can synchronize to actual vsync on Windows.

isatty 9y ago
Very helpful, thank you.

jchb 9y ago

If you use OpenGL or Metal with triple buffering (the blog post mentions MTLView which is a Metal view) I believe this doesn't matter. You'll only be using the CVDisplayLink timer events to generate frames spaced at a even interval - which is equal to the display vsync interval, depending how you configure CVDisplayLink. Then, the actual presentation of those frames will happen in sync with the actual display vsync. See Metal Triple Buffering (https://developer.apple.com/library/content/documentation/3DDrawing/Conceptual/MTLBestPracticesGuide/TripleBuffering.html#//apple_ref/doc/uid/TP40016642-CH5-SW1 https://developer.apple.com/library/content/documentation/3D...) and MTLCommandBuffer presentDrawable. For OpenGL there are similar mechanisms, eg. NSOpenGLCPSwapInterval.

wtallis 9y ago
If you're aiming for the lowest latency possible, you will still want to have access to the actual display refresh timing information. But these days with variable refresh rate monitors growing in popularity, this gets pretty complicated.
- izacus 9y ago
  Does Apple even support any of the variable refresh rate standards?
  - tvararu 9y ago
    Not sure about Apple, but on a hackintosh you can install Nvidia Geforce drivers that allow you to turn on GSync.
    - paxswill 9y ago
      Which is kinda buggy last I checked, crashing if more than one display was attached.
seandougall 9y ago
If you set the swap interval to 1, my recollection is that glFlush() (or -[NSOpenGLContext flushBuffers]) will block until the frame is presented. I’ve never relied on that behavior, but I’ve always wondered if the display link timer was even necessary, or if you could just use a while loop.

danra 9y ago

Wow, this is bad. If the author is reading this, is there any chance this is a regression in recent OSes? I remember a WWDC video from quite a few years ago showing exactly why it’s important to use CVDisplayLink which is linked to when your display needs new frames, showing a similar illustration to the one shown above about how otherwise you lose frames.

lobster_johnson 9y ago
It's not bad, it's by design: https://news.ycombinator.com/item?id=15890910 https://news.ycombinator.com/item?id=15890910.

Someone 9y ago

The documentation (https://developer.apple.com/library/content/documentation/GraphicsImaging/Conceptual/CoreVideo/CVProg_Concepts/CVProg_Concepts.html https://developer.apple.com/library/content/documentation/Gr...) doesn’t mention vertical refresh interrupts. It also says ”In the past, synchronizing your video frames with the display’s refresh rate was often a problem, especially if you also had audio. You could only make simple guesses for when to output a frame […] which didn’t take into account possible latency from user interactions, CPU loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies. […] If for some reason the processing takes longer than expected (that is, the display link’s estimate is off), the video graphics card can still drop frames or otherwise compensate for the timing error as necessary.”. So, disassembling the code wasn’t strictly necessary. Combining this with this article, it seems they try to call your program as late as possible, possibly to decrease latency between the time you have influence on what is displayed and the time it is displayed. Could that especially if you also had audio be the reason they do that, as it might make it easier to keep audio in sync with video?

wtallis 9y ago
Audio synchronization is a problem if the graphics pipeline has multi-frame latency, but when you're working to reduce sub-frame input latency by delaying the start of rendering, I think audio synchronization problems are by that point a minor concern that few people could notice.
trishume 9y ago
Yah I had noticed their docs never said it called you after vsync. It's not that it calls you as late as possible though, as my experiments show it just calls you at an arbitrary but consistent point in the frame like a timer.
seandougall 9y ago
The callback gets a timestamp as an argument, so audio sync is easy to compensate for if you’re doing it systematically. (Plus 1/60 second is well within human perception, when it comes to audio/video synchronization.) I think the bigger concern is latency between HID input and visual feedback, which people are more sensitive to, and which you can’t schedule in advance for.

trishume 9y ago

tfw you do a bunch of research you're proud of and write an article that reaches the HN front page, and then someone chimes in on Twitter that actually the research is flawed and only applies to multiple displays, and you have to update the post about how wrong you were. :/ I'm just sad about all the people that already read it and won't see the update so will go away with incorrect knowledge.

pcwalton 9y ago
Interesting. So if I'm reading this properly, CVDisplayLink is still a timer--it's just that it tries to synchronize itself to the "vblTime" if there's only one of those to synchronize to. If I'm not mistaken, then, it would seem you can do the same thing yourself by just blocking on CGLFlushDrawable() and setting up a timer on the refresh rate as soon as it returns. (I guess that's similar to what mstange suggested in the comments.) At least CVDisplayLink tweaks timer parameters in order to get the timer to fire more reliably.
dannyobrien 9y ago
Thank you for your correction. I appreciate your commitment to the truth: it raises not lowers my opinion of you!

stmw 9y ago

Interesting turnabout from classic MacOS, where the VSync interrupt was both synced and really important. It was the way you got lots of non-graphical processing done, too - lots of event timers & the like went in there.