11 ms·
Looking over this stuff, I get an uneasy feeling. X was a brilliant solution to problems 35 years ago, where you barely had enough memory for a full frame buffe
by raphlinus 7y ago
Looking over this stuff, I get an uneasy feeling. X was a brilliant solution to problems 35 years ago, where you barely had enough memory for a full frame buffer and the graphics model was primitive (no antialiasing or transparency). From what I've seen, Wayland is a good solution for the model maybe 15 years ago, where you're willing to accept the frame-and-a-half latency hit from having a compositor in the path.
Basically, my question is whether anybody is working on a solution for the hardware we have today, where there are hardware overlays, the GPU throughput is orders of magnitude higher than what's needed to composite a desktop, but we still care about memory bandwidth, largely because all of resolution, bit depth, and frame rate is still going up. And you really want to be able to schedule every pixel that gets presented with a high degree of timing accuracy.
I've become familiar with Windows, but feel like it's a bunch of hacks. They will promote a window to an overlay, but it's all a heuristic, and I'm pretty sure latency will stutter while it does the transition in and out. And the new flip-model stuff actually regresses the ability to do smooth window resizing.
I'm not even sure macOS is working on lowering latency. I haven't seen any mention from them of hardware overlays.
Most of the attention these days seems to be in the VR/AR space, where latency is obviously important, and there's more of a green field with respect to legacy compatibility, but I'd love to see this problem space actually solved on the desktop. Doing this right is not easy - it probably requires some form of soft real time discipline across the graphics stack (including GPU scheduling) so you can race the beam.
- Jasper_ 7y agoI have a lot of replies to this, but the back-and-forth format that HN has isn't the best for collaborative discussion. Imagine us sitting in a room with a whiteboard, rather than bickering at each other at loud volume. If you're ever in south bay area and want to meet up at a coffee shop, let's :) > Basically, my question is whether anybody is working on a solution for the hardware we have today, where there are hardware overlays, the GPU throughput is orders of magnitude higher than what's needed to composite a desktop Hardware overlays are... not that popular on desktop devices? You just stomp on the gas pedal with bandwidth and power consumption on consumer desktop gaming GPUs. Besides a couple YUV planes and a cursor overlay, you don't see that much in the way of CRTC capabilities on desktop. All of that's happening in the mobile space, because of power consumption reasons, where you basically have an SNES PPU in your CRTC with multi-plane compositing. I remain unconvinced that compositors need to add a frame and a half of latency. 16ms is a lot of time to render a frame, and composition isn't too difficult of a task. It just comes down to buffer handoff, which can be smoothed out over time. Similarly, the new flip-model isn't the thing that makes smooth resizing difficult, buffer allocation and resizing has always been a pain and racy on Windows/X11. This is one of the things that Wayland fixes -- it was actually the thing that caused Kristian to start Wayland forever ago. Presentation and timing is something I think should be possible, but I don't think it makes sense for every app to have to handle it. Multi-monitor scenarios are what make global timing difficult, and having per-monitor surfaces is just a pain to deal with from an API standpoint; if we did the work to have per-monitor timing information and split surface rendering, nobody wants to go through the work to set up two devices and present different ones at the right time. You might protest that all monitor timing is roughly the same these days, but someone hooking up their laptop to a projector, you get radically different timing. In many respects, I think graphics is in a local minima right now: it works well enough and we understand it well enough that nobody's really interested in pushing us to the next thing. VR/AR research might, but that's been relatively unimportant as a target platform, commercially.
- pcwalton 7y ago> If you're ever in south bay area and want to meet up at a coffee shop, let's :) For what it's worth, I would be very interested in chatting about this too, as this is my area right now. I'm working on what will hopefully become the Firefox OS compositor integration library, with an eye toward optimum power consumption, and am knee-deep in details of cross-platform composition. It's a very interesting topic.
- zaxcellent 7y agoThis topic is also of interest to me. I work on a compositor integration protocol called virtio-wayland that is intended to allow Wayland apps running inside of VMs to communicate with a Wayland compositor on the host, shared memory and all. It's used for the Linux apps support for Chrome OS.
- Mathnerd314 7y agoWayland's latency was improved a few years ago to about half a frame, there's probably more room for improvement, but I think just getting a 240hz monitor is better investment of resources: https://ppaalanen.blogspot.com/2015/02/weston-repaint-scheduling.html https://ppaalanen.blogspot.com/2015/02/weston-repaint-schedu... The only applications strongly sensitive to latency are video games or similar, and they take a special low-latency path through the graphics stack, exclusive full-screen mode. Monitor, graphics card, and game engine developers are all doing interesting things with scheduling there, but the main function of the OS / compositor is to get out of the way. Beyond that there's video playback, but 60fps video is rare, videos have (relatively) huge buffers of decoded video, and dropped frames are easy to overlook, so I don't think it's much of a priority for the OS. Nobody will notice the window manager adding an extra 10ms delay to the video. Businesses don't care about visual fidelity at all, the most common monitor size is still 1366 x 768. So there's no big push commercially, besides the aforementioned VR/AR.
- rasz 7y ago>The only applications strongly sensitive to latency are video games paining with a pen/tablet/mouse
- Mathnerd314 7y agoYou can still paint with 200ms of latency, it just isn't very pleasant. Atom adds an average of 50ms of latency, people still use it: https://pavelfatin.com/typing-with-pleasure/ https://pavelfatin.com/typing-with-pleasure/. As far as pointing devices, people are apparently habituated to the 50ms response time, and massaging of the lower-latency data only showed effects above 16ms: https://wp.cs.ucl.ac.uk/sebastianfriston/wp-content/uploads/sites/3/2014/09/The-Effects-of-Low-Latency-on-Pointing-and-Steering-Tasks.pdf https://wp.cs.ucl.ac.uk/sebastianfriston/wp-content/uploads/... I think artists are more interested in pen/display combinations though. There's a study of touch dragging, where the average is somewhere around 6ms: https://www.tactuallabs.com/papers/designingLowLatencyDirectTouchInputUIST12.pdf https://www.tactuallabs.com/papers/designingLowLatencyDirect... It still seems like it's mostly about having the bandwidth to drive a 240hz or higher display though. The ongoing computation required for painting is minimal. Unless Wayland or X are doing a lot of IPC on every frame there's no reason to write a new graphics stack.
- imglorp 7y agoI would like to see someone reimplement NeWS now. Hardware is way faster and we have more choices of embedded interpreter. Not that PostScript was, and is, not genius for that purpose, but there choices. Also, Don Hopkins hangs out on HN, where he's already weighed in a few times with some ideas for future work if you grok through his comments. https://en.wikipedia.org/wiki/NeWS https://en.wikipedia.org/wiki/NeWS https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=news%20don%20hopkins&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- kragen 7y agoOn the contrary, PostScript was a brilliant choice for scripting your window system at the time (≈1990). The modern crop of dynamic languages hadn't matured (JS, Guile, Python, Perl, Ruby, even Tcl), so the only real alternative would have been a Lisp, and PostScript already had a truly great graphics API — an improvement in many ways over many of its successors, including X11 and Windows GDI. Gosling certainly could have implemented something like Emacs Lisp, and it would have been easier to program in than PostScript, but it also would have been painfully slow, and he would have had to do the graphics API design work — and probably fucked it up, as the Win16 and X11 teams did. Even today, PostScript interpreters are quite a bit faster than most of the language implementations I described above, though modern JS, LuaJIT, Self, and of course HotSpot Java do beat them.
- kragen 7y agoI agree, and I've written a bunch of stuff in Dercuano about this problem, notes/real-time-windowing.html and notes/wercam-scriptable-windows.html for example. I'm interested to hear what approaches you think are likely to help with the problem. The current version of Dercuano, in the unlikely chance that you want to read my incomplete and unedited notebook, is in http://canonical.org/~kragen/dercuano-20190915.tar.gz http://canonical.org/~kragen/dercuano-20190915.tar.gz.