17 ms·

The compositor is evil

248 points by raphlinus 6y ago

jankotek 6y ago
Most of those problems could be avoided by using higher refresh rate. 120+Hz is pretty common in gaming and phones. In reality we render stuff with slow javascript. Latency after key press may not be 16ms, but 16 seconds.
- colordrops 6y ago
  Hate to go down this road, but JavaScript is anything but slow. What is slow is the layer upon layer of complexity + poor coding of many web apps. That's not an exclusive problem to JavaScript.
  - nikki93 6y ago
    There is some amount of overhead in the VM due to the dynamic nature of JavaScript values and the need for a GC that can't be avoided (even though VM implementations have had years of effort and optimization put in to address this). You can get around a lot of this by using WASM and having control over the data layout though. In my experiments some times the browser just kicks in and drops some frames in any case. :/ Things just feel a lot smoother in native apps always. Figma for example is doing a pretty good job with a C++ engine running in WASM and using WebGL for their canvas (https://www.figma.com/blog/webassembly-cut-figmas-load-time-by-3x/ https://www.figma.com/blog/webassembly-cut-figmas-load-time-...).
    - colordrops 6y ago
      It can be avoided to a degree if you understand how JavaScript memory allocation works for a particular VM, and use custom methods instead of built-ins for certain operations, to avoid the overhead of the generalized nature of the built-in.
      - nikki93 6y ago
        You might reduce it for sure, but you aren't going to avoid it entirely. The best is when it gets to a tracing JIT and all of the calls have a similar codepath. And the fact that you need to jump through hoops like this instead of just writing normal code and expect it to do only what you're saying instead of doing a bunch of other stuff too, is part of the problem. Like, what you are saying, is exactly the point: the default method of working introduces performance issues and makes it hard to reason about performance in a way that is predictable, requiring you to understand VM internals and just 'hope' that the runtime works it out. So you are essentially saying that the default path is slower. It's also hard to ensure that your objects have a predictable layout in memory (important for cache issues) other than by just using a `TypedArray` and re-inventing structs within that. The semantics of JS itself make certain optimizations hard. Even a simple VM can do arithmetic faster than no-JIT Node.js and JSC: https://godbolt.org/z/5GbhnK https://godbolt.org/z/5GbhnK (an example I've been working on lately) (I'm interested in no-JIT perf because that's what you're allowed to do in your own runtime on iOS). LuaJIT is the best at this, and even there Lua's semantics prevent certain optimizations--https://news.ycombinator.com/item?id=11327201 https://news.ycombinator.com/item?id=11327201 I think you can do a lot better with a bytecode that is more 'typed', like in SPIR-V: https://www.khronos.org/registry/spir-v/specs/1.0/SPIRV.html#OpTypeVoid https://www.khronos.org/registry/spir-v/specs/1.0/SPIRV.html...
        colordrops 6y ago
        Everything you say is correct, but I was just addressing the statement that "it can't be avoided".
        nikki93 6y ago
        For sure, I think you can mitigate it, but I would take that as being different from avoidance (simply because there are other languages / runtimes where there is clear actual avoidance). Regardless I think we have enough details in the conversation at this point to understand what each other is saying. :)
  - jcelerier 6y ago
    > Hate to go down this road, but JavaScript is anything but slow. What is slow is the layer upon layer of complexity + poor coding of many web apps. That's not an exclusive problem to JavaScript. where are those fast javascript apps ? where ? compare the super snappiness of a Telegram Desktop with molasses $every_other_chat_client_using_electron for instance. And if you are gonna say that VSCode is fast you have no idea how truly fast software feels like - if you come across try a QNX 6 demo iso and be amazed at how fast your computer answers to literally everything.
    - colordrops 6y ago
      That doesn't contradict my statement at all. The problems you speak of are due to the browser runtime, not JavaScript.
      - jcelerier 6y ago
        What makes you say that they aren't also due to the kind of programming style that JS encourages ? Also, my anecdotal experience writing some audio algorithms in pure JS is that it's ... not an order of magnitude, but not far away, from the performance of the same code written in C++ with std::vector<double>s. Likely there's some magic that could be done to improve it, but the C++ code is written naïvely and ends up extremely well optimized.
    - pjmlp 6y ago
      VSCode seems fast until one starts Notepad++ or Sublime Text. Microsoft's React Native team has charts where it shows Electron has 300x slowdown versus native applications.
  - Diggsey 6y ago
    The poor coding is one thing, but it's javascript's fault that layers of complexity inherently result in performance losses beyond those caused by leaky abstractions.
    - colordrops 6y ago
      JavaScript is just a programming language with the same constructs as any other. And it's much faster than most languages, especially dynamically typed languages. It sounds like you don't understand how a browser works. Performance issues are typically due to the browser runtime environment which includes a slow and complex DOM. JavaScript will mop up most CLI apps in performance, including python and ruby, and will get close in perf to C if written carefully.
      - rcoveson 6y ago
        > JavaScript will mop up most CLI apps in performance I'll bet half the coreutils are a minimum of 10x slower in their day-to-day use cases (i.e. small n) when re-implemented in any interpreted language like JS. Cold start overhead matters a lot for CLI programs.
      - baybal2 6y ago
        > JavaScript will mop up most CLI apps in performance, including python and ruby, and will get close in perf to C if written carefully. asm.js is what you mean? If you so, you are not better to handwriting an assembly yourself. I am kind of an involuntary webdev from time to time, and I am often the one invited to fix consequences of coding under influence, like a decision to write a GPS server with 100k+ packets per second load in Javascript. People claiming that JS can be made be as fast as C have no idea at all what they say. Even Java with hand-jacked VM is easier to scale for such load than a JS server reduced to 1k lines of vanilla JS, using all and every performance trick Nodejs provides.
        colordrops 6y ago
        No I don't mean asm.js. You can write vanilla JS that avoids memory allocation beyond initialization as well as garbage collection, and runs certain algorithms faster than typical C due to differences such as JIT optimizations or the lack of a sentinel to terminate strings, etc. My point was never to say that JS is in general as fast as C, only that a careful dev that is familiar with the VM's operation can get close for some tasks.
  - jankotek 6y ago
    Giants like Airbnb have resources to write fast and responsive apps. But it is still slow. There is no point to worry about 16ms latency, if web app refreshes 6x times just to load couple of images.
modeless 6y ago
Great article. The brute force solution to this problem is switching to high refresh rate monitors, but if you want to beat Apple II latency by brute force without fixing the underlying issues in modern software then 120 Hz probably isn't enough. You need crazy high refresh rates like 240 or 360 Hz. I do enjoy high refresh rates, but they are not great for power consumption and limit the resolution you can support.
- Daishiman 6y ago
  A high refresh monitor isn't going to do much if your synchronization primitives and scheduling capabilities aren't up to the task. A millisecond of latency in the compositor is still a millisecond no matter what output refresh rate you have.
  - wffurr 6y ago
    Latency is almost always in frames, the renderer is one frame behind the compositor which is one frame behind scanout. If each steps takes 4 msec (240 Hz) instead of 16.7 msec (60 Hz), and you're the same number of steps behind, the latency is reduced by (16.7 - 4) * nsteps, or 50 msec - 12 msec = 38 msec. Now that's assuming your system can actually keep up and produce frames and composite that fast, which is where the "brute force" comes into play.
    - maximilianroos 6y ago
      Why is that delay coupled to the monitor that's plugged into the computer? Could we ask the computer to produce graphics at 240Hz and then take every 4th frame? Then we have "average" delay in ms rather than frames. (Asking as someone who doesn't know this area well)
      - modeless 6y ago
        You could do that in theory but it would be wasteful to render frames that are never displayed. If you can modify the software, better to modify it as the article advocates for, to strip latency without the waste.
Jasper_ 6y ago
So one thing to note is that hardware overlays are really uncommon on desktop devices. NVIDIA only has a single YUV overlay and a small (used to be 64x64, now it might be 256x256?) cursor overlay. And even then, the YUV overlay might as well not exist -- it has a few restrictions that mean most video players can't or won't use it. After all, you're already driving a giant power-hungry beast like an NVIDIA card, there's no logical reason to save power and move a PPU back into the CRTC. So hardware overlays won't help the desktop case. I still think we get better compositor performance by decoupling "start of frame" / "end of frame" and the drawing API. The big thing we lack on the app side is timing information -- an application doesn't know its budget for how long it should take and when it should submit its frame, because the graphics APIs only expose vsync boundaries. If the app could take ~15ms to build a frame, submit it to the compositor, and the compositor takes the remaining ~1ms to do the composite (though much likely much much less, these are just easy numbers), we could could be made to display in the current vsync cycle. We just don't have accurate timing feedback for this though. One of my favorite gamedev tricks was used on Donkey Kong Country Returns. There, the developers polled input far above the refresh rate, and rendered Donkey Kong's 3D model at the start of the frame into an offscreen buffer, and then, as the frame was being rendered, processed input and did physics. Only at the end of the frame, did they composite Donkey Kong into the updated physics. So they in fact cut the latency to be sub-frame through clever trickery, at the expense of small inaccuracies in animation. Imagine if windows get "super late composite" privileges, where it could submit its image just in the nick of time. (Also, I should probably mention that my name is "Jasper St. Pierre". There's a few tiny inaccuracies in the history -- old-school Win95/X11 still provides process separation as the display server bounds the window's drawing to the window's clip list for the app, and Windows 2000 also had a limited compositor, known as "layered windows", where certain windows could be redirected offscreen [0], but these aren't central to your thesis) [0] https://docs.microsoft.com/en-us/windows/win32/winmsg/window-features#layered-windows https://docs.microsoft.com/en-us/windows/win32/winmsg/window...
- raphlinus 6y ago
  Apologies for getting your name wrong; it should be fixed now. I also made some other changes which I hope address the points you brought up. I didn't know about Windows 2000 layered windows, but I think the section on which systems had process separation was just confusingly worded. Counting Intel, most (by number) desktop devices have pretty sophisticated hardware layer capabilities. Sky Lake has 3 display pipes, each of which has 3 display planes and a cursor. The multiple pipes are probably mostly used for multi-monitor configurations, but it's still a decent setup. From the hints I've picked up, I believe DirectFlip was largely engineered for this hardware. There are lots of interesting latency-reducing tricks. Some of those might be good inspiration, but what I'm advocating is a general interface that lets applications reliably get good performance. Thanks for the discussion!
boulos 6y ago
I was surprised that the tile-based updates from VNC, Teradici, PCoIP etc. wasn't in the related work. In particular, a challenge with "send < smaller > updates" rather than whole frames is how you decide to deal with massive updates where you'd end up causing more work / traffic than the whole frame approach. This post is focused on local compositing, but I think the same arguments apply as with the networked case: too many updates is actually worse than vsync, but the usual case of "just ship deltas" is amazing (for remote display you get the bandwidth and latency win, here you'd get latency/power/whatever). I think a low-level "updates" API would make sense for some sophisticated applications, but I'm not convinced that this quote holds: > I think this design can work effectively without much changing the compositor API, but if it really works as tiles under the hood, that opens up an intriguing possibility: the interface to applications might be redefined at a lower level to work with tiles. Seems like if you can show Chrome and VLC both working fine, that'd be a great proof of concept!
- taeric 6y ago
  I feel like this almost always comes back to a reinvention of I-P-B Frames. I am curious if there is a solid data structure that could be used to collect updates together in such a way that updates could be coalesced into a collection of other updates efficiently.
skunkworker 6y ago
It seems like we are coming full circle in try to reduce latency in all parts of the chain. With the next gen gaming consoles supporting VRR and 120hz, recently announced Nvidia Reflex low latency modes, 360hz monitors, and things like the Apple Pencil 2 reducing the latency from pen input to on screen down to 9ms we are working our way back to something close to what we used to have. But I feel like we have a number of years to go until we can really get back to where we used to be with vintage gaming consoles and CRT displays.
- perl4ever 6y ago
  I think that with the end of Moore's law, there is a gradual unwinding of software bloat and inefficiency. Once the easy hardware gains die away, it is worth studying how software can be better, but it's a cultural transition that takes time.
  - stephc_int13 6y ago
    I also think that the end of the so called Moore Law will create the necessary incentives to build better software.
    - ginko 6y ago
      What's with the "so called"? Moore's Law is a pretty well-established term.
      - stephc_int13 6y ago
        It is not a law, it is merely an observation.
  - jhayward 6y ago
    It is also the case that in some/many areas software has improved performance by several multiples of the improvement in hardware performance over the last 20 years. So it makes sense that this would invigorate software performance investment.
amelius 6y ago
I'm afraid that you'd need a realtime OS to implement a compositor with the suggested properties.
- zokier 6y ago
  I'm in the opinion that real-time desktop is already overdue. I can understand that in the olden days the perf benefit from dropping latency guarantees were worth it (classic latency vs throughput optimization), especially on servers where current operating systems draw their lineage (NT and UNIX). But these days with beefy, even overpowered CPUs, and 25 years of OS/real-time research later I would imagine that we would be in a place that the tradeoff would look different. Of course just slapping RT kernel in a modern system alone would not do much good. The whole system needs to be thought with RT in mind, starting from kernel and drivers through toolkits/frameworks and services up to the applications themselves. But ultimately the APIs for application developers should be nudging people to do the right things (whatever those would be)
  - cycloptic 6y ago
    We're probably already halfway there with modern apps, if you consider that any program with hardware-accelerated graphics is already performing rendering as a stateless operation in a separate thread pool running on an entirely different processor.
  - flohofwoe 6y ago
    You need to go down to the hardware even, a realtime kernel running on mainstream CPUs and GPUs might not be enough (I wonder if a realtime system can even be built with current mainstream CPUs and GPUs): The video systems in 8-bit home computers only worked because the entire system provided "hard realtime" guarantees. CPU instructions always were the same number of cycles and memory access happened at exactly defined clock cycles within an instruction. Interrupt timing was completely predictable down to the clock cycle, etc etc. Modern computers get most of their performace because they dumped those assumptions and as a result made timings entirely unpredictable (inside the CPU via caches, pipelining, branch prediction, etc etc...), and between system components (e.g. CPU and GPU don't run cycle-locked with each other).
monocasa 6y ago
I've always really liked the Nintendo DS's GPU, and thought something with a similar architecture but scaled up would make a lot sense in a few places in the stack. Unlike most GPUs, you submitted a static scene, and then the rendering happened pretty close to lock step with the scan out to the LCD. There was a buffer with a handful of lines stored, but it was pretty close; the latency was pretty much unbeatable. In a lot fo ways it was really a 3D extension to their earlier 2D (NES, SNES, GB, GBA) GPU designs. Something like that sitting where the scan out engines exist today (with their multiple planes the composite today) would be absolutely killer if you could do hardware software co design to take advantage of it. I've also thought that something like that would be great in VR/AR.
- aspaceman 6y ago
  You've just inspired a dive into documentation. I do GPU architecture research and reading this was exciting. Thanks.
  - monocasa 6y ago
    Awesome! I'd really suggest Martin Korth's fantastic documentation in that case. In a lot of ways it was better than even the official docs given to developers. https://problemkaputt.de/gbatek.htm https://problemkaputt.de/gbatek.htm
  - deergomoo 6y ago
    https://www.copetti.org/projects/consoles/nintendo-ds/ https://www.copetti.org/projects/consoles/nintendo-ds/ This guy has articles about most of the major consoles from the last 30 years, I find them fascinating to read even though much of it goes over my head.
- giantrobot 6y ago
  Buffers see extensive use in modern shaders. A frame will get rasterized several times for different shaders before those buffers are all composited into an output buffer. Several types of things can only be known if you've got the full scene rasterized or at least more than a few lines. If you've only got fixed function shaders and a relatively simple rasterizer the raster-to-display works fine. The DS also only has a single contributor to the output. A game on a PS4 or a PC is just one of several contributors to a compositing window/display manager which is actually doing final composition to the output device(s).
upofadown 6y ago
As someone who uses a tiling window manager it has been a long time since I have actually used a compositor. I have to ask if the cool window effects are actually worth the bother? I just can't see anyone actually caring about stuff like that in 20 years.
- iso8859-1 6y ago
  On Android, you have Picture-In-Picture video support now. How do you imagine it working without a compositor? Do you think nobody cares about that feature?
  - toast0 6y ago
    Picture in picture videos used to be done with chromakey. Playing a video in one window on top of another window isn't some new thing compositing provides. You could do it in Mac classic or Windows 3.1 or BeOS.
- CmdrKrool 6y ago
  I have set up a keyboard shortcut to toggle my compositor on and off and tbh I find it tends to remain off most of the time, since I don't care for 'cool window effects' either. Most of the especially visual applications I use - eg. video players, games - tend to handle themselves in this respect, taking their steps for vsync as necessary. On the rare occasion that I'm offended by a bit of tearing I'll just activate the shortcut and then no problem. I just worry that with the badmouthing of X11 that goes around, one day this control - the ability to run without a compositor, if I want to - might be taken away from me.
  - cycloptic 6y ago
    The X server code isn't going anywhere. Somehow I feel like it will be around forever.
    - wmf 6y ago
      I predict there will be a systemd-style debate about removing X11 by 2030 and it will be completely gone by 2040 (since someone mentioned 20 years). There won't be a single non-composited OS available at all and people born after 2000 won't really remember anything else.
nayuki 6y ago
Variable refresh rate technologies like FreeSync and G-Sync are used in some video games. Wouldn't this improve the latency of desktop compositing too? After all, the idea is that the GPU sends out a frame to the monitor once it's finished, not once a periodic interval is reached.
- raphlinus 6y ago
  We had a discussion on this in #gpu on our Zulip. The short answer is that if the screen is updating at 60fps, then FreeSync and its cousins are effectively a no-op. If the display is mostly idle, then there is potentially latency benefit in withholding scanout until triggered by a keypress. Older revisions of FreeSync have a "minimum frame rate" which actually doesn't buy you much; 40Hz is common: https://www.amd.com/en/products/freesync-monitors https://www.amd.com/en/products/freesync-monitors
  - amluto 6y ago
    I would expect that VRR helps if you try to make your compositor only start compositing as late as possible while still finishing on time. With VRR, one could delay the next frame slightly if the deadline gets missed. This would work even when rendering at 60Hz.
    - medlyyy 6y ago
      Problem is that causes jitter which is extremely noticeable even in small amounts. You can't have smooth animations if the frame timing is constantly varying.
PaulDavisThe1st 6y ago
It is fascinating to me, as someone who has been doing real time audio work for 20 years, to see how two fundamental processes have been at work during that time: * audio developers recognizing the parallels between the way audio and video hardware works, and taking as many lessons from the video world as possible. The result is that audio data flow (ignoring the monstrosity of USB audio) is now generally much better than it used to be, and in many cases is conceptually very clean. In addition, less and less processing is done by audio hardware, and more and more by the CPU. * video developers not understanding much about audio, and failing to notice the parallels between the two data handling processes. Rather than take any lessons from the audio side of things, stuff has just become more and more complex and more and more distant from what is actually happening in hardware. In addition, more and more processing is done by video hardware, and less and less by the CPU. In both cases, there is hardware which requires data periodically (on the order of a few msec). There are similar requirements to allow multiple applications to contribute what is visible/audible to the user. There are similar needs for processing pipelines between hardware and user space code. (one important difference: if you don't provide a new frame of video data, most humans will not notice the result; if you don't provide a new frame of audio fata, every human will hear the result). I feel as if both these worlds would really benefit from somehow having a full exchange about the high and low level aspects of the problems they both face, how they have been solved to date, how might be solved in the future, and how the two are both very much alike, and really quite different.
- modeless 6y ago
  > [video] stuff has just become more and more complex and more and more distant from what is actually happening in hardware There's been a huge push recently to remove layers from graphics APIs and expose hardware more directly, in the form of Vulkan, DX12, and Metal. The problem that this has uncovered is that graphics hardware is not only very complex but quite varied, so exposing "what is actually happening in hardware" is incompatible with portability and simplicity. As a result you get APIs that are either not portable or hard to use. > more and more processing is done by video hardware, and less and less by the CPU. Unfortunately this will remain a reality because it comes with a 20-100x performance advantage.
guhcampos 6y ago
"As the experience of realtime audio illuminates, it’s hard enough scheduling CPU tasks to reliably complete before timing deadlines, and it seems like GPUs are even harder to schedule with timing guarantees." Well, I suppose that's way more feasible in RISC CPUs. Damn I hate when Apple is right about something.
- monocasa 6y ago
  RISC as a concept doesn't really help you with making timing guarantees.
  - est31 6y ago
    Yeah as of today, the instruction language used isn't really relevant anyway. You still have to figure out the same problems as a chip designer.
- danhor 6y ago
  More modern/better arm cores and common environments (starting at approx. Cortex-M3 microcontrollers, where flash instruction fetching depends on page alignment) are also pretty unpredictable. In fact, most advanced pipeline designs, no matter the ISA, have very hard-to-predict instruction timings. Couple this with dynamic frequency scaling and the instruction timings are irrelevant.
mncharity 6y ago
A year or few back, I did camera-passthru XR in VR HMDs, on a DIY stack with electron.js (which is chromium) as compositor. So a couple of thoughts... The full-screen electron window apparently displaced the window system compositor. So "compositing" specialized compositors can be useful. Chromium special cased video, making it utterly trivial to do comfortable camera passthru, while a year+ later, mainstream stacks were still considering that hard to impossible. So having compositor architecture match the app is valuable. That special case would collapse if the CPU needed to touch the camera frames for analysis. So carefully managing data movement between GPU and CPU is important. That's one design objective of google's mediapipe for instance. Perhaps future compositors should permit similar pipeline specifications? My XR focus was software-dev "office" work, not games. So no game-dev "oh, the horror! an immersion-shattering visual artifact occurred!" - people don't think "Excel, it's ghastly... it leaves me aware I'm sitting in an office!". Similarly, with user balance based on nice video passthru, the rest of the rendering could be slow and jittery. Game-dev common wisdom was "VR means 90 fps, no jitters, no artifacts, or user-sick fail - we're GPU-tech limited", and I was "meh, 30/20/10 fps, whatever" on laptop integrated graphics. So... Games are hard - don't assume their constraints are yours without analysis. And different aspects of the rendered environment can have very different constraints. My fuzzy recollection was chromium could be persuaded to do 120 Hz, though I didn't have a monitor to try that. Or higher? - I fuzzily recall some variable to uncap the frame rate. I've used linux evdev (input pipeline step between kernel and libinput) directly from an electron renderer process. Latency wasn't the motivation, so I didn't measure it. But that might save an extra ms or few. At 120 Hz, that might mean whatever ms HID to OS, 0-8 ms wait for the next frame, 8 ms processing and render, plus whatever ms to light. On electron.js. New interface tech like speech and gesture recognition may start guessing at what it's hearing/seeing many ms before it provides it's final best guess at what happened. Here low latency responsiveness is perhaps more about app system architecture than pipeline tuning. With app state and UI supporting iterative speculative execution. Eye tracking changes things. "Don't bother rendering anything for the next 50 ms, the user is saccading and thus blind." "After this saccade, the eye will be pointing at xy, so only that region will need a full resolution render" (foveated rendering). Patents... but eventually.
emmanueloga_ 6y ago
> When pressing a key in, say, a text editor, the app would prepare and render a minimal damage region Is this similar to the "dirty rectangles" technique? [1] It seems difficult to implement some sort of scene system (for either an app, game or general GUI) that given a single key press can determine a minimum bounding box of changes on the screen no matter what the scene is, and is able to render just that bounding box. If the single key occurs on a text area, potentially the whole text area could be "damaged" right? Edit: I was looking at this with the "Rendering" tab of chrome dev tools, enabling "Paint flashing" and "Layout Shift Regions", it seems like the text area is its own layer and the space is partitioned pretty cleverly on things like paragraphs and lines, but from time to time the whole text area just flashes, which tells me the algorithm sometimes is not sure what is dirty and just repaints the whole thing, but not always. 1: https://wiki.c2.com/?DirtyRectangles https://wiki.c2.com/?DirtyRectangles
- munificent 6y ago
  > Is this similar to the "dirty rectangles" technique? Yes. > It seems difficult to implement some sort of scene system (for either an app, game or general GUI) that given a single key press can determine a minimum bounding box of changes on the screen no matter what the scene is, and is able to render just that bounding box. I don't think it's that bad. In the days before hardware graphics, basically every 2D sprite engine used in every computer game you liked had to do this logic. Text editors are a harder case in some ways because of line wrapping, but the editor does need to figure out where everything goes spatially in order to render, so extending that to tell which things have not moved is, I think, not that difficult.
crazyloglad 6y ago
A few pointers: The first bit is that practically all that is argued in for here, is in Android - and has been so for a long time (~2013 last I worked on the topic in that context) and it is still not enough - they will go (and are going) higher refresh rate. Even at the time there were crazy things like tuning CPU governor to wake up when touch screen detects incoming 'presence'(before you can get an accurate measurement of where and how hard the finger hit) - it's not like you are going to change your muscle intention mid flight. Input events were specifically tagged so that the memory would be re-used rather than GCed and reallocated. A sunny frame could go from motion to photon in 30ms. Then something happens and the next takes 110ms. That something is often garbage collectors and other runtime systems that think it is safe to do things as there is no shared systemic synchronization mechanism around for these things. The second is that this is -ultimately- a systemic issue and the judge, jury and executioner is the user's own perception. That's what you are optimising for. Treating it as a graphics only thing is not putting the finger on something, it is picking your nose. The input needs to be in on it, audio needs to be in on it, and the producer needs to be cooperative. Incidentally, the guitar hero style games and anything VR are good qualitative evaluation targets but with brinelling like incremental loads. (Larger scale) 1. communicate deadlines to client so the renderer can judge if it is even worth doing anything. 2. communicate presentation time to client so animations and video frames arrive at the right time. 3. have a compositor scheduler than will optimize for throughput, latency, minimizing jitter or energy consumption. 4. inform the scheduler about the task that is in focus, and bias client unlock and event routing to distinguish between input focus and the thundering herd. 5. type annotate client resources so the scheduler knows what they are for. 6. coalesce / resample input and resize events. 7. align client audio buffers to video buffers. 8. clients with state store/restore capabilities can rollback, inject and fastforward (http://filthypants.blogspot.com/2016/03/using-rollback-to-hide-latency-in.html http://filthypants.blogspot.com/2016/03/using-rollback-to-hi...) There is a bunch of more eccentric big stuff after that as well as the well known quality of life improvements (beyond the obvious gamedev lessons like don't just malloc in your renderloop and minimize the amount of syscall jitter in the entire pipeline), but baby steps. Oh, and all of the above++ is already in Arcan and yet it is still nowhere near good enough.
- baybal2 6y ago
Androider 6y ago
There's some hardware fixes that are becoming available. Variable refresh rate (VRR / G-Sync / FreeSync / Adaptive Sync) and high-refresh rate hardware is now starting to gain traction on the high-end, usually with both technologies combined. Apple's ProMotion is often touted as a "120Hz" feature, but it's also a variable refresh in that it can support much lower framerates. Without a static v-sync interval, you can display the frames as they become available, for both perfect tear-free visuals and lower latency. It's quite likely we'll have that on Mac's as well. I recently got a 144Hz FreeSync (G-Sync compatible) "business" monitor, so this tech is starting to filter down from the gamer side. It works great with a compositor on Linux! Ultra smooth mouse and input response at 144Hz, and completely tear-free. I would highly recommend it for developers as well.
- badsectoracula 6y ago
  > It works great with a compositor on Linux! Ultra smooth mouse and input response at 144Hz, and completely tear-free. I would highly recommend it for developers as well. I got a 165Hz monitor and while it is indeed smoother than a 60Hz monitor with a vsync'd compositor, it still isn't as smooth as a 60Hz monitor without a compositor.
deleted 6y ago
[deleted]
tomxor 6y ago
> [...] the worst that can happen is just jank, which people are used to, rather than visual artifacts. This is an important point. There are many solutions out there (beyond compositing) that add incredible complexity along with other costs in order to solve a perceived problem by it's proponents... not uncommonly those same people pushing the solution are incapable of fairly judging the cost benefit - sometimes the solutions are just not worth the cost. There is currently an issue in chromium compositing with checkerboarding (blanking whole regions of the screen with white) when scrolling due to an optimization released to reduce jank: https://bugs.chromium.org/p/chromium/issues/detail?id=1062662 https://bugs.chromium.org/p/chromium/issues/detail?id=106266... You will notice this in recent releases of chromium when you scroll any non trivial page fast enough... or a simple page very fast by yanking the scroll bar... unless you have an extremely fast computer and graphics card. When reading through that thread you will notice how defensive they are, it's a complex and no doubt incredible bit of coding they have done - unfortunately the side-effect is worse than the original problem that most users don't even notice. In the long run I think time will prove this problem and it's solutions are purely transient... In the same way font anti-aliasing is becoming obsolete with DPIs higher than we can perceive, solving jank isn't necessary if you cannot perceive it (i.e using frame rates > 60 Hz)... and the original problem really isn't as bad as all the people attempting to solve it think it is anyway.
- Firadeoclus 6y ago
  > In the same way font anti-aliasing is becoming obsolete with DPIs higher than we can perceive Screens might exceed acuity of most users but not hyperacuity. Even on a 400ppi phone screen at typical viewing distances it is possible to tell whether a slanted line is anti-aliased or not. Font anti-aliasing is not becoming obsolete any time soon.
  - pekim 6y ago
    I agree, anti-aliasing is certainly still required at 400dpi. Although subpixel anti-aliasing is not.
roca 6y ago
The beam-racing compositor is a cool idea. It seems to me, however, that it could create visible jitter for applications that use traditional vsync-driven frame timing --- e.g. an animation of vertical motion would see a one-frame latency change when the damage rectangle crosses some vertical threshold.
- raphlinus 6y ago
  If you're animating (or doing smooth window resizing), then your "present" request includes a target frame sequence number. The way I analyze the text editor case is that it's something of a special case of triple-buffering, but instead of spamming frames as fast as possible at the swapchain, you do multiple revisions for a single target sequence number. The last revision to make it ahead of the beam wins. If you have a smooth clock animation in addition to some latency-critical input sensitive elements, then the clock hands would be in the same position for each revision, only the other elements would change. If done carefully, this would give you smoother animations at considerably lower power consumption than traditional triple buffering. It's not the way present APIs are designed today, though, one of my many sadnesses. As I hinted, this is a good way to synchronize window resize as well. The window manager tells the application, "as of frame number 98765, your new size is 2345x1234." It also sends the window frame and shadows etc with the same frame number. These are treated as a transaction; both have to arrive before the deadline for the transaction to go through. This is not rocket science, but requires careful attention to detail.
LargoLasskhyfv 6y ago
I don't get this obsession with compositors. I just switch them off. Has no benefit for me. If I'm so inclined I can move windows with video running in Full-HD without any tearing across several screens like mad. When I'm in a terminal window of some sort I don't care for the real transparency, because it is unergonomic. When I'm in some other application the same applies. Can someone enlighten me about the real benefits? Seriously.
- sjy 6y ago
  The app thumbnails in macOS’s Mission Control [1] and GNOME’s Activities overview [2] provide a nice way of managing graphical windows and workspaces which requires a compositor. Animated window management in general is a UI affordance that I value. I’m not sure if it’s a “real benefit,” but consistently getting this right is one thing that attracts users to macOS and iOS. [1] https://support.apple.com/en-us/HT204100 https://support.apple.com/en-us/HT204100 [2] https://help.gnome.org/users/gnome-help/stable/shell-introduction.html.en#activities https://help.gnome.org/users/gnome-help/stable/shell-introdu...
  - LargoLasskhyfv 6y ago
    I know them. Maybe I'm just an old fart and "set in my ways", but it's not even "nice to have" for me when it compromises the general haptics of the environment, as in dragging it down with sluggishness.
andrekandre 6y ago
without a compositor, how does one implement a feature like expose/mission control effeciently?

248 points by raphlinus 6y ago

Most of those problems could be avoided by using higher refresh rate. 120+Hz is pretty common in gaming and phones. In reality we render stuff with slow javascript. Latency after key press may not be 16ms, but 16 seconds.

colordrops 6y ago
Hate to go down this road, but JavaScript is anything but slow. What is slow is the layer upon layer of complexity + poor coding of many web apps. That's not an exclusive problem to JavaScript.
- nikki93 6y ago
  There is some amount of overhead in the VM due to the dynamic nature of JavaScript values and the need for a GC that can't be avoided (even though VM implementations have had years of effort and optimization put in to address this). You can get around a lot of this by using WASM and having control over the data layout though. In my experiments some times the browser just kicks in and drops some frames in any case. :/ Things just feel a lot smoother in native apps always. Figma for example is doing a pretty good job with a C++ engine running in WASM and using WebGL for their canvas (https://www.figma.com/blog/webassembly-cut-figmas-load-time-by-3x/ https://www.figma.com/blog/webassembly-cut-figmas-load-time-...).
  - colordrops 6y ago
    It can be avoided to a degree if you understand how JavaScript memory allocation works for a particular VM, and use custom methods instead of built-ins for certain operations, to avoid the overhead of the generalized nature of the built-in.
    - nikki93 6y ago
      You might reduce it for sure, but you aren't going to avoid it entirely. The best is when it gets to a tracing JIT and all of the calls have a similar codepath. And the fact that you need to jump through hoops like this instead of just writing normal code and expect it to do only what you're saying instead of doing a bunch of other stuff too, is part of the problem. Like, what you are saying, is exactly the point: the default method of working introduces performance issues and makes it hard to reason about performance in a way that is predictable, requiring you to understand VM internals and just 'hope' that the runtime works it out. So you are essentially saying that the default path is slower. It's also hard to ensure that your objects have a predictable layout in memory (important for cache issues) other than by just using a `TypedArray` and re-inventing structs within that. The semantics of JS itself make certain optimizations hard. Even a simple VM can do arithmetic faster than no-JIT Node.js and JSC: https://godbolt.org/z/5GbhnK https://godbolt.org/z/5GbhnK (an example I've been working on lately) (I'm interested in no-JIT perf because that's what you're allowed to do in your own runtime on iOS). LuaJIT is the best at this, and even there Lua's semantics prevent certain optimizations--https://news.ycombinator.com/item?id=11327201 https://news.ycombinator.com/item?id=11327201 I think you can do a lot better with a bytecode that is more 'typed', like in SPIR-V: https://www.khronos.org/registry/spir-v/specs/1.0/SPIRV.html#OpTypeVoid https://www.khronos.org/registry/spir-v/specs/1.0/SPIRV.html...
      - colordrops 6y ago
        Everything you say is correct, but I was just addressing the statement that "it can't be avoided".
        nikki93 6y ago
        For sure, I think you can mitigate it, but I would take that as being different from avoidance (simply because there are other languages / runtimes where there is clear actual avoidance). Regardless I think we have enough details in the conversation at this point to understand what each other is saying. :)
- jcelerier 6y ago
  > Hate to go down this road, but JavaScript is anything but slow. What is slow is the layer upon layer of complexity + poor coding of many web apps. That's not an exclusive problem to JavaScript. where are those fast javascript apps ? where ? compare the super snappiness of a Telegram Desktop with molasses $every_other_chat_client_using_electron for instance. And if you are gonna say that VSCode is fast you have no idea how truly fast software feels like - if you come across try a QNX 6 demo iso and be amazed at how fast your computer answers to literally everything.
  - colordrops 6y ago
    That doesn't contradict my statement at all. The problems you speak of are due to the browser runtime, not JavaScript.
    - jcelerier 6y ago
      What makes you say that they aren't also due to the kind of programming style that JS encourages ? Also, my anecdotal experience writing some audio algorithms in pure JS is that it's ... not an order of magnitude, but not far away, from the performance of the same code written in C++ with std::vector<double>s. Likely there's some magic that could be done to improve it, but the C++ code is written naïvely and ends up extremely well optimized.
  - pjmlp 6y ago
    VSCode seems fast until one starts Notepad++ or Sublime Text. Microsoft's React Native team has charts where it shows Electron has 300x slowdown versus native applications.
- Diggsey 6y ago
  The poor coding is one thing, but it's javascript's fault that layers of complexity inherently result in performance losses beyond those caused by leaky abstractions.
  - colordrops 6y ago
    JavaScript is just a programming language with the same constructs as any other. And it's much faster than most languages, especially dynamically typed languages. It sounds like you don't understand how a browser works. Performance issues are typically due to the browser runtime environment which includes a slow and complex DOM. JavaScript will mop up most CLI apps in performance, including python and ruby, and will get close in perf to C if written carefully.
    - rcoveson 6y ago
      > JavaScript will mop up most CLI apps in performance I'll bet half the coreutils are a minimum of 10x slower in their day-to-day use cases (i.e. small n) when re-implemented in any interpreted language like JS. Cold start overhead matters a lot for CLI programs.
    - baybal2 6y ago
      > JavaScript will mop up most CLI apps in performance, including python and ruby, and will get close in perf to C if written carefully. asm.js is what you mean? If you so, you are not better to handwriting an assembly yourself. I am kind of an involuntary webdev from time to time, and I am often the one invited to fix consequences of coding under influence, like a decision to write a GPS server with 100k+ packets per second load in Javascript. People claiming that JS can be made be as fast as C have no idea at all what they say. Even Java with hand-jacked VM is easier to scale for such load than a JS server reduced to 1k lines of vanilla JS, using all and every performance trick Nodejs provides.
      - colordrops 6y ago
        No I don't mean asm.js. You can write vanilla JS that avoids memory allocation beyond initialization as well as garbage collection, and runs certain algorithms faster than typical C due to differences such as JIT optimizations or the lack of a sentinel to terminate strings, etc. My point was never to say that JS is in general as fast as C, only that a careful dev that is familiar with the VM's operation can get close for some tasks.
- jankotek 6y ago
  Giants like Airbnb have resources to write fast and responsive apps. But it is still slow. There is no point to worry about 16ms latency, if web app refreshes 6x times just to load couple of images.

modeless 6y ago

Great article. The brute force solution to this problem is switching to high refresh rate monitors, but if you want to beat Apple II latency by brute force without fixing the underlying issues in modern software then 120 Hz probably isn't enough. You need crazy high refresh rates like 240 or 360 Hz. I do enjoy high refresh rates, but they are not great for power consumption and limit the resolution you can support.

Daishiman 6y ago
A high refresh monitor isn't going to do much if your synchronization primitives and scheduling capabilities aren't up to the task. A millisecond of latency in the compositor is still a millisecond no matter what output refresh rate you have.
- wffurr 6y ago
  Latency is almost always in frames, the renderer is one frame behind the compositor which is one frame behind scanout. If each steps takes 4 msec (240 Hz) instead of 16.7 msec (60 Hz), and you're the same number of steps behind, the latency is reduced by (16.7 - 4) * nsteps, or 50 msec - 12 msec = 38 msec. Now that's assuming your system can actually keep up and produce frames and composite that fast, which is where the "brute force" comes into play.
  - maximilianroos 6y ago
    Why is that delay coupled to the monitor that's plugged into the computer? Could we ask the computer to produce graphics at 240Hz and then take every 4th frame? Then we have "average" delay in ms rather than frames. (Asking as someone who doesn't know this area well)
    - modeless 6y ago
      You could do that in theory but it would be wasteful to render frames that are never displayed. If you can modify the software, better to modify it as the article advocates for, to strip latency without the waste.

Jasper_ 6y ago

So one thing to note is that hardware overlays are really uncommon on desktop devices. NVIDIA only has a single YUV overlay and a small (used to be 64x64, now it might be 256x256?) cursor overlay. And even then, the YUV overlay might as well not exist -- it has a few restrictions that mean most video players can't or won't use it. After all, you're already driving a giant power-hungry beast like an NVIDIA card, there's no logical reason to save power and move a PPU back into the CRTC. So hardware overlays won't help the desktop case. I still think we get better compositor performance by decoupling "start of frame" / "end of frame" and the drawing API. The big thing we lack on the app side is timing information -- an application doesn't know its budget for how long it should take and when it should submit its frame, because the graphics APIs only expose vsync boundaries. If the app could take ~15ms to build a frame, submit it to the compositor, and the compositor takes the remaining ~1ms to do the composite (though much likely much much less, these are just easy numbers), we could could be made to display in the current vsync cycle. We just don't have accurate timing feedback for this though. One of my favorite gamedev tricks was used on Donkey Kong Country Returns. There, the developers polled input far above the refresh rate, and rendered Donkey Kong's 3D model at the start of the frame into an offscreen buffer, and then, as the frame was being rendered, processed input and did physics. Only at the end of the frame, did they composite Donkey Kong into the updated physics. So they in fact cut the latency to be sub-frame through clever trickery, at the expense of small inaccuracies in animation. Imagine if windows get "super late composite" privileges, where it could submit its image just in the nick of time. (Also, I should probably mention that my name is "Jasper St. Pierre". There's a few tiny inaccuracies in the history -- old-school Win95/X11 still provides process separation as the display server bounds the window's drawing to the window's clip list for the app, and Windows 2000 also had a limited compositor, known as "layered windows", where certain windows could be redirected offscreen [0], but these aren't central to your thesis) [0] https://docs.microsoft.com/en-us/windows/win32/winmsg/window-features#layered-windows https://docs.microsoft.com/en-us/windows/win32/winmsg/window...

raphlinus 6y ago
Apologies for getting your name wrong; it should be fixed now. I also made some other changes which I hope address the points you brought up. I didn't know about Windows 2000 layered windows, but I think the section on which systems had process separation was just confusingly worded. Counting Intel, most (by number) desktop devices have pretty sophisticated hardware layer capabilities. Sky Lake has 3 display pipes, each of which has 3 display planes and a cursor. The multiple pipes are probably mostly used for multi-monitor configurations, but it's still a decent setup. From the hints I've picked up, I believe DirectFlip was largely engineered for this hardware. There are lots of interesting latency-reducing tricks. Some of those might be good inspiration, but what I'm advocating is a general interface that lets applications reliably get good performance. Thanks for the discussion!

boulos 6y ago

I was surprised that the tile-based updates from VNC, Teradici, PCoIP etc. wasn't in the related work. In particular, a challenge with "send < smaller > updates" rather than whole frames is how you decide to deal with massive updates where you'd end up causing more work / traffic than the whole frame approach. This post is focused on local compositing, but I think the same arguments apply as with the networked case: too many updates is actually worse than vsync, but the usual case of "just ship deltas" is amazing (for remote display you get the bandwidth and latency win, here you'd get latency/power/whatever). I think a low-level "updates" API would make sense for some sophisticated applications, but I'm not convinced that this quote holds: > I think this design can work effectively without much changing the compositor API, but if it really works as tiles under the hood, that opens up an intriguing possibility: the interface to applications might be redefined at a lower level to work with tiles. Seems like if you can show Chrome and VLC both working fine, that'd be a great proof of concept!

taeric 6y ago
I feel like this almost always comes back to a reinvention of I-P-B Frames. I am curious if there is a solid data structure that could be used to collect updates together in such a way that updates could be coalesced into a collection of other updates efficiently.

skunkworker 6y ago

It seems like we are coming full circle in try to reduce latency in all parts of the chain. With the next gen gaming consoles supporting VRR and 120hz, recently announced Nvidia Reflex low latency modes, 360hz monitors, and things like the Apple Pencil 2 reducing the latency from pen input to on screen down to 9ms we are working our way back to something close to what we used to have. But I feel like we have a number of years to go until we can really get back to where we used to be with vintage gaming consoles and CRT displays.

perl4ever 6y ago
I think that with the end of Moore's law, there is a gradual unwinding of software bloat and inefficiency. Once the easy hardware gains die away, it is worth studying how software can be better, but it's a cultural transition that takes time.
- stephc_int13 6y ago
  I also think that the end of the so called Moore Law will create the necessary incentives to build better software.
  - ginko 6y ago
    What's with the "so called"? Moore's Law is a pretty well-established term.
    - stephc_int13 6y ago
      It is not a law, it is merely an observation.
- jhayward 6y ago
  It is also the case that in some/many areas software has improved performance by several multiples of the improvement in hardware performance over the last 20 years. So it makes sense that this would invigorate software performance investment.

amelius 6y ago

I'm afraid that you'd need a realtime OS to implement a compositor with the suggested properties.

zokier 6y ago
I'm in the opinion that real-time desktop is already overdue. I can understand that in the olden days the perf benefit from dropping latency guarantees were worth it (classic latency vs throughput optimization), especially on servers where current operating systems draw their lineage (NT and UNIX). But these days with beefy, even overpowered CPUs, and 25 years of OS/real-time research later I would imagine that we would be in a place that the tradeoff would look different. Of course just slapping RT kernel in a modern system alone would not do much good. The whole system needs to be thought with RT in mind, starting from kernel and drivers through toolkits/frameworks and services up to the applications themselves. But ultimately the APIs for application developers should be nudging people to do the right things (whatever those would be)
- cycloptic 6y ago
  We're probably already halfway there with modern apps, if you consider that any program with hardware-accelerated graphics is already performing rendering as a stateless operation in a separate thread pool running on an entirely different processor.
- flohofwoe 6y ago
  You need to go down to the hardware even, a realtime kernel running on mainstream CPUs and GPUs might not be enough (I wonder if a realtime system can even be built with current mainstream CPUs and GPUs): The video systems in 8-bit home computers only worked because the entire system provided "hard realtime" guarantees. CPU instructions always were the same number of cycles and memory access happened at exactly defined clock cycles within an instruction. Interrupt timing was completely predictable down to the clock cycle, etc etc. Modern computers get most of their performace because they dumped those assumptions and as a result made timings entirely unpredictable (inside the CPU via caches, pipelining, branch prediction, etc etc...), and between system components (e.g. CPU and GPU don't run cycle-locked with each other).

monocasa 6y ago

I've always really liked the Nintendo DS's GPU, and thought something with a similar architecture but scaled up would make a lot sense in a few places in the stack. Unlike most GPUs, you submitted a static scene, and then the rendering happened pretty close to lock step with the scan out to the LCD. There was a buffer with a handful of lines stored, but it was pretty close; the latency was pretty much unbeatable. In a lot fo ways it was really a 3D extension to their earlier 2D (NES, SNES, GB, GBA) GPU designs. Something like that sitting where the scan out engines exist today (with their multiple planes the composite today) would be absolutely killer if you could do hardware software co design to take advantage of it. I've also thought that something like that would be great in VR/AR.

aspaceman 6y ago
You've just inspired a dive into documentation. I do GPU architecture research and reading this was exciting. Thanks.
- monocasa 6y ago
  Awesome! I'd really suggest Martin Korth's fantastic documentation in that case. In a lot of ways it was better than even the official docs given to developers. https://problemkaputt.de/gbatek.htm https://problemkaputt.de/gbatek.htm
- deergomoo 6y ago
  https://www.copetti.org/projects/consoles/nintendo-ds/ https://www.copetti.org/projects/consoles/nintendo-ds/ This guy has articles about most of the major consoles from the last 30 years, I find them fascinating to read even though much of it goes over my head.
giantrobot 6y ago
Buffers see extensive use in modern shaders. A frame will get rasterized several times for different shaders before those buffers are all composited into an output buffer. Several types of things can only be known if you've got the full scene rasterized or at least more than a few lines. If you've only got fixed function shaders and a relatively simple rasterizer the raster-to-display works fine. The DS also only has a single contributor to the output. A game on a PS4 or a PC is just one of several contributors to a compositing window/display manager which is actually doing final composition to the output device(s).

upofadown 6y ago

As someone who uses a tiling window manager it has been a long time since I have actually used a compositor. I have to ask if the cool window effects are actually worth the bother? I just can't see anyone actually caring about stuff like that in 20 years.

iso8859-1 6y ago
On Android, you have Picture-In-Picture video support now. How do you imagine it working without a compositor? Do you think nobody cares about that feature?
- toast0 6y ago
  Picture in picture videos used to be done with chromakey. Playing a video in one window on top of another window isn't some new thing compositing provides. You could do it in Mac classic or Windows 3.1 or BeOS.
CmdrKrool 6y ago
I have set up a keyboard shortcut to toggle my compositor on and off and tbh I find it tends to remain off most of the time, since I don't care for 'cool window effects' either. Most of the especially visual applications I use - eg. video players, games - tend to handle themselves in this respect, taking their steps for vsync as necessary. On the rare occasion that I'm offended by a bit of tearing I'll just activate the shortcut and then no problem. I just worry that with the badmouthing of X11 that goes around, one day this control - the ability to run without a compositor, if I want to - might be taken away from me.
- cycloptic 6y ago
  The X server code isn't going anywhere. Somehow I feel like it will be around forever.
  - wmf 6y ago
    I predict there will be a systemd-style debate about removing X11 by 2030 and it will be completely gone by 2040 (since someone mentioned 20 years). There won't be a single non-composited OS available at all and people born after 2000 won't really remember anything else.

nayuki 6y ago

Variable refresh rate technologies like FreeSync and G-Sync are used in some video games. Wouldn't this improve the latency of desktop compositing too? After all, the idea is that the GPU sends out a frame to the monitor once it's finished, not once a periodic interval is reached.

raphlinus 6y ago
We had a discussion on this in #gpu on our Zulip. The short answer is that if the screen is updating at 60fps, then FreeSync and its cousins are effectively a no-op. If the display is mostly idle, then there is potentially latency benefit in withholding scanout until triggered by a keypress. Older revisions of FreeSync have a "minimum frame rate" which actually doesn't buy you much; 40Hz is common: https://www.amd.com/en/products/freesync-monitors https://www.amd.com/en/products/freesync-monitors
- amluto 6y ago
  I would expect that VRR helps if you try to make your compositor only start compositing as late as possible while still finishing on time. With VRR, one could delay the next frame slightly if the deadline gets missed. This would work even when rendering at 60Hz.
  - medlyyy 6y ago
    Problem is that causes jitter which is extremely noticeable even in small amounts. You can't have smooth animations if the frame timing is constantly varying.

PaulDavisThe1st 6y ago

It is fascinating to me, as someone who has been doing real time audio work for 20 years, to see how two fundamental processes have been at work during that time: * audio developers recognizing the parallels between the way audio and video hardware works, and taking as many lessons from the video world as possible. The result is that audio data flow (ignoring the monstrosity of USB audio) is now generally much better than it used to be, and in many cases is conceptually very clean. In addition, less and less processing is done by audio hardware, and more and more by the CPU. * video developers not understanding much about audio, and failing to notice the parallels between the two data handling processes. Rather than take any lessons from the audio side of things, stuff has just become more and more complex and more and more distant from what is actually happening in hardware. In addition, more and more processing is done by video hardware, and less and less by the CPU. In both cases, there is hardware which requires data periodically (on the order of a few msec). There are similar requirements to allow multiple applications to contribute what is visible/audible to the user. There are similar needs for processing pipelines between hardware and user space code. (one important difference: if you don't provide a new frame of video data, most humans will not notice the result; if you don't provide a new frame of audio fata, every human will hear the result). I feel as if both these worlds would really benefit from somehow having a full exchange about the high and low level aspects of the problems they both face, how they have been solved to date, how might be solved in the future, and how the two are both very much alike, and really quite different.

modeless 6y ago
> [video] stuff has just become more and more complex and more and more distant from what is actually happening in hardware There's been a huge push recently to remove layers from graphics APIs and expose hardware more directly, in the form of Vulkan, DX12, and Metal. The problem that this has uncovered is that graphics hardware is not only very complex but quite varied, so exposing "what is actually happening in hardware" is incompatible with portability and simplicity. As a result you get APIs that are either not portable or hard to use. > more and more processing is done by video hardware, and less and less by the CPU. Unfortunately this will remain a reality because it comes with a 20-100x performance advantage.

guhcampos 6y ago

"As the experience of realtime audio illuminates, it’s hard enough scheduling CPU tasks to reliably complete before timing deadlines, and it seems like GPUs are even harder to schedule with timing guarantees." Well, I suppose that's way more feasible in RISC CPUs. Damn I hate when Apple is right about something.

monocasa 6y ago
RISC as a concept doesn't really help you with making timing guarantees.
- est31 6y ago
  Yeah as of today, the instruction language used isn't really relevant anyway. You still have to figure out the same problems as a chip designer.
danhor 6y ago
More modern/better arm cores and common environments (starting at approx. Cortex-M3 microcontrollers, where flash instruction fetching depends on page alignment) are also pretty unpredictable. In fact, most advanced pipeline designs, no matter the ISA, have very hard-to-predict instruction timings. Couple this with dynamic frequency scaling and the instruction timings are irrelevant.

mncharity 6y ago

A year or few back, I did camera-passthru XR in VR HMDs, on a DIY stack with electron.js (which is chromium) as compositor. So a couple of thoughts... The full-screen electron window apparently displaced the window system compositor. So "compositing" specialized compositors can be useful. Chromium special cased video, making it utterly trivial to do comfortable camera passthru, while a year+ later, mainstream stacks were still considering that hard to impossible. So having compositor architecture match the app is valuable. That special case would collapse if the CPU needed to touch the camera frames for analysis. So carefully managing data movement between GPU and CPU is important. That's one design objective of google's mediapipe for instance. Perhaps future compositors should permit similar pipeline specifications? My XR focus was software-dev "office" work, not games. So no game-dev "oh, the horror! an immersion-shattering visual artifact occurred!" - people don't think "Excel, it's ghastly... it leaves me aware I'm sitting in an office!". Similarly, with user balance based on nice video passthru, the rest of the rendering could be slow and jittery. Game-dev common wisdom was "VR means 90 fps, no jitters, no artifacts, or user-sick fail - we're GPU-tech limited", and I was "meh, 30/20/10 fps, whatever" on laptop integrated graphics. So... Games are hard - don't assume their constraints are yours without analysis. And different aspects of the rendered environment can have very different constraints. My fuzzy recollection was chromium could be persuaded to do 120 Hz, though I didn't have a monitor to try that. Or higher? - I fuzzily recall some variable to uncap the frame rate. I've used linux evdev (input pipeline step between kernel and libinput) directly from an electron renderer process. Latency wasn't the motivation, so I didn't measure it. But that might save an extra ms or few. At 120 Hz, that might mean whatever ms HID to OS, 0-8 ms wait for the next frame, 8 ms processing and render, plus whatever ms to light. On electron.js. New interface tech like speech and gesture recognition may start guessing at what it's hearing/seeing many ms before it provides it's final best guess at what happened. Here low latency responsiveness is perhaps more about app system architecture than pipeline tuning. With app state and UI supporting iterative speculative execution. Eye tracking changes things. "Don't bother rendering anything for the next 50 ms, the user is saccading and thus blind." "After this saccade, the eye will be pointing at xy, so only that region will need a full resolution render" (foveated rendering). Patents... but eventually.

emmanueloga_ 6y ago

> When pressing a key in, say, a text editor, the app would prepare and render a minimal damage region Is this similar to the "dirty rectangles" technique? [1] It seems difficult to implement some sort of scene system (for either an app, game or general GUI) that given a single key press can determine a minimum bounding box of changes on the screen no matter what the scene is, and is able to render just that bounding box. If the single key occurs on a text area, potentially the whole text area could be "damaged" right? Edit: I was looking at this with the "Rendering" tab of chrome dev tools, enabling "Paint flashing" and "Layout Shift Regions", it seems like the text area is its own layer and the space is partitioned pretty cleverly on things like paragraphs and lines, but from time to time the whole text area just flashes, which tells me the algorithm sometimes is not sure what is dirty and just repaints the whole thing, but not always. 1: https://wiki.c2.com/?DirtyRectangles https://wiki.c2.com/?DirtyRectangles

munificent 6y ago
> Is this similar to the "dirty rectangles" technique? Yes. > It seems difficult to implement some sort of scene system (for either an app, game or general GUI) that given a single key press can determine a minimum bounding box of changes on the screen no matter what the scene is, and is able to render just that bounding box. I don't think it's that bad. In the days before hardware graphics, basically every 2D sprite engine used in every computer game you liked had to do this logic. Text editors are a harder case in some ways because of line wrapping, but the editor does need to figure out where everything goes spatially in order to render, so extending that to tell which things have not moved is, I think, not that difficult.

crazyloglad 6y ago

A few pointers: The first bit is that practically all that is argued in for here, is in Android - and has been so for a long time (~2013 last I worked on the topic in that context) and it is still not enough - they will go (and are going) higher refresh rate. Even at the time there were crazy things like tuning CPU governor to wake up when touch screen detects incoming 'presence'(before you can get an accurate measurement of where and how hard the finger hit) - it's not like you are going to change your muscle intention mid flight. Input events were specifically tagged so that the memory would be re-used rather than GCed and reallocated. A sunny frame could go from motion to photon in 30ms. Then something happens and the next takes 110ms. That something is often garbage collectors and other runtime systems that think it is safe to do things as there is no shared systemic synchronization mechanism around for these things. The second is that this is -ultimately- a systemic issue and the judge, jury and executioner is the user's own perception. That's what you are optimising for. Treating it as a graphics only thing is not putting the finger on something, it is picking your nose. The input needs to be in on it, audio needs to be in on it, and the producer needs to be cooperative. Incidentally, the guitar hero style games and anything VR are good qualitative evaluation targets but with brinelling like incremental loads. (Larger scale) 1. communicate deadlines to client so the renderer can judge if it is even worth doing anything. 2. communicate presentation time to client so animations and video frames arrive at the right time. 3. have a compositor scheduler than will optimize for throughput, latency, minimizing jitter or energy consumption. 4. inform the scheduler about the task that is in focus, and bias client unlock and event routing to distinguish between input focus and the thundering herd. 5. type annotate client resources so the scheduler knows what they are for. 6. coalesce / resample input and resize events. 7. align client audio buffers to video buffers. 8. clients with state store/restore capabilities can rollback, inject and fastforward (http://filthypants.blogspot.com/2016/03/using-rollback-to-hide-latency-in.html http://filthypants.blogspot.com/2016/03/using-rollback-to-hi...) There is a bunch of more eccentric big stuff after that as well as the well known quality of life improvements (beyond the obvious gamedev lessons like don't just malloc in your renderloop and minimize the amount of syscall jitter in the entire pipeline), but baby steps. Oh, and all of the above++ is already in Arcan and yet it is still nowhere near good enough.

baybal2 6y ago

Androider 6y ago

There's some hardware fixes that are becoming available. Variable refresh rate (VRR / G-Sync / FreeSync / Adaptive Sync) and high-refresh rate hardware is now starting to gain traction on the high-end, usually with both technologies combined. Apple's ProMotion is often touted as a "120Hz" feature, but it's also a variable refresh in that it can support much lower framerates. Without a static v-sync interval, you can display the frames as they become available, for both perfect tear-free visuals and lower latency. It's quite likely we'll have that on Mac's as well. I recently got a 144Hz FreeSync (G-Sync compatible) "business" monitor, so this tech is starting to filter down from the gamer side. It works great with a compositor on Linux! Ultra smooth mouse and input response at 144Hz, and completely tear-free. I would highly recommend it for developers as well.

badsectoracula 6y ago
> It works great with a compositor on Linux! Ultra smooth mouse and input response at 144Hz, and completely tear-free. I would highly recommend it for developers as well. I got a 165Hz monitor and while it is indeed smoother than a 60Hz monitor with a vsync'd compositor, it still isn't as smooth as a 60Hz monitor without a compositor.

deleted 6y ago

[deleted]

tomxor 6y ago

> [...] the worst that can happen is just jank, which people are used to, rather than visual artifacts. This is an important point. There are many solutions out there (beyond compositing) that add incredible complexity along with other costs in order to solve a perceived problem by it's proponents... not uncommonly those same people pushing the solution are incapable of fairly judging the cost benefit - sometimes the solutions are just not worth the cost. There is currently an issue in chromium compositing with checkerboarding (blanking whole regions of the screen with white) when scrolling due to an optimization released to reduce jank: https://bugs.chromium.org/p/chromium/issues/detail?id=1062662 https://bugs.chromium.org/p/chromium/issues/detail?id=106266... You will notice this in recent releases of chromium when you scroll any non trivial page fast enough... or a simple page very fast by yanking the scroll bar... unless you have an extremely fast computer and graphics card. When reading through that thread you will notice how defensive they are, it's a complex and no doubt incredible bit of coding they have done - unfortunately the side-effect is worse than the original problem that most users don't even notice. In the long run I think time will prove this problem and it's solutions are purely transient... In the same way font anti-aliasing is becoming obsolete with DPIs higher than we can perceive, solving jank isn't necessary if you cannot perceive it (i.e using frame rates > 60 Hz)... and the original problem really isn't as bad as all the people attempting to solve it think it is anyway.

Firadeoclus 6y ago
> In the same way font anti-aliasing is becoming obsolete with DPIs higher than we can perceive Screens might exceed acuity of most users but not hyperacuity. Even on a 400ppi phone screen at typical viewing distances it is possible to tell whether a slanted line is anti-aliased or not. Font anti-aliasing is not becoming obsolete any time soon.
- pekim 6y ago
  I agree, anti-aliasing is certainly still required at 400dpi. Although subpixel anti-aliasing is not.

roca 6y ago

The beam-racing compositor is a cool idea. It seems to me, however, that it could create visible jitter for applications that use traditional vsync-driven frame timing --- e.g. an animation of vertical motion would see a one-frame latency change when the damage rectangle crosses some vertical threshold.

raphlinus 6y ago
If you're animating (or doing smooth window resizing), then your "present" request includes a target frame sequence number. The way I analyze the text editor case is that it's something of a special case of triple-buffering, but instead of spamming frames as fast as possible at the swapchain, you do multiple revisions for a single target sequence number. The last revision to make it ahead of the beam wins. If you have a smooth clock animation in addition to some latency-critical input sensitive elements, then the clock hands would be in the same position for each revision, only the other elements would change. If done carefully, this would give you smoother animations at considerably lower power consumption than traditional triple buffering. It's not the way present APIs are designed today, though, one of my many sadnesses. As I hinted, this is a good way to synchronize window resize as well. The window manager tells the application, "as of frame number 98765, your new size is 2345x1234." It also sends the window frame and shadows etc with the same frame number. These are treated as a transaction; both have to arrive before the deadline for the transaction to go through. This is not rocket science, but requires careful attention to detail.

LargoLasskhyfv 6y ago

I don't get this obsession with compositors. I just switch them off. Has no benefit for me. If I'm so inclined I can move windows with video running in Full-HD without any tearing across several screens like mad. When I'm in a terminal window of some sort I don't care for the real transparency, because it is unergonomic. When I'm in some other application the same applies. Can someone enlighten me about the real benefits? Seriously.

sjy 6y ago
The app thumbnails in macOS’s Mission Control [1] and GNOME’s Activities overview [2] provide a nice way of managing graphical windows and workspaces which requires a compositor. Animated window management in general is a UI affordance that I value. I’m not sure if it’s a “real benefit,” but consistently getting this right is one thing that attracts users to macOS and iOS. [1] https://support.apple.com/en-us/HT204100 https://support.apple.com/en-us/HT204100 [2] https://help.gnome.org/users/gnome-help/stable/shell-introduction.html.en#activities https://help.gnome.org/users/gnome-help/stable/shell-introdu...
- LargoLasskhyfv 6y ago
  I know them. Maybe I'm just an old fart and "set in my ways", but it's not even "nice to have" for me when it compromises the general haptics of the environment, as in dragging it down with sluggishness.

andrekandre 6y ago

without a compositor, how does one implement a feature like expose/mission control effeciently?