9 ms·

The bottleneck of UI is not the rendering. A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don'

by yason 4y ago

The bottleneck of UI is not the rendering. A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don't need a GPU for that today. What causes user interfaces to hick up is that it's too easy to do stuff in the main UI thread. First it doesn't matter but stuff does accumulate, and eventually the UI begins to freeze briefly for example, after you press a button. The user interface gets intermingled with the program logic, and the execution of the program will visibly relay its operations to the user. It would be very much possible to keep the user interface running in a thread, dedicated to a single CPU on a priority task, updating at vsync rate as soon as there are dirty areas in the window, merely sending UI events to the processing thread, and doing absolutely nothing more. This is closer to how games work: the rendering thread does rendering and there are other, more slow-paced threads running physics, simulation, and game logic at a suitable pace. With games it's obvious because rendering is hard and it needs to be fast so anything else that might slow down rendering must be moved away but UIs shouldn't be any different. An instantly reacting UI feels natural to a human, one that takes its time to act will slow down the brain. But you don't need a GPU for that.

DeathArrow 4y ago
I think the bottleneck comes from updating each UI element instead updating them in batches and updating elements that don't need to be updated.
- kaba0 4y ago
  That’s just retained mode GUIs calculating what got damaged and only updating those. That’s how most GUI frameworks work since many decades.
eptcyka 4y ago
It's significantly cheaper to render a fullscreen window on the GPU than it is on the CPU if you're running at 2560x1600 or 4K. To push that many pixels, you have to send significant amounts of data (framebuffers) at 60 FPS to the GPU anyway - which is no small feat and is bound to eat into the battery. It's just more efficient to run on the GPU.
seanalltogether 4y ago
Separating the logic thread from the render thread doesn't end up being the silver bullet to performance. If I press the letter 'a' on my keyboard and my IDE suddenly stalls on a bunch of code hinting logic, that new state is still not going to make it to the render thread for a couple frames. As long as the render thread is dependent on state coming in from the logic thread, the stalls still propogate. You could have some wins around scrolling and any animations that you can commit ahead of time
- tracker1 4y ago
  That's a big reason why VS Code pushed the extensions (for code completion logic, etc) to another process. Where VS proper was in-thread... man, some web projects in VS are just painful. I do think that they will need to do something similar with a higher level language support for extensions. Could probably piggy-back the Deno work for using V8/JS/TS for extensions similar, maybe even compatible with VS Code extensions. It's a massive user space, but seeing the core feature set in another editor could be really nice. For me the killer feature of VS Code is really the integrated terminal. Not having to switch windows, or leave the app to run a quick command is really big imo.
ben-schaaf 4y ago
Unfortunately this isn't true anymore when you get to very high resolutions like 8k. Just clearing the framebuffer of an 8k display at 60hz requires ~6GB/s; about 1/10th of the theoretical memory bandwidth of modern desktop processors. Add in compositing for multiple windows, text rendering and none of that likely being multi-threaded it's pretty clear a CPU has no chance of keeping up.
- fxtentacle 4y ago
  You are correct, but so is the comment you replied to. The latency that matters for a text editor is how long it takes for a new keypress to show up. And that's usually a 32x32 pixel area or less.
  - gizmo 4y ago
    That's not sufficient, though. You want the 0.1% case where you press undo a couple of times and big edit operations get reversed to be smooth. You have to hit your frame target when a lot is happening, the individual key press case is easy. It's just like a video game. A consistent 60fps is much better than an average frame rate of 120fps that drops to 15fps when the shooting starts and things get blown up. You spend all the time optimizing for the worst case where all your caches get invalidated at the same time.
    - fxtentacle 4y ago
      In a video game, you'd just blur the shit out of everything else while you grant yourself a 2nd frame to re-render the rest of the content [1]. Or you do temporal remapping like the Oculus does if you rotate your head too quickly. [1] https://developer.nvidia.com/blog/nvidia-vrss-2-dynamic-foveated-rendering-no-assembly-required/ https://developer.nvidia.com/blog/nvidia-vrss-2-dynamic-fove...
  - ben-schaaf 4y ago
    The other latency that matters is how long it takes to draw each frame when scrolling through the code, which is usually most if not all of the framebuffer.
zodester 4y ago
React Native follows this pattern by moving most JS processing off the main thread allowing scrolling and other input to happen without blocking for a response from the JS VM. However this does end up causing a lot of problems with text input and gestures as now you have a sync issue between the threads, if you get caught processing a bunch of stuff in the JS thread the app may appear to be responsive with scrolling but nothing happens in response to button taps or text insertion. It is the only way RN was going to work on lower end hardware though so probably is the right solution if you assume running react everywhere is a good idea.
- mike_hearn 4y ago
  Yes, making GUIs responsive isn't as simple as just "don't run stuff on the UI thread". There are good reasons to run stuff there even if you're going to hang up the app for a brief period, namely, the user won't see partial/incorrect updates like non-syntax highlighted text or incorrectly clipped shapes, and - especially important - it means you can't end up with invalid GUI states like the user pressing a button that does X and then immediately pressing another button that does the opposite of X, where you should have disabled the other button but didn't get to it in time. Web apps have this sort of problem if they aren't using enough JS, and it can cause all kinds of weird bugs and errors. The reality is that moving things off the UI thread is always a calculated engineering decision. It can be worth it, but it complicates the code and should only be done if there's a realistic chance of the user noticing.
Traubenfuchs 4y ago
Proper development of multithreaded desktop apps and not blocking the UI thread appears to be a lost art. I remember I was drawing all the threads on a person wide piece of paper on the wall when I was working on my first commercial winforms application. It's not exactly hard, but requires basic understanding for UX, threading and the platforms you are working with. Nowadays I still regularly see applications with (temporarily) frozen windows and I just don't understand how that's possible. When I was developing my winforms apps, anything that would do more than perfectly predictable UI manipulations was run on a background thread (in a task), would be forbidden from starting twice at the same time and only updated the UI when done.
- HideousKojima 4y ago
  In .NET Core/.NET 5+ I think it even defaults to having awaited Tasks run on a different thread. So long as you're using async/await properly it's almost impossible to screw up (no need to worry about SynchronizationContext etc.), yet I still see tons of examples of people who simply don't understand threading, async/await, etc. and screw it all up
  - Traubenfuchs 4y ago
    As I don't like async/await (read: I probably don't properly understand it) I just added two extension methods to Control that took a lambda for updating UI stuff (one for Invoke, one for BeginInvoke). Probably kind of like this: https://stackoverflow.com/a/36107907/2306536 https://stackoverflow.com/a/36107907/2306536 I haven't developed C# for almost a decade, so I probably didn't even have access to async/await back then.
    - HideousKojima 4y ago
      Even if you don't really understand async/await in C#, you can do it mostly right by following just a few rules: 1) Make anything that accesses a resource over the network, in a database, or in the filesystem async. So basically any time you make an HTTP request, execute a SQL query, or open a file etc. Otherwise you should generally stick to synchronous since async code is actually slightly slower than synchronous code. An exception to this might be if you're doing some heavy computation that's going to take a while and don't want it blocking the UI thread 2) Async is viral. If you have a method that calls an async method, you need to make the calling method async too. You need to then make its own calling method async as well all the way back to Main() 3) Never use async void as a return type, with the sole notable exception to this rule being event handlers. Use async Task instead as the return type (or async Task<T> if it returns a value) 4) Always call async code with the "await SomeAsyncFunction()" syntax. Calling it other ways like "SomeAsyncFunction().Result" will eliminate most of the benefits of using async in the first place since it is blocking, can cause deadlocks in some cases, etc. Follow those 4 rules and you don't really even need to understand how async works under the hood, though personally I'd recommend learning it as it will help you a lot when you run into issues with edge cases.
kllrnohj 4y ago
> A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don't need a GPU for that today. No we didn't. We had nothing close to that in the 90s. Typing was responsive, but that's not anywhere close to 60fps and a very small region to update to boot (120wpm = 600 keys/minute = 10 keys/s, or 10fps). Scrolling benefits from 60fps+, but in the 90s scrolling jumped by lines at a time because it couldn't do anything better. You need a GPU to keep up with smooth scrolling at modern resolutions. But this also shouldn't be a surprise. What the article talks about is bog standard stuff for all current UI toolkits. It's what mobile platforms like Android have been doing for a decade now.
- TylerE 4y ago
  60 is the new 30. Was amazed at how good my latest iPhone feels with 120hz scrolling. It's like magic.
  - MaxikCZ 4y ago
    I thought moving to 144 Hz monitor made sense just for gaming, but now on 60 Hz monitor just moving the mouse on desktop feels as if the PC is struggling

by yason 4y ago

DeathArrow 4y ago

I think the bottleneck comes from updating each UI element instead updating them in batches and updating elements that don't need to be updated.

kaba0 4y ago
That’s just retained mode GUIs calculating what got damaged and only updating those. That’s how most GUI frameworks work since many decades.

eptcyka 4y ago

It's significantly cheaper to render a fullscreen window on the GPU than it is on the CPU if you're running at 2560x1600 or 4K. To push that many pixels, you have to send significant amounts of data (framebuffers) at 60 FPS to the GPU anyway - which is no small feat and is bound to eat into the battery. It's just more efficient to run on the GPU.

seanalltogether 4y ago

Separating the logic thread from the render thread doesn't end up being the silver bullet to performance. If I press the letter 'a' on my keyboard and my IDE suddenly stalls on a bunch of code hinting logic, that new state is still not going to make it to the render thread for a couple frames. As long as the render thread is dependent on state coming in from the logic thread, the stalls still propogate. You could have some wins around scrolling and any animations that you can commit ahead of time

tracker1 4y ago
That's a big reason why VS Code pushed the extensions (for code completion logic, etc) to another process. Where VS proper was in-thread... man, some web projects in VS are just painful. I do think that they will need to do something similar with a higher level language support for extensions. Could probably piggy-back the Deno work for using V8/JS/TS for extensions similar, maybe even compatible with VS Code extensions. It's a massive user space, but seeing the core feature set in another editor could be really nice. For me the killer feature of VS Code is really the integrated terminal. Not having to switch windows, or leave the app to run a quick command is really big imo.

ben-schaaf 4y ago

Unfortunately this isn't true anymore when you get to very high resolutions like 8k. Just clearing the framebuffer of an 8k display at 60hz requires ~6GB/s; about 1/10th of the theoretical memory bandwidth of modern desktop processors. Add in compositing for multiple windows, text rendering and none of that likely being multi-threaded it's pretty clear a CPU has no chance of keeping up.

fxtentacle 4y ago
You are correct, but so is the comment you replied to. The latency that matters for a text editor is how long it takes for a new keypress to show up. And that's usually a 32x32 pixel area or less.
- gizmo 4y ago
  That's not sufficient, though. You want the 0.1% case where you press undo a couple of times and big edit operations get reversed to be smooth. You have to hit your frame target when a lot is happening, the individual key press case is easy. It's just like a video game. A consistent 60fps is much better than an average frame rate of 120fps that drops to 15fps when the shooting starts and things get blown up. You spend all the time optimizing for the worst case where all your caches get invalidated at the same time.
  - fxtentacle 4y ago
    In a video game, you'd just blur the shit out of everything else while you grant yourself a 2nd frame to re-render the rest of the content [1]. Or you do temporal remapping like the Oculus does if you rotate your head too quickly. [1] https://developer.nvidia.com/blog/nvidia-vrss-2-dynamic-foveated-rendering-no-assembly-required/ https://developer.nvidia.com/blog/nvidia-vrss-2-dynamic-fove...
- ben-schaaf 4y ago
  The other latency that matters is how long it takes to draw each frame when scrolling through the code, which is usually most if not all of the framebuffer.

zodester 4y ago

React Native follows this pattern by moving most JS processing off the main thread allowing scrolling and other input to happen without blocking for a response from the JS VM. However this does end up causing a lot of problems with text input and gestures as now you have a sync issue between the threads, if you get caught processing a bunch of stuff in the JS thread the app may appear to be responsive with scrolling but nothing happens in response to button taps or text insertion. It is the only way RN was going to work on lower end hardware though so probably is the right solution if you assume running react everywhere is a good idea.

mike_hearn 4y ago
Yes, making GUIs responsive isn't as simple as just "don't run stuff on the UI thread". There are good reasons to run stuff there even if you're going to hang up the app for a brief period, namely, the user won't see partial/incorrect updates like non-syntax highlighted text or incorrectly clipped shapes, and - especially important - it means you can't end up with invalid GUI states like the user pressing a button that does X and then immediately pressing another button that does the opposite of X, where you should have disabled the other button but didn't get to it in time. Web apps have this sort of problem if they aren't using enough JS, and it can cause all kinds of weird bugs and errors. The reality is that moving things off the UI thread is always a calculated engineering decision. It can be worth it, but it complicates the code and should only be done if there's a realistic chance of the user noticing.

Traubenfuchs 4y ago

Proper development of multithreaded desktop apps and not blocking the UI thread appears to be a lost art. I remember I was drawing all the threads on a person wide piece of paper on the wall when I was working on my first commercial winforms application. It's not exactly hard, but requires basic understanding for UX, threading and the platforms you are working with. Nowadays I still regularly see applications with (temporarily) frozen windows and I just don't understand how that's possible. When I was developing my winforms apps, anything that would do more than perfectly predictable UI manipulations was run on a background thread (in a task), would be forbidden from starting twice at the same time and only updated the UI when done.

HideousKojima 4y ago
In .NET Core/.NET 5+ I think it even defaults to having awaited Tasks run on a different thread. So long as you're using async/await properly it's almost impossible to screw up (no need to worry about SynchronizationContext etc.), yet I still see tons of examples of people who simply don't understand threading, async/await, etc. and screw it all up
- Traubenfuchs 4y ago
  As I don't like async/await (read: I probably don't properly understand it) I just added two extension methods to Control that took a lambda for updating UI stuff (one for Invoke, one for BeginInvoke). Probably kind of like this: https://stackoverflow.com/a/36107907/2306536 https://stackoverflow.com/a/36107907/2306536 I haven't developed C# for almost a decade, so I probably didn't even have access to async/await back then.
  - HideousKojima 4y ago
    Even if you don't really understand async/await in C#, you can do it mostly right by following just a few rules: 1) Make anything that accesses a resource over the network, in a database, or in the filesystem async. So basically any time you make an HTTP request, execute a SQL query, or open a file etc. Otherwise you should generally stick to synchronous since async code is actually slightly slower than synchronous code. An exception to this might be if you're doing some heavy computation that's going to take a while and don't want it blocking the UI thread 2) Async is viral. If you have a method that calls an async method, you need to make the calling method async too. You need to then make its own calling method async as well all the way back to Main() 3) Never use async void as a return type, with the sole notable exception to this rule being event handlers. Use async Task instead as the return type (or async Task<T> if it returns a value) 4) Always call async code with the "await SomeAsyncFunction()" syntax. Calling it other ways like "SomeAsyncFunction().Result" will eliminate most of the benefits of using async in the first place since it is blocking, can cause deadlocks in some cases, etc. Follow those 4 rules and you don't really even need to understand how async works under the hood, though personally I'd recommend learning it as it will help you a lot when you run into issues with edge cases.

kllrnohj 4y ago

> A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don't need a GPU for that today. No we didn't. We had nothing close to that in the 90s. Typing was responsive, but that's not anywhere close to 60fps and a very small region to update to boot (120wpm = 600 keys/minute = 10 keys/s, or 10fps). Scrolling benefits from 60fps+, but in the 90s scrolling jumped by lines at a time because it couldn't do anything better. You need a GPU to keep up with smooth scrolling at modern resolutions. But this also shouldn't be a surprise. What the article talks about is bog standard stuff for all current UI toolkits. It's what mobile platforms like Android have been doing for a decade now.

TylerE 4y ago
60 is the new 30. Was amazed at how good my latest iPhone feels with 120hz scrolling. It's like magic.
- MaxikCZ 4y ago
  I thought moving to 144 Hz monitor made sense just for gaming, but now on 60 Hz monitor just moving the mouse on desktop feels as if the PC is struggling