5 ms·

I get why people like immediate mode GUIs but the performance tends to be pretty bad.

joshmarinacci 5y ago
Bad is relative. If it's a game where you have to repaint the entire screen anyway, then it doesn't matter. But a lot of potential optimizations are lost when you don't use a retained scenegraph.
runawaybottle 5y ago
I mean, I’m looking at this on mobile and desktop and just like a another poster mentioned, in relative terms, the damn thing is smoother than most web apps. What’s the end goal here, right? The web sucks at the moment and I’m open to all ideas.
- naasking 5y ago
  Smoother yes, but to be fair how much CPU and power is it using? That's invisible to you right now so appearances can be deceptive.
- andrewmcwatters 5y ago
  As soon as you want to do anything remotely as complex as the web does, like say arbitrary UI element sizes with scrollable overflow, you immediately have to do the same thing CSS 2.1 rasterizer engines do.
  - runawaybottle 5y ago
    I think you are just dead wrong about this. I’ve played video games my whole life and they always always had more complicated UIs, akin to web apps. It’s always been a richer experience.
    - andrewmcwatters 5y ago
      I am the author of a CSS 2.1 rasterizer, as well as the author of GUI software for game engines. They have different ways they draw user-interfaces. CSS 2.1 and the associated modules and extensions to it that you would use today to build UIs are much more complicated to implement than game engine UI. If the opposite were true, you'd see more web browsers that weren't KHTML derivatives, but that's not the case.
      - runawaybottle 5y ago
        I am the author of a CSS 2.1 rasterizer Yikes, checkmate. I guess, the frustration expressed has more to do with applications shifting to the browser. However we are rendering now is just not acceptable for web apps. My gut said ‘fuck it, do what games do’. We need something different, and I’d be happy to hear your thoughts.
        andrewmcwatters 5y ago
        I agree with you. I think the real issue is that web developers don't have any ability to control the compositor besides hinting it to move portions of documents into explicit backing layers. You do this today with CSS transform operations, because those operations are done in shaders today. That being said, I don't know if web developers should have that type of access, either. The slowness you experience is based primarily on not redrawing something once, but several times, as modern web browsers need to break down very large web pages into tiles, then repeat draw instructions per tile where intersections exist. If I recall correctly, WebKit actually has an option to let you see the tile boundaries and when layers invalidate. This is my best understanding of the architecture KHTML derivatives use today. Compositing is very slow. And layout algorithms are very, very slow. So you can either do some fun stuff that I think imgui does, like sending VBOs over which is very performant, because you can draw lots of elements but you can't at all do things that people care about like compositing (think blurs behind panels, or layer opacity, etc.), or you can composite layers, but this becomes very expensive and slow to do. If you simply said, hey redraw this portion of a web page into a single framebuffer, you'd cut down on draw time a little bit, I'm sure, but because you can't allocate framebuffers reliably at arbitrary sizes, or beyond the viewport size, you run into GUI software that has weird limitations like all of a sudden you notice that for some reason you can't resize windows beyond the bounds of your screen, etc. I think Valve actually did this somehow, but I'm not sure how they did it with their old VGUI code. Anyway, all of that being said, today's modern GUI architectures allow for two choices: fast and lot of elements with no compositing, or slow and really good looking GUIs. It turns out most people want the latter, but complain about it, because reasonably(!) it's not fast, and hardware keeps getting faster, but GUIs are one of the prime examples of us eating up all of that performance.
  - kragen 5y ago
    Which same thing do you mean? I read your very interesting longer comment below, and you mentioned a lot of differences, and it's not clear which of them you mean arbitrary UI element sizes with scrollable overflow requires you to do. Do you mean that it requires you to use very slow layout algorithms? Maybe the problem there is that CSS wasn't designed to be efficiently implementable; I feel like CSS doesn't usually give me better layouts than LaTeX, which runs orders of magnitude faster than any CSS engine. Is there a specific reason that scrollable overflow (rather than visible or hidden, which TeX can do) inherently makes the problem much harder than what TeX does? Naively I'd think making the overflow scrollable wouldn't affect the computational complexity at all, but clearly I know a lot less about the space than you do. Are there other things in CSS that are very useful for UI layout but very difficult to implement with submillisecond timing guarantees? As I understand it, for example, iOS UI autolayout uses Cassowary, which uses the simplex algorithm and I think may at a better point in the expressiveness/performance tradeoff space than CSS. (Hilariously in this context, the original Cassowary paper https://constraints.cs.washington.edu/solvers/cassowary-tochi.pdf https://constraints.cs.washington.edu/solvers/cassowary-toch... has an overfull hbox.) But Cassowary is from 02002, and there have been a lot of advancements in constraint solving and optimization algorithms in the last 19 years, both in theory and in practice: • Z3 wasn't released until 02012; • SAT/SMT solvers in general have gotten astoundingly better (the SAT Competition started in 02002); • there have been enormous advances in conic optimization (overview in https://arxiv.org/abs/1709.08841 https://arxiv.org/abs/1709.08841), including open-source libraries like CVXOPT that are widely used (in the context of conic optimization, anyway); • for more general optimization, automatic differentiation has gone mainstream; and • there have been huge advances in gradient-descent variants including AdaGrad (02011), RMSprop (02014?), and Adam (02014). In general, gradient-descent solvers and the interior-point methods often used for conic optimization are anytime algorithms, so you can trade off responsiveness against stability/correctness. If your layout algorithm hasn't converged by the frame deadline, you can just draw the best layout it's come up with so far; it's guaranteed to have all the right stuff in it, just maybe overlapping or in the wrong place, probably only by a few pixels, and displaying something is often better than displaying nothing. The conic optimization algorithms can also guarantee that the layout they deliver is "feasible" in the sense of respecting all the hard constraints in the constraint set; it might just be suboptimal. (Gradient-descent algorithms don't really have hard constraints; they have to simulate them by adding heavy penalties to the loss function.) I don't know of anybody actually applying this stuff to UI layout; do you?
dkersten 5y ago
Citation needed? I have a not too complex but definitely non-trivial editor I made for a little toy game engine and the Dear ImGUI code and related UI logic runs in under 1/10th of a millisecond per frame. My past Qt applications performed worse than that. My single page web applications often perform worse than that But like the other reply says, bad is relative.
- andrewmcwatters 5y ago
  Immediate mode GUIs are not designed for production. As soon as you want to do compositing, you can’t, because compositing is done with retained structures. Immediate mode GUIs already lie to you by retaining some data then flushing it later.
  - formerly_proven 5y ago
    It's not a real immediate mode GUI unless it starts drawing pixels instantly? That's not even how real immediate mode graphics APIs worked.
    - andrewmcwatters 5y ago
      I don't know what your qualm is. I just said that.
      - formerly_proven 5y ago
        Well why do you think it's important (important enough to call immediate mode a lie, even)? Immediate vs. deferred is in my view entirely about the API design, it does not really matter whether an immediate-mode GUI framework is appending to a drawlist when you call into a widget's code or is writing some pixels into some buffer; the point is that it does not have retained state about individual widgets (ImGUI doesn't, don't know about egui). As you say, compositing on a widget level is much, much harder, but even traditional retained frameworks generally only support compositing on a top-level window basis (e.g. partially translucent windows, translucent menus or drop-downs) because as we are seeing (and you point out in another post), compositing between essentially arbitrary stuff is heinously complex to implement, let alone implement with decent performance. I'd be interested to hear what use cases you have in mind for intrawindow compositing.
gameswithgo 5y ago
There are some misconceptions about this. The first issue is people assume you have to render the whole gui every frame even if nothing has changed. This is the usual pattern in games, when you have to redraw the screen anyway, but you do not have to do that, in applications where it doesn't make sense.

Bad is relative. If it's a game where you have to repaint the entire screen anyway, then it doesn't matter. But a lot of potential optimizations are lost when you don't use a retained scenegraph.

runawaybottle 5y ago

I mean, I’m looking at this on mobile and desktop and just like a another poster mentioned, in relative terms, the damn thing is smoother than most web apps. What’s the end goal here, right? The web sucks at the moment and I’m open to all ideas.

naasking 5y ago
Smoother yes, but to be fair how much CPU and power is it using? That's invisible to you right now so appearances can be deceptive.
andrewmcwatters 5y ago
As soon as you want to do anything remotely as complex as the web does, like say arbitrary UI element sizes with scrollable overflow, you immediately have to do the same thing CSS 2.1 rasterizer engines do.
- runawaybottle 5y ago
  I think you are just dead wrong about this. I’ve played video games my whole life and they always always had more complicated UIs, akin to web apps. It’s always been a richer experience.
  - andrewmcwatters 5y ago
    I am the author of a CSS 2.1 rasterizer, as well as the author of GUI software for game engines. They have different ways they draw user-interfaces. CSS 2.1 and the associated modules and extensions to it that you would use today to build UIs are much more complicated to implement than game engine UI. If the opposite were true, you'd see more web browsers that weren't KHTML derivatives, but that's not the case.
    - runawaybottle 5y ago
      I am the author of a CSS 2.1 rasterizer Yikes, checkmate. I guess, the frustration expressed has more to do with applications shifting to the browser. However we are rendering now is just not acceptable for web apps. My gut said ‘fuck it, do what games do’. We need something different, and I’d be happy to hear your thoughts.
      - andrewmcwatters 5y ago
        I agree with you. I think the real issue is that web developers don't have any ability to control the compositor besides hinting it to move portions of documents into explicit backing layers. You do this today with CSS transform operations, because those operations are done in shaders today. That being said, I don't know if web developers should have that type of access, either. The slowness you experience is based primarily on not redrawing something once, but several times, as modern web browsers need to break down very large web pages into tiles, then repeat draw instructions per tile where intersections exist. If I recall correctly, WebKit actually has an option to let you see the tile boundaries and when layers invalidate. This is my best understanding of the architecture KHTML derivatives use today. Compositing is very slow. And layout algorithms are very, very slow. So you can either do some fun stuff that I think imgui does, like sending VBOs over which is very performant, because you can draw lots of elements but you can't at all do things that people care about like compositing (think blurs behind panels, or layer opacity, etc.), or you can composite layers, but this becomes very expensive and slow to do. If you simply said, hey redraw this portion of a web page into a single framebuffer, you'd cut down on draw time a little bit, I'm sure, but because you can't allocate framebuffers reliably at arbitrary sizes, or beyond the viewport size, you run into GUI software that has weird limitations like all of a sudden you notice that for some reason you can't resize windows beyond the bounds of your screen, etc. I think Valve actually did this somehow, but I'm not sure how they did it with their old VGUI code. Anyway, all of that being said, today's modern GUI architectures allow for two choices: fast and lot of elements with no compositing, or slow and really good looking GUIs. It turns out most people want the latter, but complain about it, because reasonably(!) it's not fast, and hardware keeps getting faster, but GUIs are one of the prime examples of us eating up all of that performance.
- kragen 5y ago
  Which same thing do you mean? I read your very interesting longer comment below, and you mentioned a lot of differences, and it's not clear which of them you mean arbitrary UI element sizes with scrollable overflow requires you to do. Do you mean that it requires you to use very slow layout algorithms? Maybe the problem there is that CSS wasn't designed to be efficiently implementable; I feel like CSS doesn't usually give me better layouts than LaTeX, which runs orders of magnitude faster than any CSS engine. Is there a specific reason that scrollable overflow (rather than visible or hidden, which TeX can do) inherently makes the problem much harder than what TeX does? Naively I'd think making the overflow scrollable wouldn't affect the computational complexity at all, but clearly I know a lot less about the space than you do. Are there other things in CSS that are very useful for UI layout but very difficult to implement with submillisecond timing guarantees? As I understand it, for example, iOS UI autolayout uses Cassowary, which uses the simplex algorithm and I think may at a better point in the expressiveness/performance tradeoff space than CSS. (Hilariously in this context, the original Cassowary paper https://constraints.cs.washington.edu/solvers/cassowary-tochi.pdf https://constraints.cs.washington.edu/solvers/cassowary-toch... has an overfull hbox.) But Cassowary is from 02002, and there have been a lot of advancements in constraint solving and optimization algorithms in the last 19 years, both in theory and in practice: • Z3 wasn't released until 02012; • SAT/SMT solvers in general have gotten astoundingly better (the SAT Competition started in 02002); • there have been enormous advances in conic optimization (overview in https://arxiv.org/abs/1709.08841 https://arxiv.org/abs/1709.08841), including open-source libraries like CVXOPT that are widely used (in the context of conic optimization, anyway); • for more general optimization, automatic differentiation has gone mainstream; and • there have been huge advances in gradient-descent variants including AdaGrad (02011), RMSprop (02014?), and Adam (02014). In general, gradient-descent solvers and the interior-point methods often used for conic optimization are anytime algorithms, so you can trade off responsiveness against stability/correctness. If your layout algorithm hasn't converged by the frame deadline, you can just draw the best layout it's come up with so far; it's guaranteed to have all the right stuff in it, just maybe overlapping or in the wrong place, probably only by a few pixels, and displaying something is often better than displaying nothing. The conic optimization algorithms can also guarantee that the layout they deliver is "feasible" in the sense of respecting all the hard constraints in the constraint set; it might just be suboptimal. (Gradient-descent algorithms don't really have hard constraints; they have to simulate them by adding heavy penalties to the loss function.) I don't know of anybody actually applying this stuff to UI layout; do you?

dkersten 5y ago

Citation needed? I have a not too complex but definitely non-trivial editor I made for a little toy game engine and the Dear ImGUI code and related UI logic runs in under 1/10th of a millisecond per frame. My past Qt applications performed worse than that. My single page web applications often perform worse than that But like the other reply says, bad is relative.

andrewmcwatters 5y ago
Immediate mode GUIs are not designed for production. As soon as you want to do compositing, you can’t, because compositing is done with retained structures. Immediate mode GUIs already lie to you by retaining some data then flushing it later.
- formerly_proven 5y ago
  It's not a real immediate mode GUI unless it starts drawing pixels instantly? That's not even how real immediate mode graphics APIs worked.
  - andrewmcwatters 5y ago
    I don't know what your qualm is. I just said that.
    - formerly_proven 5y ago
      Well why do you think it's important (important enough to call immediate mode a lie, even)? Immediate vs. deferred is in my view entirely about the API design, it does not really matter whether an immediate-mode GUI framework is appending to a drawlist when you call into a widget's code or is writing some pixels into some buffer; the point is that it does not have retained state about individual widgets (ImGUI doesn't, don't know about egui). As you say, compositing on a widget level is much, much harder, but even traditional retained frameworks generally only support compositing on a top-level window basis (e.g. partially translucent windows, translucent menus or drop-downs) because as we are seeing (and you point out in another post), compositing between essentially arbitrary stuff is heinously complex to implement, let alone implement with decent performance. I'd be interested to hear what use cases you have in mind for intrawindow compositing.

gameswithgo 5y ago

There are some misconceptions about this. The first issue is people assume you have to render the whole gui every frame even if nothing has changed. This is the usual pattern in games, when you have to redraw the screen anyway, but you do not have to do that, in applications where it doesn't make sense.