5 ms·

In practice, an immediate mode API is clearly the way to go. We've all seen "smarter" approaches failing more than a few times, obviously on the simplicity sid

In practice, an immediate mode API is clearly the way to go. We've all seen "smarter" approaches failing more than a few times, obviously on the simplicity side but also, sadly, on the performance side, even with all the caching and retained structures. This is the strange beauty of brute force algorithms, they may look dumb and ugly for our refined human minds, but they tend to map very well to the capacities of computers. It is often faster to not bother and blindly render something instead of running complex logic to decide if you really need to render or not. And if you're not satisfied with the performances of a "dumb" imgui, you can always add some smarter caching without touching the API. Altough I bet this approach would only get you marginal benefits.

raphlinus 6y ago
How do you plan to address accessibility using imgui? As I argue in the post, the model of an imgui-like API for clear expression of app logic, backed by a more traditional retained render object tree, has the potential to give you the best of both worlds.
- stephc_int13 6y ago
  >How do you plan to address accessibility using imgui? I am not sure I understand this question.
  - raphlinus 6y ago
    > I am not sure I understand this question. I rest my case.
    - boulos 6y ago
      I think it’s fair to say that accessibility is an ambiguous term. Which one did you mean? (My charitable guess was a11y in the screen reader, color blindness, font size, etc. sense)
      - raphlinus 6y ago
        You are right, but I couldn't resist the opportunity. Yes, just "accessibility" is a fairly broad term, but I do mean it in the context of assistive technology such as screen readers and navigation by keyboard and devices like directional controllers. I should note, we haven't actually done any of that work in Druid yet (we're struggling with scope as it is), but trying to think ahead. I believe that having a retained tree will help enormously with building this, and that, in the meantime, tab focus navigation is a good proxy for at least some of the problems we expect to face.
        stephc_int13 6y ago
        It is a bit of work but navigation with keyboard/gamepad is possible with an immediate-mode GUI, as demonstrated by the excellent work of Omar Cornut (Dear Imgui) You have to set a compilation flag to enable the keyboard/gamepad navigation, if you missed it from the source code.
        akiselev 6y ago
        Basic navigation with keyboard/gamepad is a trivial case compared to screen reader accessibility and navigation. Take a look at the Windows Narrator keyboard shortcuts [1] like "Next Heading" or "Go to next landmark/region" or "Read row header" - these require integration with the system UI framework or the accessibility APIs directly to provide extra semantic information about what's drawn on screen. Even with integration into system accessibility APIs, Narrator (and other screen readers) performs a lot of heuristics to improve the user experience. These heuristics depend on a separate tree data structure [2] and how it changes over time, so in order to implement even the most basic accessibility, IMGUI would have to retain that information instead of throwing it away every frame. Component identity is crucial to how most (all?) screen readers work today and it's a happy accident that it also allows an efficient hybrid retained-immediate mode architecture. I don't know if it has been fixed or not, but a year or two ago you could completely nuke the Windows screen reader just by assigning React components randomly generated keys so that the DOM nodes would be destroyed and recreated every redraw. [1] https://dequeuniversity.com/screenreaders/narrator-keyboard-shortcuts https://dequeuniversity.com/screenreaders/narrator-keyboard-... [2] https://docs.microsoft.com/en-us/windows/win32/winauto/inspect-objects https://docs.microsoft.com/en-us/windows/win32/winauto/inspe...
  - boulos 6y ago
    I assume Raph meant in the screen reader sense of the term.
    - stephc_int13 6y ago
      I don't see any limitations on this side, intuitively, but I might not know all the intricacies tied to this kind of implementation. If the ability to add metadata to any widget is not sufficient then I don't know.
      - boulos 6y ago
        Yeah, I’m not an a11y person (nor a UI or even 2d graphics person). What about the “zoom under my cursor” type things? That seems much harder (though not impossible) if your interface to clients is “just draw directly”. Though thinking about it, it’s actually probably what macOS does: as a post process, take your frame buffer and then magnify the stuff in a region of interest and re-render that on top. You’re winning me over :). Edit: I assume your metadata solution is “if someone asks, the text under this region of interest says < string >”.
        stephc_int13 6y ago
        There is a common misconception about immediate-mode APIs, there are in practice very few implementations where rendering is done directly. A structure is built every frame with all the data to render later, as a batch. And of course if you need a magnify lense the obvious way would be to render it as a post process.
- liuliu 6y ago
  And current crop of imgui libraries already maintaining states cross updates (after all, that is how you can implement things as simple as drag & drop). However, the reason you need to have a retained render object tree somewhere for accessibility is due to its interaction mode (mostly request & response) different from GUI (which renders at 60~120fps). The mismatch in interactivity thus compensated by the retained render mode. It felt almost the old days where your UI is derived strictly from business logic with fixed set of widgets is better for accessibility than our new crop of apps where you have more gestures and shortcuts. It is no wonder that imgui shines if you need more gestures and eye-candy animations.
  - mwcampbell 6y ago
    The mismatch between the periodic from-scratch rendering (push) approach of IMGUI and the request/response (pull) model of current accessibility APIs is a tractable problem. Imagine a streaming protocol that allows the application or GUI toolkit to emit the contents of an accessibility tree, either in whole or as incremental updates, whenever it renders. An IMGUI could produce this data each time it renders, even if it doesn't have a retained tree in memory. It would be up to the consumer of the protocol to create and update a retained tree from this stream. That consumer could be a library running inside the same application, which could then implement current accessibility APIs such as UI Automation. Or, if the protocol caught on and was supported by assistive technologies (e.g. screen readers), those ATs themselves could be the consumers. Indeed, I believe that applications pushing their accessibility information to ATs, which can then create and update their own retained trees, would be a better alternative to the pull-based request/response model that you mention. IMO, the bigger problem is the one that Raph pointed out: widgets (including simple ones like static text) need to have stable identities throughout their lifetimes, even as their state, contents, and locations change. It seems to me that IMGUIs, or at least some of them, make this particularly difficult.
    - liuliu 6y ago
      Yes, that would work. But that would be a departure from existing imgui paradigm where the output of imgui library is a 2d bitmap. At that point, it gets really close to the vDOM territory ... As for the stable identities, yes, it is important. But I also don't see much of a problem with explicit Key: the UI components ultimately will have map back into a data object, which should have enough contextual information to provide the said Key.
adamnemecek 6y ago
Immediate rendering consumes an insane amount of CPU.
- stephc_int13 6y ago
  Not in my book. I am running a prototype immediate mode gui that I built for my yet unreleased engine and the frame time is staying below 500ns for a fullscreen app with many widgets and layers. The whole thing is built on C + Vulkan but is not entirely optimized yet. And this is running on a decent but not extremely fast computer. (An Intel NUC from 2018)
  - cycloptic 6y ago
    Now take that same code and run it on a mobile device. Can you say for certain that it's making optimal use of power? And if not, what changes would you have to make so it does?
    - stephc_int13 6y ago
      I'll do that later but I don't think it will lead to extreme CPU consumption. There are many ways to improve performances of idle/almost idle ui. Caching can be useful for large blocks of text as it tend to be the slowest thing to render. But it is preferable to incrementally optimize/add complexity and check assumptions and not make large architectural decisions ahead of time. One way to reason about it is to think in terms of pure data transformations.
      - cycloptic 6y ago
        If you're caching data, you now have stored state and it's no longer a pure data transformation. You will run into these performance problems with any large data set, not just blocks of text.
  - adamnemecek 6y ago
    Can I see a screenshot?
boulos 6y ago
Hmm. I’m a big fan of immediate mode myself, because it’s kind of amazing how much you can actually push through via brute force. However, wouldn’t you agree that Unity, Open Scene Graph, etc. are successful counter arguments? I would say it depends on your audience. The bottom layer of the stack might be a very simple retained mode, or fully brute force immediate, but it’s not clear to me that a “UI framework” for application developers should be immediate mode.
- stephc_int13 6y ago
  In my opinion, the reason why immediate-mode API are so efficient and practical is because it helps to avoid structure duplication. An UI, from an abstract point of view, is nothing but a big structure. When you're writing an application or a game, you very often already have an existing structure containing everything you need. Take a simple example : a leaderboard. Using anything but an immediate-mode API will force you to create a second, complex structure to hold this leaderboard, and a lot of glue code to copy data to and from this structure.
  - boulos 6y ago
    Interesting. I think that argument is “people can’t usefully agree on common abstractions, why bother forcing a translation”. And so for users of scene graphs, it’s basically “well, we can all agree there’s a basic graph and bounding boxes are a thing” so you’re not giving up much. But for UI, is the argument that it’s often more like drawing/painting, so you spend more cycles (in both mental and processing senses) on trying to express the thing you already knew you wanted to achieve?
- learc83 6y ago
  I don’t know. Unity is completely redoing their UI stack, so I don’t think even they are happy with the current incarnation.
  - stephc_int13 6y ago
    It is absurdly easy to make a sluggish and extremely memory inefficient UI with the old Unity API. I've seen that made many times.
deleted 6y ago
[deleted]
moonchild 6y ago
I make a retained mode-ish gui library. (I am not sure if it falls exactly into the tradition of retained mode gui, but it involves constructing trees of widget objects, so—close enough.) Its design is fundamentally much simpler than that of the immediate mode UI libraries. There is no need for implicit state. There is no need for layout to be a core feature of the library; layout can be 'just another widget'. This opens up the possibility for different types of layouts: constraint-based, grid-based, etc. It's even possible to implement an imgui on top of this fairly easily; but not the other way around, because imgui has implicit global state to manage. Imgui is not slow, especially with GPUs. But it definitely loses on the simplicity front.

raphlinus 6y ago

How do you plan to address accessibility using imgui? As I argue in the post, the model of an imgui-like API for clear expression of app logic, backed by a more traditional retained render object tree, has the potential to give you the best of both worlds.

stephc_int13 6y ago
>How do you plan to address accessibility using imgui? I am not sure I understand this question.
- raphlinus 6y ago
  > I am not sure I understand this question. I rest my case.
  - boulos 6y ago
    I think it’s fair to say that accessibility is an ambiguous term. Which one did you mean? (My charitable guess was a11y in the screen reader, color blindness, font size, etc. sense)
    - raphlinus 6y ago
      You are right, but I couldn't resist the opportunity. Yes, just "accessibility" is a fairly broad term, but I do mean it in the context of assistive technology such as screen readers and navigation by keyboard and devices like directional controllers. I should note, we haven't actually done any of that work in Druid yet (we're struggling with scope as it is), but trying to think ahead. I believe that having a retained tree will help enormously with building this, and that, in the meantime, tab focus navigation is a good proxy for at least some of the problems we expect to face.
      - stephc_int13 6y ago
        It is a bit of work but navigation with keyboard/gamepad is possible with an immediate-mode GUI, as demonstrated by the excellent work of Omar Cornut (Dear Imgui) You have to set a compilation flag to enable the keyboard/gamepad navigation, if you missed it from the source code.
        akiselev 6y ago
        Basic navigation with keyboard/gamepad is a trivial case compared to screen reader accessibility and navigation. Take a look at the Windows Narrator keyboard shortcuts [1] like "Next Heading" or "Go to next landmark/region" or "Read row header" - these require integration with the system UI framework or the accessibility APIs directly to provide extra semantic information about what's drawn on screen. Even with integration into system accessibility APIs, Narrator (and other screen readers) performs a lot of heuristics to improve the user experience. These heuristics depend on a separate tree data structure [2] and how it changes over time, so in order to implement even the most basic accessibility, IMGUI would have to retain that information instead of throwing it away every frame. Component identity is crucial to how most (all?) screen readers work today and it's a happy accident that it also allows an efficient hybrid retained-immediate mode architecture. I don't know if it has been fixed or not, but a year or two ago you could completely nuke the Windows screen reader just by assigning React components randomly generated keys so that the DOM nodes would be destroyed and recreated every redraw. [1] https://dequeuniversity.com/screenreaders/narrator-keyboard-shortcuts https://dequeuniversity.com/screenreaders/narrator-keyboard-... [2] https://docs.microsoft.com/en-us/windows/win32/winauto/inspect-objects https://docs.microsoft.com/en-us/windows/win32/winauto/inspe...
- boulos 6y ago
  I assume Raph meant in the screen reader sense of the term.
  - stephc_int13 6y ago
    I don't see any limitations on this side, intuitively, but I might not know all the intricacies tied to this kind of implementation. If the ability to add metadata to any widget is not sufficient then I don't know.
    - boulos 6y ago
      Yeah, I’m not an a11y person (nor a UI or even 2d graphics person). What about the “zoom under my cursor” type things? That seems much harder (though not impossible) if your interface to clients is “just draw directly”. Though thinking about it, it’s actually probably what macOS does: as a post process, take your frame buffer and then magnify the stuff in a region of interest and re-render that on top. You’re winning me over :). Edit: I assume your metadata solution is “if someone asks, the text under this region of interest says < string >”.
      - stephc_int13 6y ago
        There is a common misconception about immediate-mode APIs, there are in practice very few implementations where rendering is done directly. A structure is built every frame with all the data to render later, as a batch. And of course if you need a magnify lense the obvious way would be to render it as a post process.
liuliu 6y ago
And current crop of imgui libraries already maintaining states cross updates (after all, that is how you can implement things as simple as drag & drop). However, the reason you need to have a retained render object tree somewhere for accessibility is due to its interaction mode (mostly request & response) different from GUI (which renders at 60~120fps). The mismatch in interactivity thus compensated by the retained render mode. It felt almost the old days where your UI is derived strictly from business logic with fixed set of widgets is better for accessibility than our new crop of apps where you have more gestures and shortcuts. It is no wonder that imgui shines if you need more gestures and eye-candy animations.
- mwcampbell 6y ago
  The mismatch between the periodic from-scratch rendering (push) approach of IMGUI and the request/response (pull) model of current accessibility APIs is a tractable problem. Imagine a streaming protocol that allows the application or GUI toolkit to emit the contents of an accessibility tree, either in whole or as incremental updates, whenever it renders. An IMGUI could produce this data each time it renders, even if it doesn't have a retained tree in memory. It would be up to the consumer of the protocol to create and update a retained tree from this stream. That consumer could be a library running inside the same application, which could then implement current accessibility APIs such as UI Automation. Or, if the protocol caught on and was supported by assistive technologies (e.g. screen readers), those ATs themselves could be the consumers. Indeed, I believe that applications pushing their accessibility information to ATs, which can then create and update their own retained trees, would be a better alternative to the pull-based request/response model that you mention. IMO, the bigger problem is the one that Raph pointed out: widgets (including simple ones like static text) need to have stable identities throughout their lifetimes, even as their state, contents, and locations change. It seems to me that IMGUIs, or at least some of them, make this particularly difficult.
  - liuliu 6y ago
    Yes, that would work. But that would be a departure from existing imgui paradigm where the output of imgui library is a 2d bitmap. At that point, it gets really close to the vDOM territory ... As for the stable identities, yes, it is important. But I also don't see much of a problem with explicit Key: the UI components ultimately will have map back into a data object, which should have enough contextual information to provide the said Key.

adamnemecek 6y ago

Immediate rendering consumes an insane amount of CPU.

stephc_int13 6y ago
Not in my book. I am running a prototype immediate mode gui that I built for my yet unreleased engine and the frame time is staying below 500ns for a fullscreen app with many widgets and layers. The whole thing is built on C + Vulkan but is not entirely optimized yet. And this is running on a decent but not extremely fast computer. (An Intel NUC from 2018)
- cycloptic 6y ago
  Now take that same code and run it on a mobile device. Can you say for certain that it's making optimal use of power? And if not, what changes would you have to make so it does?
  - stephc_int13 6y ago
    I'll do that later but I don't think it will lead to extreme CPU consumption. There are many ways to improve performances of idle/almost idle ui. Caching can be useful for large blocks of text as it tend to be the slowest thing to render. But it is preferable to incrementally optimize/add complexity and check assumptions and not make large architectural decisions ahead of time. One way to reason about it is to think in terms of pure data transformations.
    - cycloptic 6y ago
      If you're caching data, you now have stored state and it's no longer a pure data transformation. You will run into these performance problems with any large data set, not just blocks of text.
- adamnemecek 6y ago
  Can I see a screenshot?

boulos 6y ago

Hmm. I’m a big fan of immediate mode myself, because it’s kind of amazing how much you can actually push through via brute force. However, wouldn’t you agree that Unity, Open Scene Graph, etc. are successful counter arguments? I would say it depends on your audience. The bottom layer of the stack might be a very simple retained mode, or fully brute force immediate, but it’s not clear to me that a “UI framework” for application developers should be immediate mode.

stephc_int13 6y ago
In my opinion, the reason why immediate-mode API are so efficient and practical is because it helps to avoid structure duplication. An UI, from an abstract point of view, is nothing but a big structure. When you're writing an application or a game, you very often already have an existing structure containing everything you need. Take a simple example : a leaderboard. Using anything but an immediate-mode API will force you to create a second, complex structure to hold this leaderboard, and a lot of glue code to copy data to and from this structure.
- boulos 6y ago
  Interesting. I think that argument is “people can’t usefully agree on common abstractions, why bother forcing a translation”. And so for users of scene graphs, it’s basically “well, we can all agree there’s a basic graph and bounding boxes are a thing” so you’re not giving up much. But for UI, is the argument that it’s often more like drawing/painting, so you spend more cycles (in both mental and processing senses) on trying to express the thing you already knew you wanted to achieve?
learc83 6y ago
I don’t know. Unity is completely redoing their UI stack, so I don’t think even they are happy with the current incarnation.
- stephc_int13 6y ago
  It is absurdly easy to make a sluggish and extremely memory inefficient UI with the old Unity API. I've seen that made many times.

deleted 6y ago

[deleted]

moonchild 6y ago

I make a retained mode-ish gui library. (I am not sure if it falls exactly into the tradition of retained mode gui, but it involves constructing trees of widget objects, so—close enough.) Its design is fundamentally much simpler than that of the immediate mode UI libraries. There is no need for implicit state. There is no need for layout to be a core feature of the library; layout can be 'just another widget'. This opens up the possibility for different types of layouts: constraint-based, grid-based, etc. It's even possible to implement an imgui on top of this fairly easily; but not the other way around, because imgui has implicit global state to manage. Imgui is not slow, especially with GPUs. But it definitely loses on the simplicity front.