5 ms·

I used and implemented immediate mode GUIs, and I think it’s only good for simple stuff. Accessibility is hard. The OS needs to know visual tree to be able to

I used and implemented immediate mode GUIs, and I think it’s only good for simple stuff. Accessibility is hard. The OS needs to know visual tree to be able to conclude things like “this rectangle is a clickable button with Hello World text”. Complex layouts are hard. Desktop software have been doing fluid layouts decades before the term was coined for web apps, because different screen resolutions, and because some windows are user-resizable. Layout can be expensive (one reason is measuring pixel size of text), you want to reuse the data between frames. Animations are hard. Users expect them because smartphones introduced them back in 2007: https://streamable.com/okvhl https://streamable.com/okvhl Note the kinetic scrolling, and soft “bumps” when trying to scroll to the end. Drag and drop is very hard.

flohofwoe 6y ago
Apart from accessibility, isn't this mostly caused by the "immediate mode API" versus "immediate mode implementation" confusion I described in another sub-thread? There's nothing in the idea that forbids immediate mode UI frameworks from keeping any amount of internal state between frames to keep track of changes over time (like animations or drag'n'drop actions), the difference to traditional UI frameworks is just that this persistent state tree is hidden from the outside world. Layout problems can be solved by running several passes over the UI description before rendering happens. For accessibility, the ball is mainly in the court of the operating system and browser vendors. There need to be accessibility APIs which let user interface frameworks connect to screen readers and other accessibility features (this is not a problem limited to immediate mode UIs, but custom UIs in general).
- enqk 6y ago
  these APIs are UI Automation on windows and NSAccessibility on macos
- Const-me 6y ago
  > There's nothing in the idea that forbids immediate mode UI frameworks from keeping any amount of internal state between frames to keep track of changes over time When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame, or a new one. You’re simply not providing that data to the framework. It’s often good enough for rendering, they don’t really care if it’s an old or new button as long as the GPU renders identical pixels. When you need state however (animations certainly do), the approach is no longer useful. A framework might use heuristics like match text of the button or relative order of things, but none of them is reliable enough. Buttons may change text, they may be dynamically shown or hidden, things may be reordered in runtime. > Layout problems can be solved by running several passes over the UI description before rendering happens. You’re correct that it’s technically solvable. It’s just becomes computationally expensive for complex GUIs. You can’t reliably attach state to visual elements, gonna have to re-measure them every frame. > There need to be accessibility APIs which let user interface frameworks connect to screen readers Windows has it for decades now. For Win32 everything is automatic and implemented by the OS. Good custom-painted GUI frameworks like WPF or UWP handle WM_GETOBJECT message and return IAccessible COM interface. That COM interface implements reflection of the complete GUI, exposing an objects hierarchy. I’m not saying it’s impossible to do with immediate rendering, just very hard without explicit visual tree that persists through the lifetime of the window.
  - shrimpx 6y ago
    > When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame, or a new one. Can you solve this by adding stable IDs to the API, so you'd call `ImGui::Button("my-button-id", button_text)`?
    - Const-me 6y ago
      To consistently assign these IDs you gonna have a tree on your side of the API. You’ll then have two visual trees, one on your side on the API, another one on the framework’s side of the API. And then you gonna be debugging things. Who and when should destroy backend visual nodes? If you won’t use an ID in some frame, does it indicate the backend should drop the related state? What if an animation is still playing on the element in question? What if the missing control re-appears later i.e. was only hidden temporarily? What should backend do if you reuse same ID for different things? One can implement simple stuff using any approach, immediate GUI is often the simplest of them. It’s when use cases are complicated you need a full-blown retrained mode OO framework. Win32, HTML DOM, MS XAML, QT, UIKit are all implemented that way, that’s not a coincidence.
  - flohofwoe 6y ago
    > When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame ImGui does have stable widget ids, otherwise many features which require keeping state across frames wouldn't work. In case of your button example, the widget id is hashed from the label string "Hello World": https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114c034b24eb5bff149e7/imgui_widgets.cpp#L673 https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114... This is just the default behaviour, there's also a convention to define the id separately from the visible label string behind a '##': https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114c034b24eb5bff149e7/imgui_demo.cpp#L1646 https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114... Of course other immediate mode UIs might have a different way to define widget ids.
Chris_Newton 6y ago
When building more complicated UIs, an architectural pattern I have often found useful is to have two distinct levels of state/data behind the rendering code, so my hierarchy looks something like this: 1. Application data 2. Presentation data 3. Presentation rendering The first is the single source of truth for your application state, what we often call a “model” or “store”. It’s where you represent the data from your problem domain, which is usually also the data that needs to be persistent. The second is where you collect any additional data needed for a particular way of presenting some or all of your application data. This can come from the current state of the view (list sort order, current page for a paginated table, position and zoom level over a map, etc.) or the underlying application data (building a sorted version of a list, laying out a diagram, etc.) or any combination of the two (finding the top 10 best sellers from the application data, according to the user’s current sort order set in the UI). This is often a relatively simple part of the system, but there is no reason it has to be: it could just as well include setting up a complicated scene for rendering, or co-ordinating a complicated animation. The final part is the rendering code, which is a translation from the application and presentation data into whatever presentation is required. There isn’t any complicated logic here, and usually no state is being maintained at this level either. The data required for rendering all comes ready-prepared from the lower layers. Any interactions that should change the view or application state are immediately passed down to the lower layers to be handled. The important idea is that everything you would do to keep things organised around the application data also applies to the presentation data. Each can expose its current state relatively directly for reading by any part of the system that needs to know. Each can require changes of state to be made through a defined interface, which might be some sort of command/action handler pattern to keep the design open and flexible. Each can be observable, so other parts of the system can react to changes in its state. It just happens that now, instead of a single cycle where application data gets rendered and changes from the UI get passed back to the application data layer, we have two cycles. One goes from application data through presentation data to rendering, with changes going back to the application data layer. The other just goes from the presentation data to the rendering and back. I have found this kind of architecture “plays nicely” with almost any other requirements. For example, data sync with a server if our GUI is the front end of a web app can easily be handled in a separate module elsewhere in the system. It can observe the application data to know when to upload changes. It can update the application data according to any downloaded information via the usual interface for changing the state. I have also found this kind of architecture to be very flexible and to fit with almost any choice of libraries for things like state modelling, server comms, rendering the actual view, etc. Or, if your requirements are either too simple to justify bringing in those dependencies or too complicated for a ready-made library to do what you need, you have a systematic overall structure in place to implement whatever you need directly, using all the same software design and scaling techniques you normally would.

flohofwoe 6y ago

Apart from accessibility, isn't this mostly caused by the "immediate mode API" versus "immediate mode implementation" confusion I described in another sub-thread? There's nothing in the idea that forbids immediate mode UI frameworks from keeping any amount of internal state between frames to keep track of changes over time (like animations or drag'n'drop actions), the difference to traditional UI frameworks is just that this persistent state tree is hidden from the outside world. Layout problems can be solved by running several passes over the UI description before rendering happens. For accessibility, the ball is mainly in the court of the operating system and browser vendors. There need to be accessibility APIs which let user interface frameworks connect to screen readers and other accessibility features (this is not a problem limited to immediate mode UIs, but custom UIs in general).

enqk 6y ago
these APIs are UI Automation on windows and NSAccessibility on macos
Const-me 6y ago
> There's nothing in the idea that forbids immediate mode UI frameworks from keeping any amount of internal state between frames to keep track of changes over time When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame, or a new one. You’re simply not providing that data to the framework. It’s often good enough for rendering, they don’t really care if it’s an old or new button as long as the GPU renders identical pixels. When you need state however (animations certainly do), the approach is no longer useful. A framework might use heuristics like match text of the button or relative order of things, but none of them is reliable enough. Buttons may change text, they may be dynamically shown or hidden, things may be reordered in runtime. > Layout problems can be solved by running several passes over the UI description before rendering happens. You’re correct that it’s technically solvable. It’s just becomes computationally expensive for complex GUIs. You can’t reliably attach state to visual elements, gonna have to re-measure them every frame. > There need to be accessibility APIs which let user interface frameworks connect to screen readers Windows has it for decades now. For Win32 everything is automatic and implemented by the OS. Good custom-painted GUI frameworks like WPF or UWP handle WM_GETOBJECT message and return IAccessible COM interface. That COM interface implements reflection of the complete GUI, exposing an objects hierarchy. I’m not saying it’s impossible to do with immediate rendering, just very hard without explicit visual tree that persists through the lifetime of the window.
- shrimpx 6y ago
  > When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame, or a new one. Can you solve this by adding stable IDs to the API, so you'd call `ImGui::Button("my-button-id", button_text)`?
  - Const-me 6y ago
    To consistently assign these IDs you gonna have a tree on your side of the API. You’ll then have two visual trees, one on your side on the API, another one on the framework’s side of the API. And then you gonna be debugging things. Who and when should destroy backend visual nodes? If you won’t use an ID in some frame, does it indicate the backend should drop the related state? What if an animation is still playing on the element in question? What if the missing control re-appears later i.e. was only hidden temporarily? What should backend do if you reuse same ID for different things? One can implement simple stuff using any approach, immediate GUI is often the simplest of them. It’s when use cases are complicated you need a full-blown retrained mode OO framework. Win32, HTML DOM, MS XAML, QT, UIKit are all implemented that way, that’s not a coincidence.
- flohofwoe 6y ago
  > When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame ImGui does have stable widget ids, otherwise many features which require keeping state across frames wouldn't work. In case of your button example, the widget id is hashed from the label string "Hello World": https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114c034b24eb5bff149e7/imgui_widgets.cpp#L673 https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114... This is just the default behaviour, there's also a convention to define the id separately from the visible label string behind a '##': https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114c034b24eb5bff149e7/imgui_demo.cpp#L1646 https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114... Of course other immediate mode UIs might have a different way to define widget ids.

Chris_Newton 6y ago

When building more complicated UIs, an architectural pattern I have often found useful is to have two distinct levels of state/data behind the rendering code, so my hierarchy looks something like this: 1. Application data 2. Presentation data 3. Presentation rendering The first is the single source of truth for your application state, what we often call a “model” or “store”. It’s where you represent the data from your problem domain, which is usually also the data that needs to be persistent. The second is where you collect any additional data needed for a particular way of presenting some or all of your application data. This can come from the current state of the view (list sort order, current page for a paginated table, position and zoom level over a map, etc.) or the underlying application data (building a sorted version of a list, laying out a diagram, etc.) or any combination of the two (finding the top 10 best sellers from the application data, according to the user’s current sort order set in the UI). This is often a relatively simple part of the system, but there is no reason it has to be: it could just as well include setting up a complicated scene for rendering, or co-ordinating a complicated animation. The final part is the rendering code, which is a translation from the application and presentation data into whatever presentation is required. There isn’t any complicated logic here, and usually no state is being maintained at this level either. The data required for rendering all comes ready-prepared from the lower layers. Any interactions that should change the view or application state are immediately passed down to the lower layers to be handled. The important idea is that everything you would do to keep things organised around the application data also applies to the presentation data. Each can expose its current state relatively directly for reading by any part of the system that needs to know. Each can require changes of state to be made through a defined interface, which might be some sort of command/action handler pattern to keep the design open and flexible. Each can be observable, so other parts of the system can react to changes in its state. It just happens that now, instead of a single cycle where application data gets rendered and changes from the UI get passed back to the application data layer, we have two cycles. One goes from application data through presentation data to rendering, with changes going back to the application data layer. The other just goes from the presentation data to the rendering and back. I have found this kind of architecture “plays nicely” with almost any other requirements. For example, data sync with a server if our GUI is the front end of a web app can easily be handled in a separate module elsewhere in the system. It can observe the application data to know when to upload changes. It can update the application data according to any downloaded information via the usual interface for changing the state. I have also found this kind of architecture to be very flexible and to fit with almost any choice of libraries for things like state modelling, server comms, rendering the actual view, etc. Or, if your requirements are either too simple to justify bringing in those dependencies or too complicated for a ready-made library to do what you need, you have a systematic overall structure in place to implement whatever you need directly, using all the same software design and scaling techniques you normally would.