16 ms·

Immediate Mode GUI

185 points by sytelus 8y ago

60654 8y ago
Oh no, not again! (Just kidding :) ) Immediate mode UIs come up every now and then, especially in games, and for relatively simple things they're a neat idea (imagine not retaining any extra view state!) But - and there's a big "but" coming up - it does not scale to complex and stateful UIs. The problem is that, for games that have complex needs - think simulation and strategy games, not racing or arcade - complex UIs are needed to express complex relationships between underlying data and the relevant UX flows. But how do you do that in IMGUI? This information about UI state, layout, etc. is not in the UI anymore, but it has to live somewhere - so you end up having to build your own layout and UI management stuff, and basically duplicate what a RMGUI library would offer, but now with less generality and potentially more bugs. For the probably best known example, Unity used to have an IMGUI [1] but they've actively moved away from it. It's still there in vestigial form, the API is not removed, but game developers don't really use it. [1] https://docs.unity3d.com/Manual/GUIScriptingGuide.html https://docs.unity3d.com/Manual/GUIScriptingGuide.html PS. And there's also the lack of visual layout tools, which is directly related to IMGUI's reliance on code to produce layout.
- mellinoe 8y ago
  > For the probably best known example, Unity used to have an IMGUI [1] but they've actively moved away from it. It's still there in vestigial form, the API is not removed, but game developers don't really use it. It's true that the newer retained mode GUI has supplanted the older IMGUI for "game UI" (e.g. what users interact with when playing an actual game built with Unity). On the other hand, the Unity editor itself is still built entirely with their immediate-mode GUI library, and I haven't heard of any plan to move away from that. Building "editor-like" tools (including custom user tools) is still significantly easier to accomplish using immediate-mode libraries.
  - jayd16 8y ago
    In Unity you're still forced to use the IMGUI for making editor tools. In fact their new xml layout prototype still uses the IMGUI to draw an instantiated layout! What a mess...
  - 60654 8y ago
    Oh, they're switching to RMGUI in the editor as well, though more slowly. More info here: https://blogs.unity3d.com/2019/04/23/whats-new-with-uielements-in-2019-1/ https://blogs.unity3d.com/2019/04/23/whats-new-with-uielemen...
  - vardump 8y ago
    Interesting (even a surprising) choice Unity is making. What are the performance implications of this change?
- overgard 8y ago
  I wouldn’t use one for a production UI, but they’re super handy for building light weight debugging tools.
- DonHopkins 8y ago
  Immediate mode GUI's don't let you apply useful object oriented programming techniques like subclassing and prefabs (persistence) to implement and configure custom components. Properly object oriented retained mode GUIs let you subclass components to implement custom views and event handling, and develop and publish reusable prefabs and libraries of pre-configured, stylized, specialized component.
  - chii 8y ago
    > don't let you apply useful object oriented programming techniques [to get] reusable libraries of specialized component. But immediate mode GUI lets you compose very well. It's a bottoms up technique of programming, where instead of following a framework established by the library author, you are given low level building blocks. You will find it hard to mix and match two retained mode GUI libraries unless they all follow a common set of standard apis (e.g., java swing). But with immediate mode GUI, it's actually easy to mix and match (as the immedate mode gui has no need to know what framework you're working under nor need to make any assumptions about threading or api).
    - DonHopkins 8y ago
      I don't know what you mean about easily composing immediate mode API's, or agree that different immediate mode APIs don't need to know about each other. They depend on a lot of behind-the-scenes implicit hidden state (practically a hidden shadow-DOM-like retained mode), so I suspect in many cases they'd walk all over each other (especially text editors dealing with input focus and keyboard navigation). Unity3D has two immediate mode APIs, one for the editor and one for runtime, and they sure don't mix. I've implemented various adaptor wrappers for composing different retained mode API's myself, including wrapping The NeWS Toolkit OPEN LOOK widgets in HyperLook widgets, which I used for the first Unix port of SimCity to HyperLook/NeWS. (The OPEN LOOK pin-up menus and pie menus and buttons and sliders are implemented in TNT, but I wrapped them for HyperLook (which has its own persistence system and class hierarchy), so you can dynamically create and edit them at runtime with property sheets and script editors, ala HyperCard.) https://medium.com/@donhopkins/hyperlook-nee-hypernews-nee-goodnews-99f411e58ce4 https://medium.com/@donhopkins/hyperlook-nee-hypernews-nee-g... For X11/NeWS at Sun, we even made an ICCCM X11 window manager that wrapped X11 windows in NeWS tabbed window frames with pie menus! (The X11 window manager was written in object oriented "retained mode" PostScript code, subclassing standard and custom NeWS window and menu widgets, like tabbed window frames and pie menus. While PostScript has an immediate mode drawing API, it was driven by extensible "retained mode" widget instances and classes represented by PostScript dictionaries, pushed on the dict stack.) Open Window Manager PostScript source: https://www.donhopkins.com/home/archive/NeWS/owm.ps.txt https://www.donhopkins.com/home/archive/NeWS/owm.ps.txt Tabbed window extension: https://donhopkins.com/home/archive/NeWS/win/tab.ps.txt https://donhopkins.com/home/archive/NeWS/win/tab.ps.txt Discussion of problems with NeWS: https://news.ycombinator.com/item?id=15327339 https://news.ycombinator.com/item?id=15327339 Political discussion from the Window System Wars, about Sun's problems merging X11 and NeWS: https://donhopkins.com/home/archive/NeWS/sevans.txt https://donhopkins.com/home/archive/NeWS/sevans.txt More recently (and more successfully), I've wrapped HTML components and floated them over OpenLaszlo/Flash (not so recently) and Unity3D (quite recently) retained mode user interface layouts in the web browser. For example, I'm embedding an ACE code editor in a Unity3D retained mode user interface running in WebGL, that falls back to a regular TextMeshPro input field on other builds. There's no way you could do that practically with an immediate mode API. WebGLAce_TMP_InputField.cs (C# retained mode GUI TMP_InputField subclass): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/Libraries/WebGLAce/Scripts/WebGLAce_TMP_InputField.cs https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... WebGLAce.cs (C# P/Invoke Unity extension wrapper): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/Libraries/WebGLAce/Scripts/WebGLAce.cs https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... WebGLAce.jslib (JavaScript library Unity extension, compiled by Unity to WebAssembly, converts between C#/WebAssembly and JavaScript data types and thunks messages back and forth between the browser and the Unity3D app): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/Libraries/WebGLAce/Plugins/WebGL/WebGLAce.jslib https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... WebGLAce.jss (ACE code editor adaptor, pure JS source loaded by the browser, as well as ACE JS libraries themselves): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/StreamingAssets/WebGLAce/WebGLAce.jss https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... Notice how WebGLAce_TMP_InputField simply subclasses the standard TextMeshPro TMP_InputField, and adds some more hidden state like the editorID and editorConfigScript of the corresponding ACE code editor running in the web browser. All that is hidden from the code using it, so it doesn't need to come up with another place to store that editorID and editorConfigScript, or even know if it's getting an Ace editor under WebGL or falling back to TextMeshPro on other builds. That's what I mean by leveraging OOP to subclass existing components and hide internal state, which you can't do in immediate mode. (My plan for non-WebGL builds that aren't already running inside a web browser, is to create an embedded web browser just to run the Ace code editor, and the C# WebGLAce_TMP_InputField API will remain the same, and it will just work transparently on different platforms. I want to support live coding Unity apps in JavaScript, but there just isn't anything for Unity that approaches ACE for editing code, so it's worth going through all the trouble to make an adaptor.) How would you write a functional extension for Unity3D's old immediate mode GUI that let you embed an ACE code editor in a Unity WebGL app? What would the API and state management even look like? How could you make it platform independent? And even if you solved all of those problems with an immediate mode API, by its very nature it still wouldn't enable you to build GUIs in the interface editor or store them in prefabs, and you'd still have to recompile you app (which can take half an hour with Unity) every time you wanted to tweak the user interface (which is why I like programming Unity apps in JavaScript as much as possible). UnityJS Unity3D / JavaScript bridge architecture: https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy.txt https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy... UnityJS pie menu component for Unity3D, written in JavaScript, drawn with canvas, using TextMesh Pro labels: https://github.com/SimHacker/UnityJS/blob/master/Libraries/UnityJS/StreamingAssets/UnityJS/pie.jss https://github.com/SimHacker/UnityJS/blob/master/Libraries/U... Uses a Unity3D pie menu tracker to translate low level input to high level events, written in C#: https://github.com/SimHacker/UnityJS/blob/master/Libraries/UnityJS/Libraries/UnityJS/Scripts/BridgeObject/PieTracker.cs https://github.com/SimHacker/UnityJS/blob/master/Libraries/U... Of course you can take it too far, trying to please everyone by combining the worst of all possible systems and then giving it a ridiculous sounding name (see "MoOLIT"), but we're discussing whether it's possible and practical, not whether it's desirable. And even if it's not desirable, it ends up being necessary. https://en.wikipedia.org/wiki/MoOLIT https://en.wikipedia.org/wiki/MoOLIT http://nova.polymtl.ca/~coyote/open-look/01-general/faq-doc-1.html http://nova.polymtl.ca/~coyote/open-look/01-general/faq-doc-... Embedding object oriented "retained mode" widgets with different APIs inside of each other is old hat and common for backwards compatibility. Whenever you write a new GUI toolkit, embedding widgets from the last toolkit is one of the first things you do (including recursively embedding widgets from the toolkit-before-last). Concrete example: Microsoft lets you embed old fashioned OLE controls in Windows Forms / Presentation Foundation applications, which might be implemented in MFC themselves. And MFC is all about embedding old Win32 widgets implemented in C, and newer OLE components implemented in Visual Basic or whatever, in venerable C++ MFC user interfaces. Say what you want about how much Win32/MFC/OLE/WF/WPF sucks, and I'll wholeheartedly agree, but if you're running Windows, you probably have widgets on your screen using several different retained mode APIs embedded more than two levels deep right now. https://en.wikipedia.org/wiki/Windows_Forms#Architecture https://en.wikipedia.org/wiki/Windows_Forms#Architecture
- jpochtar 8y ago
  I mean, React is essentially a hack to get an immediate mode API on top of a retain mode platform (DOM), because the immediate mode API scales better to handle complex state. I'd push that yes, you do have to maintain your state externally in an immediate mode UI, but when you have a lot of "state" that's really just a function of other state, that's exactly what you want.
  - btown 8y ago
    The interesting thing about React is actually that it encapsulates a lot of state retention inside an immediate-style API. state/setState (and now state hooks) are the unsung heroes of the ecosystem, because they let you focus on only the business-relevant global state, making it natural for a dropdown menu to handle its own open/closed state and insulate its user from that complexity. And many intermediate mode libraries gloss over this need, so you have to track much more state than you’d like. It’s a fascinating hybrid - rather than “BYO State” where the library does nothing but layout and draw, or “BYO hierarchy” where you need to retain instantiated components, React is “hierarchy tracking included” - all the benefits of persistent components, but tracked by the renderer rather than the caller. It’s an evolutionary step that I think desktop programming would do well to adopt and continue to iterate on.
    - deleted 8y ago
      [deleted]
  - zamalek 8y ago
    React is half-and-half. It is immediate mode in usage, but the data structure (vdom) is what you'd expect from retained.
- dan-robertson 8y ago
  Well with retained mode one still has to handle a bunch of application state, and handle keeping that state synchronised with the view. Immediate mode typically forces you to manage the view state too but it’s typically easier to keep it synchronised with the application state that way. I think the following is an interesting example here: your application state is some type, Config which would correspond to some boring components, e.g. text boxes and radio buttons, but the interface you want to present looks like: [X] default settings Or [ ] default settings ... Now the type of data you want by the end of the form looks like (eg Haskell) Maybe Config Where Nothing means “do the default thing”. But the state of the view has to look more like: Bool * Config Because you want to remember what options the user had changed in case they change their mind a few times about using the default state. And now maybe you want a state like (mutable) Bool * Config * (mutable) Config Because the default state is not static and you want to make fields bold if they are not default. And so now you still have to do the complicated conversion from view to application state (but at least you don’t have to go back). I suppose this is like the very early web dev style of having no direct state and directly reading values from controls, but at least in this world one has control over how the view state is managed and represented so it can be easier to avoid getting into invalid states.
- naasking 8y ago
  > But how do you do that in IMGUI? This information about UI state, layout, etc. is not in the UI anymore, but it has to live somewhere - so you end up having to build your own layout and UI management stuff Immediate mode is about the interface to rendering the UI, not necessarily about the implementation (although some people conflate these). So your UI elements can be stateful and retain their own state, but they can be released once they are no longer rendered in the current frame.
- cmrdporcupine 8y ago
  Honestly -- given the amount of time I've spent adapting my own data model classes and structures to the straightjacket of various GUI or web toolkits -- not having the toolkit enforce its normative lifestyle assumptions on data structure feels like a feature to me. I'd rather manage the state and relationships myself.
  - jayd16 8y ago
    >given the amount of time I've spent adapting my own data model classes and structures to the straightjacket of various GUI This is why every tells you to use MVC or some variant that separates out the UI from the data. IMGUI lets you bend the UI to the data instead but the real lesson is to decouple the view and the model.
    - cmrdporcupine 8y ago
      I'm very familiar with MVC -- I've been in the industry for 20+ years -- but so far disgusted with the way pretty much every GUI framework expects you to handle this. In reality I may already have a data model, but often have to wrap _another_ model around my model, just to satisfy the GUI framework's constraints on what it thinks a model needs to inherit from, etc. Recently ran into this with Qt. Very frustrating.
      - jayd16 8y ago
        The trick is to just accept that the UI model, the business logic model and the denormalized db model all have different goals and often times should be decoupled and adapted.
        cmrdporcupine 8y ago
        Which is why it seems almost every project with a complicated UI turns into a bloated pile of semi-duplicated boiler plate 'model' classes with little business logic. Clearly, MVC as the industry understands it, has a problem. Which is why the original Smalltalk people -- who invented MVC -- mostly switched to Self's Morphic framework in their later projects (Squeak, etc.)
- DonHopkins 8y ago
  Here's a demo of Unity3D's new retained mode UIElements user interface for the editor, that lets you wrap old immediate mode IMGUI code in IMGUIContainer wrappers! https://youtu.be/MNNURw0LeoQ?t=17m37s https://youtu.be/MNNURw0LeoQ?t=17m37s
deleted 8y ago
[deleted]
sytelus 8y ago
TLDR; Typically we build GUIs using event driven callbacks. This causes problems in managing life times, dynamic layouts, threading issues etc. The idea behind immediate mode GUI is that you get a frame and in each frame you create/update layout, check for user actions - just like you do in render frame in game loop. So there are no call backs and you don't have to worry about issues arising from event-driven code.
- jboles 8y ago
  Does this pretty-much rule out (without a huge amount of effort) nonmodal GUIs? It seems without callbacks, you would need to roll your own window management, Z-ordering of overlapping UI elements, state transition validation (e.g. clicking on the parent dialogue’s Cancel button while a child window is still open), ...
  - jackfraser 8y ago
    There's no reason that that can't all be done as part of the underlying IMGUI implementation; what you're working with above that is essentially a DSL that renders out to a workable set of controls. Layout management and the actual rendered graphics / animations are handled by the code you don't have to see. If anything, this is actually a lot like Web 1.0 programming: server side rendering, often using templates that end up looking a lot like a DSL. Each page view is rendered based on those templates against data fetched in advance of the output. You can get pretty deep with that kind of thing, though of course the page refresh time was what killed this otherwise fairly nice style of writing an application. Users expected actions like sorting a column to be as fast in the browser as it was in a Windows 95-era file explorer or Excel, and it simply wasn't happening. (Imagine if first-class client-side-interactive UI components like a sortable table, tabbed panel, etc. like what XUL provides had been in HTML in the 90s. We wouldn't need half the Javascript bullshit people have to use now to replicate that functionality half-assedly...)
  - mmozeiko 8y ago
    Here's an browser demo for dear imgui (https://github.com/ocornut/imgui https://github.com/ocornut/imgui) library: https://pbrfrat.com/post/imgui_in_browser.html https://pbrfrat.com/post/imgui_in_browser.html For modal demo check "Popups & modal windows" -> "Modals" example. This is how it looks in C++ code: https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp#L2273 https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp#...
userbinator 8y ago
Some also argue that executing UI logic of every widget is wasteful. This might indeed be a concern when implemented poorly For a game, which is already quite busy even when "idle" (and there are probably other things going on in each frame which take far more CPU and GPU to render), it won't be much of a concern; but with an application where most of the time is spent waiting for user input, spinning the CPU to re-render the GUI every frame is very energy-inefficient.
- gameswithgo 8y ago
  you can work around that problem though.
- frnkng 8y ago
  I don’t think that an imgui app will eat more resources than a comparable electron app. Thus the Performance Argument against any IMGUI is pretty pointless.
  - theamk 8y ago
    I disagree. With all downsides of electron, it does not generally re-layout and re-render entire screen on any mouse move. IMGUI does. So while Electron will lose in memory/disk used, IMGUI will lose in CPU usage (battery life) once the software is complex enough.
    - tokyodude 8y ago
      I disagree with your disagreement :) Electron (and nearly all retained mode GUIs) will execute 10x more code trying not to execute the rendering code. Just skipping all that checking and jumping to the rendering code is arguably a win in many many cases. To put some empirical data on it, game devs use ImGUIs because they keep up with perf where as that 10x overhead of retained mode guis creating gui objects, deleting gui objects, marshaling data into and out of gui objects, adding and removing event handlers and callbacks, and then trying to write a bunch more code to optimize all of that never leads to a GUI that is performant enough for the GUIs needed for game dev. I think if you check actual apps, especially phone apps, you'll find it's not so clear which one wins. Phone apps almost always move in relation to the users fingers meaning the screen is being re-rendered. Usually new GUI elements are moving on and off the top/bottom of the screen. The overhead of creating/deleting/managing those objects is arguably greater than just re-rendering. The screen is already going to be re-rendered. At best you're getting a few cached textures from your retained mode gui all at the expensive of managing GUI objects. For desktop apps it really depends on the app. It's got nothing to do with the complexity of the UI and everything to do with what the app does. A text editor might be a win for a retained mode GUI. Photoshop/Maya it's less clear. For example Unity is ImGUI and has very complex UI. Much of the UI has to respond in realtime to the state of the user's document that is changing constantly.
jpnelson 8y ago
Are there any popular implementations of immediate mode GUI for the web? It might be interesting to have one that could then "eject" into a stateful UI for when things start to get more complicated.
- rtpg 8y ago
  I think that React is like 90% of the way there and you would just need to mess around with the VDOM a bit for this. To be honest, given that web render output is a tree instead of flat "render to screen" stuff, writing a nice immediate mode GUI is not obvious. If you're... very clever you could take React and the State stuff, at least conceptually, and get what you want here.
  - kkarakk 8y ago
    wouldn't you need some sort of gpu access to have a smooth UI rendering with immediate gui(in production)?
    - rtpg 8y ago
      This is if you expect to build an immediate mode with the same flat rendering model. imgui's model is "you draw onto pixels". You could create another mental model which is "you draw onto a tree." so instead of saying "drawText(x, y, text)" you would do something like "drawText(domLocation, text)" Granted, this model gets a bit messier, and _some_ might say defeats the purpose of immediate mode. But I think you could still get conciseness benefits on this.
- haskman 8y ago
  I wrote a UI framework called Concur which is very similar to immediate mode UIs and hence very easy to use, but also scales fantastically well to complex UIs. The original code was written in Haskell (https://github.com/ajnsit/concur https://github.com/ajnsit/concur), and then ported to Purescript (https://github.com/ajnsit/purescript-concur https://github.com/ajnsit/purescript-concur), and even Javascript (https://github.com/ajnsit/concur-js https://github.com/ajnsit/concur-js). There's an introduction/tutorial here - https://github.com/ajnsit/concur-documentation/blob/master/README.md https://github.com/ajnsit/concur-documentation/blob/master/R...
Stratoscope 8y ago
Note to authors: please define your terms when you first use them, or provide links to the definitions. Of course I know what a GUI is. I've been building them for decades, and building tools that other developers have used to build GUIs. (I was one of the authors of the "visual" part of Visual Basic.) But I have never before seen the terms "immediate mode GUI" or "retained mode GUI". And I'm pretty sure there are many developers like me who will have no idea what you're talking about. It takes only a moment to add a quick definition or a link the first time you mention a specialized term, and your readers will definitely appreciate it. Thanks! (Edited to remove fluff and make the essential point more clear.)
- kyberias 8y ago
  I first learned about those terms in early DirectX releases (circa 1998). Are there some earlier usages?
  - cmroanirgo 8y ago
    It's where it first came to prominence. The earliest D3D api was immediate mode (meaning, bare metal, talking directly via an api to the hardware). Retained mode is about retaining statefulness and some better abstractions to the underlying needs of the hardware (eg automatic texture resampling, etc). Considering it's an NVidia guy talking: he's been around for a while.
    - pavlov 8y ago
      A minor detail, but the very first version of Direct3D was a retained mode API. In 1995 Microsoft acquired a company called RenderMorphics and turned their Reality Lab [1] API into "Direct3D". The first two versions to ship from Microsoft had both retained and immediate mode APIs, but they dropped the RenderMorphics-based API due to lack of adoption. [1] https://en.wikipedia.org/wiki/Reality_Lab https://en.wikipedia.org/wiki/Reality_Lab
  - Stratoscope 8y ago
    Interesting, thanks! That would explain why I and probably many other readers, who have built plenty of GUIs but have not worked specifically with APIs like DirectX, would not be familiar with the term.
dmix 8y ago
Please add [2015] to title
- soylentgraham 8y ago
  Also, 2011, 2007, 2001, 1996 :) Immediate vs deferred, is just another paradigm that goes back and forth in trends like keyboard input/buffering, deffered/one pass lighting, monoliths vs modules, State vs events, flat vs 3D
BoorishBears 8y ago
I find it ironic that Unity is used as an example... the immediate mode UI was/is considered a joke. Terrible performance, doesn’t scale to UIs more complex than a form (... like the ones many games tend to have) It got to the point where there was a website dedicated to track when we’d get the current non-immediate UI, and Unity actually hired the developer of the leading non-immediate mode UI library in the asset store to work on their own offering.
- tokyodude 8y ago
  I think you're mis-understanding why Unity switched. They only switched their in game GUI system. The entire app of Unity itself uses an ImGUI system and it is massively more complex than any Game UI and runs just fine. The reasons they switched the in game system are many but they have nothing to do with performance.
  - BoorishBears 8y ago
    The editor has none of the constraints the games made with it have... like running on mobile and maintaining 60 FPS while doing it. ... but IMGUI is on the way out there too: https://docs.unity3d.com/Manual/UIElements.html https://docs.unity3d.com/Manual/UIElements.html The old system would stack up draw calls like nobody's bussiness, doubling drawcalls if you used the built in layout tools. OnGUI being called period would be expensive, even when you had properly implemented your rendering logic, so you’d have to disable the entire game object when you didn’t want to draw. The docs for mobile literally used to say not to use the OnGUI stuff during gameplay.
nickflood 8y ago
I've taken part in some WPF application development and I think there are some benefits in immediate-mode rendering of the UI. Particularly when the UI changes faster than a monitor refresh rate or there's too much data complexity and you want the data update and redraw processes to be independent. For my first example, updating controls and overall layout in a WPF application feels much more lightweight than in WinForms because the renderer just shows the result at the next monitor refresh instead of trying to update the layout in realtime or using hacks that disable redraw until you do everything. For the second example, I've used SciChart to display trading data and it again was much more responsive because it replaced a control-based rendering pipeline with just an immediate-mode canvas tgat would render the selected data window once per monitor refresh rate. And I think overall once you hit the performance barrier where you can't just fill a panel with controls and let them render and instead have to make a custom draw routine, the immediate-mode rendering adds this extra smoothness to the UI and gives you a bit more performance headroom.
i_don_t_know 8y ago
While I find the idea appealing, I haven't used them because I haven't figured out how to make immediate GUIs accessible to people who are visually impaired. How do you make them work with screen readers? You're drawing text onto a canvas but you don't retain a copy of the text for the screen reader.
- loup-vaillant 8y ago
  I don't know about screen reader APIs, but… can't you just send the text you're drawing to the screen reader when you drawing it? How do screen readers work anyway?
  - i_don_t_know 8y ago
    Good question. I don't know. When you use the native GUI you get all of that for free. You don't have to do anything special for accessibility. So I always thought that screen readers somehow traversed the window/control hierarchy to find the control that has the focus and ask it for any text that should be read. Maybe you're right and you can send the text to be read directly to the screen reader. Not sure how well that works when your GUI updates every 20ms. You don't want to keep resetting the reader.
    - loup-vaillant 8y ago
      I was thinking maybe resend the stuff only when the text changes, so you'd have to have a cache of some sort, hidden in the library. Anyway, I believe Qt is screen reader compatible, so it must access the relevant API somehow. I'd be surprised if it was the exclusivity of the OS vendor.
bdowling 8y ago
Immediate mode GUIs are an extension of the simple display update loop: (1) update display, (2) check input, (3) goto (1). Immediate mode GUIs add a context parameter to the update function. In the draw context, the update function draws to the screen. In the input context, the update function doesn't draw; instead it checks if mouse clicks or keystrokes should do something. For example, part of the update function might look like this: if (Button("Do That Thing")) doThatThing(); In the draw context, the call to Button will draw a button on the screen and return False. In the input context, the call to Button will check if a mouse click fell within the area it would have drawn. If so, it will return True and the callback will run. An immediate mode GUI is basically the opposite of a typical Model-View-Controller GUI because it deliberately mixes the Model, View, and Controller together into one function. Just one function draws the GUI based on whatever state it's in and whatever the model is. The same function gets reused in a different context to update the GUI state or handle callbacks.
- tokyodude 8y ago
  I'd argue an ImGUI does't mix anything. It's just a library. If you want MVC it's up to you to implement your MVC and use your ImGUI as a library. Retained mode GUIs don't magically make your code MVC and ImGUIs don't magically make your code not MVC. That's entirely up to you. Also ImGUIs are not multiple pass. That's in implementation detail. Unity's ImGUI is multi-pass. One of the most popular ImGUIs (Dear ImGUI) is single pass https://github.com/ocornut/imgui https://github.com/ocornut/imgui
avodonosov 8y ago
Again IMGUI. Several years ago there was a chain of videos and blog posts by different authors. People were drawing two buttons and saying "see how easy it is". In this post the author seems to go further than two buttons, but I'm skeptical. I wish there was a video showing how it works in dynamics - inputs clicked, texts entered, scrollbars scrolled, dialogs dragged around. Or source code available to build and try locally. From the code in the post I suppose the majority of UI is dead - can not be interacted with. I think IMGUI can only be convenient for very simple cases. In real frameworks the lowest level is immediate mode (see WM_PAINT in WinAPI), but real UI needs quickly evolve them into the traditional RMGUI. BTW, this thinking is my source of doubts about React.
- tokyodude 8y ago
  I think you have it exactly backwards. Most ImGUI use cases are extremely complex UIs and that's where they excel. One of the most complex apps out there, Unity, is entirely ImGUI based for the app itself and often has 1000s of controls visible. Going though the screenshots of typical ImGUI usage most of them are on the complex side. https://cloud.githubusercontent.com/assets/8225057/20628927/33e14cac-b329-11e6-80f6-9524e93b048a.png https://cloud.githubusercontent.com/assets/8225057/20628927/... https://raw.githubusercontent.com/wiki/ocornut/imgui/web/v160/editor_white.png https://raw.githubusercontent.com/wiki/ocornut/imgui/web/v16... https://user-images.githubusercontent.com/16171743/50757906-f2bace80-1260-11e9-9ce5-4173b68cd9a2.jpg https://user-images.githubusercontent.com/16171743/50757906-... https://user-images.githubusercontent.com/8225057/51042368-65f37600-15bc-11e9-99cf-36e9f9822610.jpg https://user-images.githubusercontent.com/8225057/51042368-6... https://user-images.githubusercontent.com/870320/51073482-ba0c6200-1671-11e9-9b44-0c573ab0fae5.png https://user-images.githubusercontent.com/870320/51073482-ba... See https://github.com/ocornut/imgui#Gallery https://github.com/ocornut/imgui#Gallery for more
mschaef 8y ago
I'm going to date myself here, but this reminds me in some ways of Microsoft's BASIC 7.0 (circa 1989). One of the new features of 7.0 was a library for producing text mode GUIs with dialog boxes and the like. Because the language didn't support anything resembling a callback, everything had to be written in what was essentially "immediate mode". Lots of control flow that would normally be down in the library stayed up in user code. The net result aligned well with the description in this article: "...much shorter but more intensive, every thing is packed in a single function." This approach also presented lots of opportunities for errors caused by failing to correctly code something that would be default behavior in a normal library. Never seen anything else quite like it, aside from this. (Which, quite frankly, I'm happy about.)
alex-e 8y ago
Do you think it's possible to build an immediate UI wrapper on top of Win32 and Cocoa?
chriswarbo 8y ago
IMGUI always brings a couple of things to mind for me, which don't seem to be discussed often: Smalltalk/Self GUIs: AFAIK, the idea (or at least terminology) of MVC comes from the Smalltalk80 GUI. MVC seems to be intrinsically retained-mode: the "view objects" (buttons, forms, etc.) are separate to the "model objects". Self was inspired by Smalltalk, and its Morphic UI seems to be immediate-mode: the UI objects often are the model objects. This idea is more explicit in the "Naked Objects" approach. Squeak Smalltalk uses Morphic, although its descendents (e.g. Pharo and Newspeak) seem to have replaced it (I'm not sure of their paradigm though) Denotative, Continuous-Time Programming (DCTP): This was originally called "Functional Reactive Programming" ( http://conal.net/papers/frp.html http://conal.net/papers/frp.html ), but that term is now mainly applied to discrete, event-driven systems like Elm and React, which can make some articles/papers a bit confusing to read ;) In this approach, anything which varies over time (e.g. user input, display widgets, application state, etc.) becomes a function taking the current time as a parameter, called a "behaviour". These behaviours can be composed together in various ways to create an application's overall behaviour, still parameterised by time. The "main loop" just samples this behaviour at an appropriate rate, e.g. a fixed frame rate, or varying based on system load, how much activity is happening, etc. This is very much IMGUI, but has some nice advantages over the fixed ticking of a traditional main loop. For example, different components of a behaviour can be sampled at different rates, e.g. rapid sampling for smooth animations, coarse sampling for expensive calculations. Interpolation behaviours can remove jerkiness (e.g. a smooth animation based on an expensive simulation).
- scroot 8y ago
  > Squeak Smalltalk uses Morphic, although its descendents (e.g. Pharo and Newspeak) seem to have replaced it (I'm not sure of their paradigm though) Pharo is still using Morphic, though there is a new graphics layer called Bloc (vector and constraints based) that has been under development to replace it.
childintime 8y ago
How does Flutter sit in the spectrum from Immediate mode to Retained mode? I presume it might provide an interesting erspective. On the one hand widgets are constructed on the fly, on the other it uses an OOP language to do so. So I suspect this maps to statefulness and statelessness?
- raphlinus 8y ago
  Flutter is implemented in layers. The RenderObject layer looks very much like a traditional retained mode UI. It's possible to write Flutter entirely at this level, nobody forces you to use the Widget layer above it. The Widget layer is basically React-style, where the API looks fairly similar to immediate mode, but it uses caching and diff computation to reflect everything into the retained RenderObjects below it.
whateveryou381 8y ago
2015

185 points by sytelus 8y ago

Oh no, not again! (Just kidding :) ) Immediate mode UIs come up every now and then, especially in games, and for relatively simple things they're a neat idea (imagine not retaining any extra view state!) But - and there's a big "but" coming up - it does not scale to complex and stateful UIs. The problem is that, for games that have complex needs - think simulation and strategy games, not racing or arcade - complex UIs are needed to express complex relationships between underlying data and the relevant UX flows. But how do you do that in IMGUI? This information about UI state, layout, etc. is not in the UI anymore, but it has to live somewhere - so you end up having to build your own layout and UI management stuff, and basically duplicate what a RMGUI library would offer, but now with less generality and potentially more bugs. For the probably best known example, Unity used to have an IMGUI [1] but they've actively moved away from it. It's still there in vestigial form, the API is not removed, but game developers don't really use it. [1] https://docs.unity3d.com/Manual/GUIScriptingGuide.html https://docs.unity3d.com/Manual/GUIScriptingGuide.html PS. And there's also the lack of visual layout tools, which is directly related to IMGUI's reliance on code to produce layout.

mellinoe 8y ago
> For the probably best known example, Unity used to have an IMGUI [1] but they've actively moved away from it. It's still there in vestigial form, the API is not removed, but game developers don't really use it. It's true that the newer retained mode GUI has supplanted the older IMGUI for "game UI" (e.g. what users interact with when playing an actual game built with Unity). On the other hand, the Unity editor itself is still built entirely with their immediate-mode GUI library, and I haven't heard of any plan to move away from that. Building "editor-like" tools (including custom user tools) is still significantly easier to accomplish using immediate-mode libraries.
- jayd16 8y ago
  In Unity you're still forced to use the IMGUI for making editor tools. In fact their new xml layout prototype still uses the IMGUI to draw an instantiated layout! What a mess...
- 60654 8y ago
  Oh, they're switching to RMGUI in the editor as well, though more slowly. More info here: https://blogs.unity3d.com/2019/04/23/whats-new-with-uielements-in-2019-1/ https://blogs.unity3d.com/2019/04/23/whats-new-with-uielemen...
- vardump 8y ago
  Interesting (even a surprising) choice Unity is making. What are the performance implications of this change?
overgard 8y ago
I wouldn’t use one for a production UI, but they’re super handy for building light weight debugging tools.
DonHopkins 8y ago
Immediate mode GUI's don't let you apply useful object oriented programming techniques like subclassing and prefabs (persistence) to implement and configure custom components. Properly object oriented retained mode GUIs let you subclass components to implement custom views and event handling, and develop and publish reusable prefabs and libraries of pre-configured, stylized, specialized component.
- chii 8y ago
  > don't let you apply useful object oriented programming techniques [to get] reusable libraries of specialized component. But immediate mode GUI lets you compose very well. It's a bottoms up technique of programming, where instead of following a framework established by the library author, you are given low level building blocks. You will find it hard to mix and match two retained mode GUI libraries unless they all follow a common set of standard apis (e.g., java swing). But with immediate mode GUI, it's actually easy to mix and match (as the immedate mode gui has no need to know what framework you're working under nor need to make any assumptions about threading or api).
  - DonHopkins 8y ago
    I don't know what you mean about easily composing immediate mode API's, or agree that different immediate mode APIs don't need to know about each other. They depend on a lot of behind-the-scenes implicit hidden state (practically a hidden shadow-DOM-like retained mode), so I suspect in many cases they'd walk all over each other (especially text editors dealing with input focus and keyboard navigation). Unity3D has two immediate mode APIs, one for the editor and one for runtime, and they sure don't mix. I've implemented various adaptor wrappers for composing different retained mode API's myself, including wrapping The NeWS Toolkit OPEN LOOK widgets in HyperLook widgets, which I used for the first Unix port of SimCity to HyperLook/NeWS. (The OPEN LOOK pin-up menus and pie menus and buttons and sliders are implemented in TNT, but I wrapped them for HyperLook (which has its own persistence system and class hierarchy), so you can dynamically create and edit them at runtime with property sheets and script editors, ala HyperCard.) https://medium.com/@donhopkins/hyperlook-nee-hypernews-nee-goodnews-99f411e58ce4 https://medium.com/@donhopkins/hyperlook-nee-hypernews-nee-g... For X11/NeWS at Sun, we even made an ICCCM X11 window manager that wrapped X11 windows in NeWS tabbed window frames with pie menus! (The X11 window manager was written in object oriented "retained mode" PostScript code, subclassing standard and custom NeWS window and menu widgets, like tabbed window frames and pie menus. While PostScript has an immediate mode drawing API, it was driven by extensible "retained mode" widget instances and classes represented by PostScript dictionaries, pushed on the dict stack.) Open Window Manager PostScript source: https://www.donhopkins.com/home/archive/NeWS/owm.ps.txt https://www.donhopkins.com/home/archive/NeWS/owm.ps.txt Tabbed window extension: https://donhopkins.com/home/archive/NeWS/win/tab.ps.txt https://donhopkins.com/home/archive/NeWS/win/tab.ps.txt Discussion of problems with NeWS: https://news.ycombinator.com/item?id=15327339 https://news.ycombinator.com/item?id=15327339 Political discussion from the Window System Wars, about Sun's problems merging X11 and NeWS: https://donhopkins.com/home/archive/NeWS/sevans.txt https://donhopkins.com/home/archive/NeWS/sevans.txt More recently (and more successfully), I've wrapped HTML components and floated them over OpenLaszlo/Flash (not so recently) and Unity3D (quite recently) retained mode user interface layouts in the web browser. For example, I'm embedding an ACE code editor in a Unity3D retained mode user interface running in WebGL, that falls back to a regular TextMeshPro input field on other builds. There's no way you could do that practically with an immediate mode API. WebGLAce_TMP_InputField.cs (C# retained mode GUI TMP_InputField subclass): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/Libraries/WebGLAce/Scripts/WebGLAce_TMP_InputField.cs https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... WebGLAce.cs (C# P/Invoke Unity extension wrapper): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/Libraries/WebGLAce/Scripts/WebGLAce.cs https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... WebGLAce.jslib (JavaScript library Unity extension, compiled by Unity to WebAssembly, converts between C#/WebAssembly and JavaScript data types and thunks messages back and forth between the browser and the Unity3D app): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/Libraries/WebGLAce/Plugins/WebGL/WebGLAce.jslib https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... WebGLAce.jss (ACE code editor adaptor, pure JS source loaded by the browser, as well as ACE JS libraries themselves): https://github.com/SimHacker/UnityJS/blob/master/Libraries/WebGLAce/StreamingAssets/WebGLAce/WebGLAce.jss https://github.com/SimHacker/UnityJS/blob/master/Libraries/W... Notice how WebGLAce_TMP_InputField simply subclasses the standard TextMeshPro TMP_InputField, and adds some more hidden state like the editorID and editorConfigScript of the corresponding ACE code editor running in the web browser. All that is hidden from the code using it, so it doesn't need to come up with another place to store that editorID and editorConfigScript, or even know if it's getting an Ace editor under WebGL or falling back to TextMeshPro on other builds. That's what I mean by leveraging OOP to subclass existing components and hide internal state, which you can't do in immediate mode. (My plan for non-WebGL builds that aren't already running inside a web browser, is to create an embedded web browser just to run the Ace code editor, and the C# WebGLAce_TMP_InputField API will remain the same, and it will just work transparently on different platforms. I want to support live coding Unity apps in JavaScript, but there just isn't anything for Unity that approaches ACE for editing code, so it's worth going through all the trouble to make an adaptor.) How would you write a functional extension for Unity3D's old immediate mode GUI that let you embed an ACE code editor in a Unity WebGL app? What would the API and state management even look like? How could you make it platform independent? And even if you solved all of those problems with an immediate mode API, by its very nature it still wouldn't enable you to build GUIs in the interface editor or store them in prefabs, and you'd still have to recompile you app (which can take half an hour with Unity) every time you wanted to tweak the user interface (which is why I like programming Unity apps in JavaScript as much as possible). UnityJS Unity3D / JavaScript bridge architecture: https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy.txt https://github.com/SimHacker/UnityJS/blob/master/doc/Anatomy... UnityJS pie menu component for Unity3D, written in JavaScript, drawn with canvas, using TextMesh Pro labels: https://github.com/SimHacker/UnityJS/blob/master/Libraries/UnityJS/StreamingAssets/UnityJS/pie.jss https://github.com/SimHacker/UnityJS/blob/master/Libraries/U... Uses a Unity3D pie menu tracker to translate low level input to high level events, written in C#: https://github.com/SimHacker/UnityJS/blob/master/Libraries/UnityJS/Libraries/UnityJS/Scripts/BridgeObject/PieTracker.cs https://github.com/SimHacker/UnityJS/blob/master/Libraries/U... Of course you can take it too far, trying to please everyone by combining the worst of all possible systems and then giving it a ridiculous sounding name (see "MoOLIT"), but we're discussing whether it's possible and practical, not whether it's desirable. And even if it's not desirable, it ends up being necessary. https://en.wikipedia.org/wiki/MoOLIT https://en.wikipedia.org/wiki/MoOLIT http://nova.polymtl.ca/~coyote/open-look/01-general/faq-doc-1.html http://nova.polymtl.ca/~coyote/open-look/01-general/faq-doc-... Embedding object oriented "retained mode" widgets with different APIs inside of each other is old hat and common for backwards compatibility. Whenever you write a new GUI toolkit, embedding widgets from the last toolkit is one of the first things you do (including recursively embedding widgets from the toolkit-before-last). Concrete example: Microsoft lets you embed old fashioned OLE controls in Windows Forms / Presentation Foundation applications, which might be implemented in MFC themselves. And MFC is all about embedding old Win32 widgets implemented in C, and newer OLE components implemented in Visual Basic or whatever, in venerable C++ MFC user interfaces. Say what you want about how much Win32/MFC/OLE/WF/WPF sucks, and I'll wholeheartedly agree, but if you're running Windows, you probably have widgets on your screen using several different retained mode APIs embedded more than two levels deep right now. https://en.wikipedia.org/wiki/Windows_Forms#Architecture https://en.wikipedia.org/wiki/Windows_Forms#Architecture
jpochtar 8y ago
I mean, React is essentially a hack to get an immediate mode API on top of a retain mode platform (DOM), because the immediate mode API scales better to handle complex state. I'd push that yes, you do have to maintain your state externally in an immediate mode UI, but when you have a lot of "state" that's really just a function of other state, that's exactly what you want.
- btown 8y ago
  The interesting thing about React is actually that it encapsulates a lot of state retention inside an immediate-style API. state/setState (and now state hooks) are the unsung heroes of the ecosystem, because they let you focus on only the business-relevant global state, making it natural for a dropdown menu to handle its own open/closed state and insulate its user from that complexity. And many intermediate mode libraries gloss over this need, so you have to track much more state than you’d like. It’s a fascinating hybrid - rather than “BYO State” where the library does nothing but layout and draw, or “BYO hierarchy” where you need to retain instantiated components, React is “hierarchy tracking included” - all the benefits of persistent components, but tracked by the renderer rather than the caller. It’s an evolutionary step that I think desktop programming would do well to adopt and continue to iterate on.
  - deleted 8y ago
    [deleted]
- zamalek 8y ago
  React is half-and-half. It is immediate mode in usage, but the data structure (vdom) is what you'd expect from retained.
dan-robertson 8y ago
Well with retained mode one still has to handle a bunch of application state, and handle keeping that state synchronised with the view. Immediate mode typically forces you to manage the view state too but it’s typically easier to keep it synchronised with the application state that way. I think the following is an interesting example here: your application state is some type, Config which would correspond to some boring components, e.g. text boxes and radio buttons, but the interface you want to present looks like: [X] default settings Or [ ] default settings ... Now the type of data you want by the end of the form looks like (eg Haskell) Maybe Config Where Nothing means “do the default thing”. But the state of the view has to look more like: Bool * Config Because you want to remember what options the user had changed in case they change their mind a few times about using the default state. And now maybe you want a state like (mutable) Bool * Config * (mutable) Config Because the default state is not static and you want to make fields bold if they are not default. And so now you still have to do the complicated conversion from view to application state (but at least you don’t have to go back). I suppose this is like the very early web dev style of having no direct state and directly reading values from controls, but at least in this world one has control over how the view state is managed and represented so it can be easier to avoid getting into invalid states.
naasking 8y ago
> But how do you do that in IMGUI? This information about UI state, layout, etc. is not in the UI anymore, but it has to live somewhere - so you end up having to build your own layout and UI management stuff Immediate mode is about the interface to rendering the UI, not necessarily about the implementation (although some people conflate these). So your UI elements can be stateful and retain their own state, but they can be released once they are no longer rendered in the current frame.
cmrdporcupine 8y ago
Honestly -- given the amount of time I've spent adapting my own data model classes and structures to the straightjacket of various GUI or web toolkits -- not having the toolkit enforce its normative lifestyle assumptions on data structure feels like a feature to me. I'd rather manage the state and relationships myself.
- jayd16 8y ago
  >given the amount of time I've spent adapting my own data model classes and structures to the straightjacket of various GUI This is why every tells you to use MVC or some variant that separates out the UI from the data. IMGUI lets you bend the UI to the data instead but the real lesson is to decouple the view and the model.
  - cmrdporcupine 8y ago
    I'm very familiar with MVC -- I've been in the industry for 20+ years -- but so far disgusted with the way pretty much every GUI framework expects you to handle this. In reality I may already have a data model, but often have to wrap _another_ model around my model, just to satisfy the GUI framework's constraints on what it thinks a model needs to inherit from, etc. Recently ran into this with Qt. Very frustrating.
    - jayd16 8y ago
      The trick is to just accept that the UI model, the business logic model and the denormalized db model all have different goals and often times should be decoupled and adapted.
      - cmrdporcupine 8y ago
        Which is why it seems almost every project with a complicated UI turns into a bloated pile of semi-duplicated boiler plate 'model' classes with little business logic. Clearly, MVC as the industry understands it, has a problem. Which is why the original Smalltalk people -- who invented MVC -- mostly switched to Self's Morphic framework in their later projects (Squeak, etc.)
DonHopkins 8y ago
Here's a demo of Unity3D's new retained mode UIElements user interface for the editor, that lets you wrap old immediate mode IMGUI code in IMGUIContainer wrappers! https://youtu.be/MNNURw0LeoQ?t=17m37s https://youtu.be/MNNURw0LeoQ?t=17m37s

deleted 8y ago

[deleted]

sytelus 8y ago

TLDR; Typically we build GUIs using event driven callbacks. This causes problems in managing life times, dynamic layouts, threading issues etc. The idea behind immediate mode GUI is that you get a frame and in each frame you create/update layout, check for user actions - just like you do in render frame in game loop. So there are no call backs and you don't have to worry about issues arising from event-driven code.

jboles 8y ago
Does this pretty-much rule out (without a huge amount of effort) nonmodal GUIs? It seems without callbacks, you would need to roll your own window management, Z-ordering of overlapping UI elements, state transition validation (e.g. clicking on the parent dialogue’s Cancel button while a child window is still open), ...
- jackfraser 8y ago
  There's no reason that that can't all be done as part of the underlying IMGUI implementation; what you're working with above that is essentially a DSL that renders out to a workable set of controls. Layout management and the actual rendered graphics / animations are handled by the code you don't have to see. If anything, this is actually a lot like Web 1.0 programming: server side rendering, often using templates that end up looking a lot like a DSL. Each page view is rendered based on those templates against data fetched in advance of the output. You can get pretty deep with that kind of thing, though of course the page refresh time was what killed this otherwise fairly nice style of writing an application. Users expected actions like sorting a column to be as fast in the browser as it was in a Windows 95-era file explorer or Excel, and it simply wasn't happening. (Imagine if first-class client-side-interactive UI components like a sortable table, tabbed panel, etc. like what XUL provides had been in HTML in the 90s. We wouldn't need half the Javascript bullshit people have to use now to replicate that functionality half-assedly...)
- mmozeiko 8y ago
  Here's an browser demo for dear imgui (https://github.com/ocornut/imgui https://github.com/ocornut/imgui) library: https://pbrfrat.com/post/imgui_in_browser.html https://pbrfrat.com/post/imgui_in_browser.html For modal demo check "Popups & modal windows" -> "Modals" example. This is how it looks in C++ code: https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp#L2273 https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp#...

userbinator 8y ago

Some also argue that executing UI logic of every widget is wasteful. This might indeed be a concern when implemented poorly For a game, which is already quite busy even when "idle" (and there are probably other things going on in each frame which take far more CPU and GPU to render), it won't be much of a concern; but with an application where most of the time is spent waiting for user input, spinning the CPU to re-render the GUI every frame is very energy-inefficient.

gameswithgo 8y ago
you can work around that problem though.
frnkng 8y ago
I don’t think that an imgui app will eat more resources than a comparable electron app. Thus the Performance Argument against any IMGUI is pretty pointless.
- theamk 8y ago
  I disagree. With all downsides of electron, it does not generally re-layout and re-render entire screen on any mouse move. IMGUI does. So while Electron will lose in memory/disk used, IMGUI will lose in CPU usage (battery life) once the software is complex enough.
  - tokyodude 8y ago
    I disagree with your disagreement :) Electron (and nearly all retained mode GUIs) will execute 10x more code trying not to execute the rendering code. Just skipping all that checking and jumping to the rendering code is arguably a win in many many cases. To put some empirical data on it, game devs use ImGUIs because they keep up with perf where as that 10x overhead of retained mode guis creating gui objects, deleting gui objects, marshaling data into and out of gui objects, adding and removing event handlers and callbacks, and then trying to write a bunch more code to optimize all of that never leads to a GUI that is performant enough for the GUIs needed for game dev. I think if you check actual apps, especially phone apps, you'll find it's not so clear which one wins. Phone apps almost always move in relation to the users fingers meaning the screen is being re-rendered. Usually new GUI elements are moving on and off the top/bottom of the screen. The overhead of creating/deleting/managing those objects is arguably greater than just re-rendering. The screen is already going to be re-rendered. At best you're getting a few cached textures from your retained mode gui all at the expensive of managing GUI objects. For desktop apps it really depends on the app. It's got nothing to do with the complexity of the UI and everything to do with what the app does. A text editor might be a win for a retained mode GUI. Photoshop/Maya it's less clear. For example Unity is ImGUI and has very complex UI. Much of the UI has to respond in realtime to the state of the user's document that is changing constantly.

jpnelson 8y ago

Are there any popular implementations of immediate mode GUI for the web? It might be interesting to have one that could then "eject" into a stateful UI for when things start to get more complicated.

rtpg 8y ago
I think that React is like 90% of the way there and you would just need to mess around with the VDOM a bit for this. To be honest, given that web render output is a tree instead of flat "render to screen" stuff, writing a nice immediate mode GUI is not obvious. If you're... very clever you could take React and the State stuff, at least conceptually, and get what you want here.
- kkarakk 8y ago
  wouldn't you need some sort of gpu access to have a smooth UI rendering with immediate gui(in production)?
  - rtpg 8y ago
    This is if you expect to build an immediate mode with the same flat rendering model. imgui's model is "you draw onto pixels". You could create another mental model which is "you draw onto a tree." so instead of saying "drawText(x, y, text)" you would do something like "drawText(domLocation, text)" Granted, this model gets a bit messier, and _some_ might say defeats the purpose of immediate mode. But I think you could still get conciseness benefits on this.
haskman 8y ago
I wrote a UI framework called Concur which is very similar to immediate mode UIs and hence very easy to use, but also scales fantastically well to complex UIs. The original code was written in Haskell (https://github.com/ajnsit/concur https://github.com/ajnsit/concur), and then ported to Purescript (https://github.com/ajnsit/purescript-concur https://github.com/ajnsit/purescript-concur), and even Javascript (https://github.com/ajnsit/concur-js https://github.com/ajnsit/concur-js). There's an introduction/tutorial here - https://github.com/ajnsit/concur-documentation/blob/master/README.md https://github.com/ajnsit/concur-documentation/blob/master/R...

Stratoscope 8y ago

Note to authors: please define your terms when you first use them, or provide links to the definitions. Of course I know what a GUI is. I've been building them for decades, and building tools that other developers have used to build GUIs. (I was one of the authors of the "visual" part of Visual Basic.) But I have never before seen the terms "immediate mode GUI" or "retained mode GUI". And I'm pretty sure there are many developers like me who will have no idea what you're talking about. It takes only a moment to add a quick definition or a link the first time you mention a specialized term, and your readers will definitely appreciate it. Thanks! (Edited to remove fluff and make the essential point more clear.)

kyberias 8y ago
I first learned about those terms in early DirectX releases (circa 1998). Are there some earlier usages?
- cmroanirgo 8y ago
  It's where it first came to prominence. The earliest D3D api was immediate mode (meaning, bare metal, talking directly via an api to the hardware). Retained mode is about retaining statefulness and some better abstractions to the underlying needs of the hardware (eg automatic texture resampling, etc). Considering it's an NVidia guy talking: he's been around for a while.
  - pavlov 8y ago
    A minor detail, but the very first version of Direct3D was a retained mode API. In 1995 Microsoft acquired a company called RenderMorphics and turned their Reality Lab [1] API into "Direct3D". The first two versions to ship from Microsoft had both retained and immediate mode APIs, but they dropped the RenderMorphics-based API due to lack of adoption. [1] https://en.wikipedia.org/wiki/Reality_Lab https://en.wikipedia.org/wiki/Reality_Lab
- Stratoscope 8y ago
  Interesting, thanks! That would explain why I and probably many other readers, who have built plenty of GUIs but have not worked specifically with APIs like DirectX, would not be familiar with the term.

dmix 8y ago

Please add [2015] to title

soylentgraham 8y ago
Also, 2011, 2007, 2001, 1996 :) Immediate vs deferred, is just another paradigm that goes back and forth in trends like keyboard input/buffering, deffered/one pass lighting, monoliths vs modules, State vs events, flat vs 3D

BoorishBears 8y ago

I find it ironic that Unity is used as an example... the immediate mode UI was/is considered a joke. Terrible performance, doesn’t scale to UIs more complex than a form (... like the ones many games tend to have) It got to the point where there was a website dedicated to track when we’d get the current non-immediate UI, and Unity actually hired the developer of the leading non-immediate mode UI library in the asset store to work on their own offering.

tokyodude 8y ago
I think you're mis-understanding why Unity switched. They only switched their in game GUI system. The entire app of Unity itself uses an ImGUI system and it is massively more complex than any Game UI and runs just fine. The reasons they switched the in game system are many but they have nothing to do with performance.
- BoorishBears 8y ago
  The editor has none of the constraints the games made with it have... like running on mobile and maintaining 60 FPS while doing it. ... but IMGUI is on the way out there too: https://docs.unity3d.com/Manual/UIElements.html https://docs.unity3d.com/Manual/UIElements.html The old system would stack up draw calls like nobody's bussiness, doubling drawcalls if you used the built in layout tools. OnGUI being called period would be expensive, even when you had properly implemented your rendering logic, so you’d have to disable the entire game object when you didn’t want to draw. The docs for mobile literally used to say not to use the OnGUI stuff during gameplay.

nickflood 8y ago

I've taken part in some WPF application development and I think there are some benefits in immediate-mode rendering of the UI. Particularly when the UI changes faster than a monitor refresh rate or there's too much data complexity and you want the data update and redraw processes to be independent. For my first example, updating controls and overall layout in a WPF application feels much more lightweight than in WinForms because the renderer just shows the result at the next monitor refresh instead of trying to update the layout in realtime or using hacks that disable redraw until you do everything. For the second example, I've used SciChart to display trading data and it again was much more responsive because it replaced a control-based rendering pipeline with just an immediate-mode canvas tgat would render the selected data window once per monitor refresh rate. And I think overall once you hit the performance barrier where you can't just fill a panel with controls and let them render and instead have to make a custom draw routine, the immediate-mode rendering adds this extra smoothness to the UI and gives you a bit more performance headroom.

i_don_t_know 8y ago

While I find the idea appealing, I haven't used them because I haven't figured out how to make immediate GUIs accessible to people who are visually impaired. How do you make them work with screen readers? You're drawing text onto a canvas but you don't retain a copy of the text for the screen reader.

loup-vaillant 8y ago
I don't know about screen reader APIs, but… can't you just send the text you're drawing to the screen reader when you drawing it? How do screen readers work anyway?
- i_don_t_know 8y ago
  Good question. I don't know. When you use the native GUI you get all of that for free. You don't have to do anything special for accessibility. So I always thought that screen readers somehow traversed the window/control hierarchy to find the control that has the focus and ask it for any text that should be read. Maybe you're right and you can send the text to be read directly to the screen reader. Not sure how well that works when your GUI updates every 20ms. You don't want to keep resetting the reader.
  - loup-vaillant 8y ago
    I was thinking maybe resend the stuff only when the text changes, so you'd have to have a cache of some sort, hidden in the library. Anyway, I believe Qt is screen reader compatible, so it must access the relevant API somehow. I'd be surprised if it was the exclusivity of the OS vendor.

bdowling 8y ago

Immediate mode GUIs are an extension of the simple display update loop: (1) update display, (2) check input, (3) goto (1). Immediate mode GUIs add a context parameter to the update function. In the draw context, the update function draws to the screen. In the input context, the update function doesn't draw; instead it checks if mouse clicks or keystrokes should do something. For example, part of the update function might look like this: if (Button("Do That Thing")) doThatThing(); In the draw context, the call to Button will draw a button on the screen and return False. In the input context, the call to Button will check if a mouse click fell within the area it would have drawn. If so, it will return True and the callback will run. An immediate mode GUI is basically the opposite of a typical Model-View-Controller GUI because it deliberately mixes the Model, View, and Controller together into one function. Just one function draws the GUI based on whatever state it's in and whatever the model is. The same function gets reused in a different context to update the GUI state or handle callbacks.

tokyodude 8y ago
I'd argue an ImGUI does't mix anything. It's just a library. If you want MVC it's up to you to implement your MVC and use your ImGUI as a library. Retained mode GUIs don't magically make your code MVC and ImGUIs don't magically make your code not MVC. That's entirely up to you. Also ImGUIs are not multiple pass. That's in implementation detail. Unity's ImGUI is multi-pass. One of the most popular ImGUIs (Dear ImGUI) is single pass https://github.com/ocornut/imgui https://github.com/ocornut/imgui

avodonosov 8y ago

Again IMGUI. Several years ago there was a chain of videos and blog posts by different authors. People were drawing two buttons and saying "see how easy it is". In this post the author seems to go further than two buttons, but I'm skeptical. I wish there was a video showing how it works in dynamics - inputs clicked, texts entered, scrollbars scrolled, dialogs dragged around. Or source code available to build and try locally. From the code in the post I suppose the majority of UI is dead - can not be interacted with. I think IMGUI can only be convenient for very simple cases. In real frameworks the lowest level is immediate mode (see WM_PAINT in WinAPI), but real UI needs quickly evolve them into the traditional RMGUI. BTW, this thinking is my source of doubts about React.

tokyodude 8y ago
I think you have it exactly backwards. Most ImGUI use cases are extremely complex UIs and that's where they excel. One of the most complex apps out there, Unity, is entirely ImGUI based for the app itself and often has 1000s of controls visible. Going though the screenshots of typical ImGUI usage most of them are on the complex side. https://cloud.githubusercontent.com/assets/8225057/20628927/33e14cac-b329-11e6-80f6-9524e93b048a.png https://cloud.githubusercontent.com/assets/8225057/20628927/... https://raw.githubusercontent.com/wiki/ocornut/imgui/web/v160/editor_white.png https://raw.githubusercontent.com/wiki/ocornut/imgui/web/v16... https://user-images.githubusercontent.com/16171743/50757906-f2bace80-1260-11e9-9ce5-4173b68cd9a2.jpg https://user-images.githubusercontent.com/16171743/50757906-... https://user-images.githubusercontent.com/8225057/51042368-65f37600-15bc-11e9-99cf-36e9f9822610.jpg https://user-images.githubusercontent.com/8225057/51042368-6... https://user-images.githubusercontent.com/870320/51073482-ba0c6200-1671-11e9-9b44-0c573ab0fae5.png https://user-images.githubusercontent.com/870320/51073482-ba... See https://github.com/ocornut/imgui#Gallery https://github.com/ocornut/imgui#Gallery for more

mschaef 8y ago

I'm going to date myself here, but this reminds me in some ways of Microsoft's BASIC 7.0 (circa 1989). One of the new features of 7.0 was a library for producing text mode GUIs with dialog boxes and the like. Because the language didn't support anything resembling a callback, everything had to be written in what was essentially "immediate mode". Lots of control flow that would normally be down in the library stayed up in user code. The net result aligned well with the description in this article: "...much shorter but more intensive, every thing is packed in a single function." This approach also presented lots of opportunities for errors caused by failing to correctly code something that would be default behavior in a normal library. Never seen anything else quite like it, aside from this. (Which, quite frankly, I'm happy about.)

alex-e 8y ago

Do you think it's possible to build an immediate UI wrapper on top of Win32 and Cocoa?

chriswarbo 8y ago

IMGUI always brings a couple of things to mind for me, which don't seem to be discussed often: Smalltalk/Self GUIs: AFAIK, the idea (or at least terminology) of MVC comes from the Smalltalk80 GUI. MVC seems to be intrinsically retained-mode: the "view objects" (buttons, forms, etc.) are separate to the "model objects". Self was inspired by Smalltalk, and its Morphic UI seems to be immediate-mode: the UI objects often are the model objects. This idea is more explicit in the "Naked Objects" approach. Squeak Smalltalk uses Morphic, although its descendents (e.g. Pharo and Newspeak) seem to have replaced it (I'm not sure of their paradigm though) Denotative, Continuous-Time Programming (DCTP): This was originally called "Functional Reactive Programming" ( http://conal.net/papers/frp.html http://conal.net/papers/frp.html ), but that term is now mainly applied to discrete, event-driven systems like Elm and React, which can make some articles/papers a bit confusing to read ;) In this approach, anything which varies over time (e.g. user input, display widgets, application state, etc.) becomes a function taking the current time as a parameter, called a "behaviour". These behaviours can be composed together in various ways to create an application's overall behaviour, still parameterised by time. The "main loop" just samples this behaviour at an appropriate rate, e.g. a fixed frame rate, or varying based on system load, how much activity is happening, etc. This is very much IMGUI, but has some nice advantages over the fixed ticking of a traditional main loop. For example, different components of a behaviour can be sampled at different rates, e.g. rapid sampling for smooth animations, coarse sampling for expensive calculations. Interpolation behaviours can remove jerkiness (e.g. a smooth animation based on an expensive simulation).

scroot 8y ago
> Squeak Smalltalk uses Morphic, although its descendents (e.g. Pharo and Newspeak) seem to have replaced it (I'm not sure of their paradigm though) Pharo is still using Morphic, though there is a new graphics layer called Bloc (vector and constraints based) that has been under development to replace it.

childintime 8y ago

How does Flutter sit in the spectrum from Immediate mode to Retained mode? I presume it might provide an interesting erspective. On the one hand widgets are constructed on the fly, on the other it uses an OOP language to do so. So I suspect this maps to statefulness and statelessness?

raphlinus 8y ago
Flutter is implemented in layers. The RenderObject layer looks very much like a traditional retained mode UI. It's possible to write Flutter entirely at this level, nobody forces you to use the Widget layer above it. The Widget layer is basically React-style, where the API looks fairly similar to immediate mode, but it uses caching and diff computation to reflect everything into the retained RenderObjects below it.

whateveryou381 8y ago

2015