Much of what you're describing reminds me of Dynamic Windows, the UI on Symbolics workstations in the 80s, and CLIM (Common Lisp Interface Manager), an attempt

Much of what you're describing reminds me of Dynamic Windows, the UI on Symbolics workstations in the 80s, and CLIM (Common Lisp Interface Manager), an attempt to embody the same ideas in a vendor-neutral standard. DW/CLIM record text+graphical output in a DOM-like tree structure with references to the underlying application objects plus a "presentation type" and parameters for how the object was presented. The interaction window presents a stream interface rebinding your application's standard-input and standard-output streams, but also implements drawing operations, and the functions 'accept' and 'present' do structured IO against the stream in terms of actual objects. The presentation-type parameter determines both how to parse/print objects and which objects on screen may be selected with the mouse as input. Everything being live objects, the lisp machine usually let you select any object on screen and pop it into an inspector. Commands defined the presentation types of their parameters, again defining how to parse/print arguments and enabling mouse selection. Presentation translators let you define how to convert an object of some type into a context where the UI wanted input of a different type. Commands themselves had a presentation type, so the top-level command processor literally loops printing a prompt, calling 'accept' for a command, then executing the command. Consistently, presentation (to command) translators let you trigger commands directly via clicking objects, drag-and-drop gestures, etc. permitting the UI to function in terms of commands even if the textual 'interactor-pane' was absent. It's a fascinating paradigm, with some shortcomings. Despite CLIM defining some more traditional GUI widgets, how to integrate then with presentations and the command processor is a bit half-baked. This is a solvable problem. A free implementation called McCLIM is still around and worked on by one or two of the devout. One of these days I hope to see a good implementation of these ideas in Javascript. A browser DOM is the perfect substrate on which to implement a modernized CLIM-like UI, rather than reinventing everything the hard way on top of X11 like McCLIM attempts. It would certainly do wonders toward spreading the ideas.