4 ms·

I am fascinated by this idea despite the fact that I don't actually understand it. Ever since I read your comment, I've been trying to think what it would mean

by aag 8y ago

I am fascinated by this idea despite the fact that I don't actually understand it. Ever since I read your comment, I've been trying to think what it would mean to build a GUI this way. It has been thought-provoking, and has caused me to read about interesting things like ligature substitution rules in OpenType, but I still haven't made much progress. I would be grateful if you would elaborate in any way at all. Thanks!

mmjaa 8y ago
Here's a simple example of what I mean: consider the title-bar of the window you're reading this page in (assuming your wm has titlebars, but lets assume it does and you know what I'm referring to..) From left to right, there is a narrative - quite a bit like words in a sentence. The resources to construct that sentence - the bitmaps for the window control buttons, the spacing and formatting of the title, the right-hand-side elements of the window title - all of these resources consume some kind of storage on the computer, and require a set of abstractions in order to process through the pipeline that eventually renders something visible to the screen .. there are multiple filters in that process - the file-opening, then parsing of the file resource (.png) for an icon or two, then opening some type file, finding the elements of the on-screen elements which construct the glyphs in the title text, etc. This all, eventually, gets stuck up on the screen using some final method - perhaps the OS is using GL, and everything is just triangles, all the way down. Perhaps there's texture blitting and line-drawing primitives too .. All of these elements can be expressed as glyphs in a typographic way, and all of the changes and states that can occur with these elements, also - as ligatures, individual glyphs, etc. My idea is that these resources can be replaced with glyphs and ligatures from a font, and that indeed GUI elements themselves can be expressed (I would argue 'better') as graphemes in a common language context. Think of it like this: instead of having a filesystem full of resources that have to be individually processed and extracted, there is simply "funkyOSfromtheFuture.ttf", and everything that is required to be displayed by the OS, indeed anything, can be expressed as an array of mappings to glyphs within that single .ttf. Its not just that I hope to achieve a 'resource compaction' down to an ultimate font file and thus do away with a filesystem full of differing technologies to produce, ultimately, graphemes - though that is a nice effect. Its more also that I think that GUI's can be expressed in the same way that letters and words can be used to construct entire useful sentences - and I wonder at what sort of programming advances can be made with an OS (or more specifically, GUI environment) that is designed around the central principle that the elements of the interface are expressible using typographic metrics commonly used to render human-readable text. To me the "close window" icon is equivalent to the "." character, in terms of its effect on my eyeballs. So, why should it be splashed up on the screen using technology vastly different to that used to render other symbols, instead of just being included in a 'new GUI alphabet'. There have been efforts made to put things into this kind of technical context in the past - take for example NextSteps' approach where "everything is just display postscript" as a metaphor. "What if we built an OS whose entire look and feel was derived from a single font, and those elements were accessible to the keyboard user?", is the thought experiment.. like, I could have an alternative element in the font, and convenient key combination, that would turn any "?" in this sentence into a clickable link .. and you, upon clicking it, are presented with a means of answering the question... Instead of having abstract concepts of communication that are entirely different (forms, fields, WIMP elements, etc.) than the base set of tools we're already used to using as humans (letters/words/sentences/symbols), we merge the set. My window title becomes "< hi there .", and if you click any of those symbols, they do things.. of course, this is a very simplistic description but can think of others. Lets try another simple one: "[ This sentence would be a draggable window because the '[]' chars are considered by the OS to be clickable/draggable and represent a 'collection' of information that the user might want to move around the screen, so to do so they just click either one of the '[]' symbols that represent the edges of the collection.]" Another one: "Do you want to answer this question? Then simply click the '?' symbol at the side of the sentence, and you will be prompted to input your data - if not, simply click the period at the end of this sentence to dismiss the element."
- aag 8y ago
  Thank you very much. That is a lot of food for thought.
  - aag 8y ago
    Given this idea, I can't figure out how to deal with the fact that fonts are finite and, even with ligatures, relatively quickly enumerable. User interfaces, on the other hand, often require drawing arbitrary pictures. As I understand it, Chartwell takes advantage of the fact that charts have finite resolution, so that it's possible to enumerate all the possible heights for a bar or angles for a pie wedge as individual glyphs. But would that be possible for a general user interface? I may be missing something obvious.
    - CMCDragonkai 8y ago
      Compositionality should help with that. But at the same time, constraints should be domain based. Within the domain model, you can have a language express what you need. Your comment brings 3 thoughts to me: 1. Compositionality should enable elegant combinations of GUIs. Our programming language primitives are finite, yet there are infinite ways to combine them. Just as there is 26 letters in the alphabet, and yet we we have a growing language. 2. Domain modelling matters. Constraints on the system should be on the domain we are working within. Hence the primitives would have to be designed according to the domain. 3. Something about "constraints" and "creativity".
- CMCDragonkai 8y ago
  I've been thinking about this within the context of data visualisation schemas like Vega (http://vega.github.io/ http://vega.github.io/). They sort of a represent a middle ground between a full programming API like matplotlib to specify a graph or the XML SVG language, and the language of matrices that bitmap images really are. Easy for a machine to parse and easy to change with a more consistent structure for a given graph, with some limitations on what you can express.

by aag 8y ago

mmjaa 8y ago

Here's a simple example of what I mean: consider the title-bar of the window you're reading this page in (assuming your wm has titlebars, but lets assume it does and you know what I'm referring to..) From left to right, there is a narrative - quite a bit like words in a sentence. The resources to construct that sentence - the bitmaps for the window control buttons, the spacing and formatting of the title, the right-hand-side elements of the window title - all of these resources consume some kind of storage on the computer, and require a set of abstractions in order to process through the pipeline that eventually renders something visible to the screen .. there are multiple filters in that process - the file-opening, then parsing of the file resource (.png) for an icon or two, then opening some type file, finding the elements of the on-screen elements which construct the glyphs in the title text, etc. This all, eventually, gets stuck up on the screen using some final method - perhaps the OS is using GL, and everything is just triangles, all the way down. Perhaps there's texture blitting and line-drawing primitives too .. All of these elements can be expressed as glyphs in a typographic way, and all of the changes and states that can occur with these elements, also - as ligatures, individual glyphs, etc. My idea is that these resources can be replaced with glyphs and ligatures from a font, and that indeed GUI elements themselves can be expressed (I would argue 'better') as graphemes in a common language context. Think of it like this: instead of having a filesystem full of resources that have to be individually processed and extracted, there is simply "funkyOSfromtheFuture.ttf", and everything that is required to be displayed by the OS, indeed anything, can be expressed as an array of mappings to glyphs within that single .ttf. Its not just that I hope to achieve a 'resource compaction' down to an ultimate font file and thus do away with a filesystem full of differing technologies to produce, ultimately, graphemes - though that is a nice effect. Its more also that I think that GUI's can be expressed in the same way that letters and words can be used to construct entire useful sentences - and I wonder at what sort of programming advances can be made with an OS (or more specifically, GUI environment) that is designed around the central principle that the elements of the interface are expressible using typographic metrics commonly used to render human-readable text. To me the "close window" icon is equivalent to the "." character, in terms of its effect on my eyeballs. So, why should it be splashed up on the screen using technology vastly different to that used to render other symbols, instead of just being included in a 'new GUI alphabet'. There have been efforts made to put things into this kind of technical context in the past - take for example NextSteps' approach where "everything is just display postscript" as a metaphor. "What if we built an OS whose entire look and feel was derived from a single font, and those elements were accessible to the keyboard user?", is the thought experiment.. like, I could have an alternative element in the font, and convenient key combination, that would turn any "?" in this sentence into a clickable link .. and you, upon clicking it, are presented with a means of answering the question... Instead of having abstract concepts of communication that are entirely different (forms, fields, WIMP elements, etc.) than the base set of tools we're already used to using as humans (letters/words/sentences/symbols), we merge the set. My window title becomes "< hi there .", and if you click any of those symbols, they do things.. of course, this is a very simplistic description but can think of others. Lets try another simple one: "[ This sentence would be a draggable window because the '[]' chars are considered by the OS to be clickable/draggable and represent a 'collection' of information that the user might want to move around the screen, so to do so they just click either one of the '[]' symbols that represent the edges of the collection.]" Another one: "Do you want to answer this question? Then simply click the '?' symbol at the side of the sentence, and you will be prompted to input your data - if not, simply click the period at the end of this sentence to dismiss the element."

aag 8y ago
Thank you very much. That is a lot of food for thought.
- aag 8y ago
  Given this idea, I can't figure out how to deal with the fact that fonts are finite and, even with ligatures, relatively quickly enumerable. User interfaces, on the other hand, often require drawing arbitrary pictures. As I understand it, Chartwell takes advantage of the fact that charts have finite resolution, so that it's possible to enumerate all the possible heights for a bar or angles for a pie wedge as individual glyphs. But would that be possible for a general user interface? I may be missing something obvious.
  - CMCDragonkai 8y ago
    Compositionality should help with that. But at the same time, constraints should be domain based. Within the domain model, you can have a language express what you need. Your comment brings 3 thoughts to me: 1. Compositionality should enable elegant combinations of GUIs. Our programming language primitives are finite, yet there are infinite ways to combine them. Just as there is 26 letters in the alphabet, and yet we we have a growing language. 2. Domain modelling matters. Constraints on the system should be on the domain we are working within. Hence the primitives would have to be designed according to the domain. 3. Something about "constraints" and "creativity".
CMCDragonkai 8y ago
I've been thinking about this within the context of data visualisation schemas like Vega (http://vega.github.io/ http://vega.github.io/). They sort of a represent a middle ground between a full programming API like matplotlib to specify a graph or the XML SVG language, and the language of matrices that bitmap images really are. Easy for a machine to parse and easy to change with a more consistent structure for a given graph, with some limitations on what you can express.