4 ms·

I’m looking at the task of making programming more visual and structured but am at a very early stage (still getting my head around OpenGL/WebGL). I do realize

I’m looking at the task of making programming more visual and structured but am at a very early stage (still getting my head around OpenGL/WebGL). I do realize text us still used for its own reasons. I’d say they are: 1) Lack of need for a proprietary tool to view/edit. 2) Density of information. 3) Nearly every existing attempt I’ve seen at visual programming is insultingly bad. So I get it why folks would be hesitant to waste time on some schmuck’s new invention. Am I missing anything on the advantages of text? This is an honest attempt to see where I’m missing perspective, not an attempt to convince. I don’t think the target audience is developers with an existing working workflow- the potential upside prob wouldn’t justify someone spending the time/mental effort to grok a new model. But I think the audience is those who have some aptitude and could be coding and aren’t yet. In my mind, primarily so that maybe individuals can have a little more influence over the code and algorithms in their own life more so than industry roles. Given that the ways you can convey information graphically are a super set of what text provides, I can’t help but think there’s some room for improvement- especially with concepts like literate programming around. Overall, I’d like to give your average personal a little more agency. Maybe that’s a graphical approach. Maybe its getting folks to do basic Rails/Django/Flask/PHP whatever. But I am curious if I’m missing obvious rationales for text…

duped 4y ago
> Am I missing anything on the advantages of text? This is an honest attempt to see where I’m missing perspective, not an attempt to convince. Well-defined semantics for search, diff, insert, and replace.
jerf 4y ago
"Am I missing anything on the advantages of text? This is an honest attempt to see where I’m missing perspective, not an attempt to convince." Acknowledged. My answer is, programming is n-dimensional and text is one of the few mechanisms we have for dealing with that. I believe this is actually the characteristic of it that has screwed so many other efforts, especially visual ones. I don't know if n-dimensional has a formal, accepted definition, but the characteristic of an n-dimensional space that is of interest to me is that at any point, suddenly a new dimension can be introduced, and this is indefinitely recursive. Let me try to demonstrate this with a web site first. Imagine there's a web site that has a home page with ten internal links. Each of these ten links then has a set of ten pages that they link to, all unique, and those pages link nowhere. You could represent this particular website in a clean 2D spreadsheet. In the A column, you have the ten top-level links, and then in B through K you can have the subsidiary pages. In this simplified model, a web page can be imagined as something like a linear collection of links, so I can describe a page as being "Link 4, Link 7", identifying the subpage from the 4th top-level link, then the 7th link of that page. Of course, this is a highly contrived example. Perhaps only and exactly one of those links goes to another top-level page with another set of 10 links that each have ten pages. So now, in the middle of this nice two-dimensional layout, in exactly one cell, suddenly you have another two dimensional layout appearing, and neither of those two dimensions is the same as the previous two dimensions, it's a fresh set. Nothing stops these pages from having another fresh set of dimensions appearing. In n-dimensional space, you never know when you're going to get a fresh set of dimensions when you cross over some unassuming-looking edge in the graph. In reality, web pages are so complexly interlinked that there isn't any visual way of representing them in the general case. My example is obviously super unrealistic. We normally wouldn't even dream of trying to represent a website in a spreadsheet like that. That doesn't mean the situation is hopeless necessarily, but it does mean we will make compromises. A sufficiently static site might have a canonical "site map" that completely organizes the site into a tree. But that is a simplified projection of a human conceptualization of the space; it isn't the real linking map of the web site. It is valuable, but it's only a particularly humanly-compelling slice. In general, trying to collapse an n-dimensional space into a finite number of dimensions is not something you can do. Especially when the number you're trying to collapse it into is as small as "two" or "three"! This, again, doesn't mean the situation is hopeless. There's a lot of angles you can take on that "in general". "In general" is, as the saying goes, doing some heavy lifting in that statement. But it is an important aspect of understanding the problem. How does text handle the n-dimensionality of code? Consider it from the point ofview of the language grammar. If I have x = 1 + 1 each of those "1"s is actually an "expression" in the grammar (or some local equivalent), which means I can do: x = 1 + (readint() + readint()) and now I've expanded that expression into a new tree. For the purposes of a visual editor, not only have I expanded that 1 into two function calls, I've also got to have a link to the functions in question. And then this nests infinitely recursively. And programmers keep coming up not only with new dimensions through the basic n-dimensionality of text conforming to tree-based grammar structures, but then new ways of adding dimensions; "x.y" as a field reference of a particular type is one thing, but then x can be polymorphic. Or decorated. Or monkey-patched. Or perhaps the language has multiple dispatch. Or overloading operators, meaning that operators now need to be be something you can follow to an implementation. Or... the list goes on. You end up with a statement like "x.y.z = a.b.c(q, q2)" having potentially dozens of graph edges departing from it, each to a declaration that may itself be a rats nest of graph edges, each to a further declaration that may be a rats nest of edges... eventually, everything has to terminate in some sort of literal or atomic value of some sort, but we want those complex things. It's why we're not programming in raw assembler. (Though that would raise its own problems.) Generally, we don't even think of these spaces as being "dimensional", which is to say, able to identify members by a specification of numbers in some number of "dimensions". But it is a thing we do. You can see this in programming in a very simple form pretty much anywhere you see dot-separated values of arbitrary length. A number of standards have some sort of 2.8.183.2.18.1377273 embedded in them. DNS names function this way too; behind every DNS dot could be another arbitrarily large and complicated subdomain scheme. You could proceed down a dozen simple elements and then only suddenly see an explosion of complexity. If you imagined a perfectly static web, you could theoretically (non-uniquely) identify web page locations by starting with a top-level catalog like Yahoo, and identifying web pages via the path you need to take to get there, like the number above. In reality, the space itself is shifting too fast for this to work, isn't constant, etc. But that's what n-dimensional looks like; you can identify the location of things, but it's not well-defined within that space how many "dimension" components you'll need. Now, there actually is a natural visual mapping to all this. You have your "x.y.z = a.b.c(q, q2)" in a function, and you make it so you can click on anything, and it "zooms in" on that portion, in a way that the original expression is now left behind and gone. The problem with this natural visual mapping is you end up with a lot less context shown rather than more in the obvious, naive implementation. (Another problem is that this is basically nothing more or less than "Jump to definition", a thing I generally already have in my textual IDE.) The next obvious elaboration is to try to "expand" instead of zooming in, but this rapidly becomes impossible for all kinds of visual reasons. Maybe if we had 5 or 10 or 20 spatial dimensions to work in this wouldn't blow up so immediately, but holy crap is a mere 2 or 2.5 dimensions a kick in the pants to work with here. There's actually two possible solutions. One is to come up with some clever visualization, the usual approach. The other approach is to attack the root of the problem and try to come up with a programming language with a much lower n-dimensionality. The latter is arguably what having s-expressions has to do with any of this, but it's still not really low enough. The problem with the latter is the resulting language is pretty unpleasant still. We kinda had them in the past, and nobody's rushing back to them. Even the crazed whackos writing Turing-incomplete languages aren't doing what you'd need for this. The need for n-dimensionality, and the value of it, just come out too readily in programming. I don't think this is a result of us having text and leaning into it, though there may be an element of that; there is intrinsic benefit in being able to take what was a constant int in the previous version of the code and inserting an arbitrarily complicated function to obtain that int some other way instead. I might just be too blinded by my own experience to be able to conceive of programming any other way, but I am even so inclined to think that's a fundamental thing everyone, everywhere will always want. And if you have that, you've introduced n-dimensionality. Kinda like accidental Turing completeness; sealing this away and still having something useful seems a non-trivial challenge to me. Text has that characteristic where it's a well-defined operation to wedge new dimensions in anywhere, e.g. replacing a 1 with a new more complicated expression, with reasonable, if not perfect, cost/benefit tradeoffs. I haven't seen any other solutions to this. Given the number of whacks taken at it, it is at least a rational conclusion to think there may not be one. But even "rational" conclusions can still be wrong (the great rationalist conundrum, but that's a separate post). If someone does crack it, from either angle, I won't complain. We can tell that in constrained problems it can still do OK. Obviously visual programming is used in some places completely routinely in some places today. But I've never seen an approach that would handle, say, the Linux kernel any better than text. I would personally say that super-high-dimensional spaces are challenging. There are ways it will trip you up for sure, like the well-known "spiky spheres" which are themselves not "real" in the way we would traditionally think of them. But n-dimensional spaces can be actively perverse; they can trip you up very effectively even with relatively small instances, your every attempt to apply some simplifying assumption (even for the purposes of visual display) being rapidly and frequently destroyed by some falsification of that assumption, e.g, "function calls usually don't have more than 3 or 4 parameters" or "functions generally only have 50-100 lines tops" or... the list goes on. Which is another way of viewing why visual approaches to this problem have fared so poorly historically.
- runlaszlorun 4y ago
  Cool, thx for writing these up. Def some useful points for me to chew on.
throwbadubadu 4y ago
I think you are, text is just both more easier and more flexible. Visual programming always gets supercomplex quick and in this situations make it harder to grok any detail. imo (have been there too btw. ;) ) But still, maybe someone didn't just find the right approach, so don't feel disillusioned!
- bcrosby95 4y ago
  I have friends that work in game dev and over the years they have shared some "amazing" screenshots of blueprints on our discord server. I've seen a lot of people in business point to it as something to uphold - that if we were writing software correctly, it would be capable of that. But it still requires fairly technical people, they can still create abominations, and oftentimes they're used as prototypes for designers until the programming team can come in and write something that is actually maintainable and performant.
  - runlaszlorun 4y ago
    By “blueprints”, do you mean the visual editors the big game engines tend to have? I think Unreal calls theirs blueprints. I’ve never done any game dev but find it an interesting compare/contrast with typical web dev approaches since they push some boundaries that typical business application doesn’t.
    - bcrosby95 4y ago
      Yes. If you want some of the worst examples search for ue blueprint spaghetti. Also my brother works in business process management. They make similar tools but for businesses. So it does exist.
hathawsh 4y ago
In programming, we always build something initially that we can understand intuitively, then once we've gained experience with that program, more things become intuitive and we expand the software in ways we couldn't have foreseen. We change the code in ways that seem a little crazy and unlikely to work well at first, but with experience the changes become completely sensible. For example, some arbitrary piece of the code that starts out as a constant may later become a variable, and then a database table, and then a user interaction, and then a machine learning algorithm, and then an API call, and then a variable again. Source code is the best way to allow arbitrary evolution of code. Visual programming cripples the kind of evolution that is possible or reasonable. Now, what might work would be a component designer where all the state of the designer is stored as the source code in human-editable form. It might be interesting to be able to right-click a class or function call in source code and "Edit as Component". If that improves my productivity, it might be worth paying for. OTOH, such a thing now has to compete with AI code generators.
yencabulator 4y ago
One common failure mode of previous visual code editing projects has been forcing the program to be grammatically valid at every moment, and maybe even that each identifier is defined at all times. This doesn't always match how people like to initially write or edit programs.
- munificent 4y ago
  It would be fascinating to instrument a programming text editor to capture the entire chain of user input. Then go through that log and determine: 1. How many edits don't preserve the AST structure of a program. Changing a leaf expression to a different expression does, as does, say, changing a `for` loop to a `while` loop. But cutting the `(cond)` part of `if (cond)` out (including the parentheses) and hoisting it to a separate expression does not because now those parentheses mean something different. 2. How often the program is in a grammatically invalid state across multiple edits. I suspect fans of always-valid visual editors massively underestimate the size of both of those.
runlaszlorun 4y ago
OK, cool. Thx for the all the inputs. If I get any where with it, it’ll hopefully pop up in HN. I know it may be a fools’ errand, but I’m always willing to step up to the plate… :)
imtringued 4y ago
You are completely ignoring that programs are built incrementally and taking them apart on the symbol layer is actually pretty common.

duped 4y ago

> Am I missing anything on the advantages of text? This is an honest attempt to see where I’m missing perspective, not an attempt to convince. Well-defined semantics for search, diff, insert, and replace.

jerf 4y ago

"Am I missing anything on the advantages of text? This is an honest attempt to see where I’m missing perspective, not an attempt to convince." Acknowledged. My answer is, programming is n-dimensional and text is one of the few mechanisms we have for dealing with that. I believe this is actually the characteristic of it that has screwed so many other efforts, especially visual ones. I don't know if n-dimensional has a formal, accepted definition, but the characteristic of an n-dimensional space that is of interest to me is that at any point, suddenly a new dimension can be introduced, and this is indefinitely recursive. Let me try to demonstrate this with a web site first. Imagine there's a web site that has a home page with ten internal links. Each of these ten links then has a set of ten pages that they link to, all unique, and those pages link nowhere. You could represent this particular website in a clean 2D spreadsheet. In the A column, you have the ten top-level links, and then in B through K you can have the subsidiary pages. In this simplified model, a web page can be imagined as something like a linear collection of links, so I can describe a page as being "Link 4, Link 7", identifying the subpage from the 4th top-level link, then the 7th link of that page. Of course, this is a highly contrived example. Perhaps only and exactly one of those links goes to another top-level page with another set of 10 links that each have ten pages. So now, in the middle of this nice two-dimensional layout, in exactly one cell, suddenly you have another two dimensional layout appearing, and neither of those two dimensions is the same as the previous two dimensions, it's a fresh set. Nothing stops these pages from having another fresh set of dimensions appearing. In n-dimensional space, you never know when you're going to get a fresh set of dimensions when you cross over some unassuming-looking edge in the graph. In reality, web pages are so complexly interlinked that there isn't any visual way of representing them in the general case. My example is obviously super unrealistic. We normally wouldn't even dream of trying to represent a website in a spreadsheet like that. That doesn't mean the situation is hopeless necessarily, but it does mean we will make compromises. A sufficiently static site might have a canonical "site map" that completely organizes the site into a tree. But that is a simplified projection of a human conceptualization of the space; it isn't the real linking map of the web site. It is valuable, but it's only a particularly humanly-compelling slice. In general, trying to collapse an n-dimensional space into a finite number of dimensions is not something you can do. Especially when the number you're trying to collapse it into is as small as "two" or "three"! This, again, doesn't mean the situation is hopeless. There's a lot of angles you can take on that "in general". "In general" is, as the saying goes, doing some heavy lifting in that statement. But it is an important aspect of understanding the problem. How does text handle the n-dimensionality of code? Consider it from the point ofview of the language grammar. If I have x = 1 + 1 each of those "1"s is actually an "expression" in the grammar (or some local equivalent), which means I can do: x = 1 + (readint() + readint()) and now I've expanded that expression into a new tree. For the purposes of a visual editor, not only have I expanded that 1 into two function calls, I've also got to have a link to the functions in question. And then this nests infinitely recursively. And programmers keep coming up not only with new dimensions through the basic n-dimensionality of text conforming to tree-based grammar structures, but then new ways of adding dimensions; "x.y" as a field reference of a particular type is one thing, but then x can be polymorphic. Or decorated. Or monkey-patched. Or perhaps the language has multiple dispatch. Or overloading operators, meaning that operators now need to be be something you can follow to an implementation. Or... the list goes on. You end up with a statement like "x.y.z = a.b.c(q, q2)" having potentially dozens of graph edges departing from it, each to a declaration that may itself be a rats nest of graph edges, each to a further declaration that may be a rats nest of edges... eventually, everything has to terminate in some sort of literal or atomic value of some sort, but we want those complex things. It's why we're not programming in raw assembler. (Though that would raise its own problems.) Generally, we don't even think of these spaces as being "dimensional", which is to say, able to identify members by a specification of numbers in some number of "dimensions". But it is a thing we do. You can see this in programming in a very simple form pretty much anywhere you see dot-separated values of arbitrary length. A number of standards have some sort of 2.8.183.2.18.1377273 embedded in them. DNS names function this way too; behind every DNS dot could be another arbitrarily large and complicated subdomain scheme. You could proceed down a dozen simple elements and then only suddenly see an explosion of complexity. If you imagined a perfectly static web, you could theoretically (non-uniquely) identify web page locations by starting with a top-level catalog like Yahoo, and identifying web pages via the path you need to take to get there, like the number above. In reality, the space itself is shifting too fast for this to work, isn't constant, etc. But that's what n-dimensional looks like; you can identify the location of things, but it's not well-defined within that space how many "dimension" components you'll need. Now, there actually is a natural visual mapping to all this. You have your "x.y.z = a.b.c(q, q2)" in a function, and you make it so you can click on anything, and it "zooms in" on that portion, in a way that the original expression is now left behind and gone. The problem with this natural visual mapping is you end up with a lot less context shown rather than more in the obvious, naive implementation. (Another problem is that this is basically nothing more or less than "Jump to definition", a thing I generally already have in my textual IDE.) The next obvious elaboration is to try to "expand" instead of zooming in, but this rapidly becomes impossible for all kinds of visual reasons. Maybe if we had 5 or 10 or 20 spatial dimensions to work in this wouldn't blow up so immediately, but holy crap is a mere 2 or 2.5 dimensions a kick in the pants to work with here. There's actually two possible solutions. One is to come up with some clever visualization, the usual approach. The other approach is to attack the root of the problem and try to come up with a programming language with a much lower n-dimensionality. The latter is arguably what having s-expressions has to do with any of this, but it's still not really low enough. The problem with the latter is the resulting language is pretty unpleasant still. We kinda had them in the past, and nobody's rushing back to them. Even the crazed whackos writing Turing-incomplete languages aren't doing what you'd need for this. The need for n-dimensionality, and the value of it, just come out too readily in programming. I don't think this is a result of us having text and leaning into it, though there may be an element of that; there is intrinsic benefit in being able to take what was a constant int in the previous version of the code and inserting an arbitrarily complicated function to obtain that int some other way instead. I might just be too blinded by my own experience to be able to conceive of programming any other way, but I am even so inclined to think that's a fundamental thing everyone, everywhere will always want. And if you have that, you've introduced n-dimensionality. Kinda like accidental Turing completeness; sealing this away and still having something useful seems a non-trivial challenge to me. Text has that characteristic where it's a well-defined operation to wedge new dimensions in anywhere, e.g. replacing a 1 with a new more complicated expression, with reasonable, if not perfect, cost/benefit tradeoffs. I haven't seen any other solutions to this. Given the number of whacks taken at it, it is at least a rational conclusion to think there may not be one. But even "rational" conclusions can still be wrong (the great rationalist conundrum, but that's a separate post). If someone does crack it, from either angle, I won't complain. We can tell that in constrained problems it can still do OK. Obviously visual programming is used in some places completely routinely in some places today. But I've never seen an approach that would handle, say, the Linux kernel any better than text. I would personally say that super-high-dimensional spaces are challenging. There are ways it will trip you up for sure, like the well-known "spiky spheres" which are themselves not "real" in the way we would traditionally think of them. But n-dimensional spaces can be actively perverse; they can trip you up very effectively even with relatively small instances, your every attempt to apply some simplifying assumption (even for the purposes of visual display) being rapidly and frequently destroyed by some falsification of that assumption, e.g, "function calls usually don't have more than 3 or 4 parameters" or "functions generally only have 50-100 lines tops" or... the list goes on. Which is another way of viewing why visual approaches to this problem have fared so poorly historically.

runlaszlorun 4y ago
Cool, thx for writing these up. Def some useful points for me to chew on.

throwbadubadu 4y ago

I think you are, text is just both more easier and more flexible. Visual programming always gets supercomplex quick and in this situations make it harder to grok any detail. imo (have been there too btw. ;) ) But still, maybe someone didn't just find the right approach, so don't feel disillusioned!

bcrosby95 4y ago
I have friends that work in game dev and over the years they have shared some "amazing" screenshots of blueprints on our discord server. I've seen a lot of people in business point to it as something to uphold - that if we were writing software correctly, it would be capable of that. But it still requires fairly technical people, they can still create abominations, and oftentimes they're used as prototypes for designers until the programming team can come in and write something that is actually maintainable and performant.
- runlaszlorun 4y ago
  By “blueprints”, do you mean the visual editors the big game engines tend to have? I think Unreal calls theirs blueprints. I’ve never done any game dev but find it an interesting compare/contrast with typical web dev approaches since they push some boundaries that typical business application doesn’t.
  - bcrosby95 4y ago
    Yes. If you want some of the worst examples search for ue blueprint spaghetti. Also my brother works in business process management. They make similar tools but for businesses. So it does exist.

hathawsh 4y ago

In programming, we always build something initially that we can understand intuitively, then once we've gained experience with that program, more things become intuitive and we expand the software in ways we couldn't have foreseen. We change the code in ways that seem a little crazy and unlikely to work well at first, but with experience the changes become completely sensible. For example, some arbitrary piece of the code that starts out as a constant may later become a variable, and then a database table, and then a user interaction, and then a machine learning algorithm, and then an API call, and then a variable again. Source code is the best way to allow arbitrary evolution of code. Visual programming cripples the kind of evolution that is possible or reasonable. Now, what might work would be a component designer where all the state of the designer is stored as the source code in human-editable form. It might be interesting to be able to right-click a class or function call in source code and "Edit as Component". If that improves my productivity, it might be worth paying for. OTOH, such a thing now has to compete with AI code generators.

yencabulator 4y ago

One common failure mode of previous visual code editing projects has been forcing the program to be grammatically valid at every moment, and maybe even that each identifier is defined at all times. This doesn't always match how people like to initially write or edit programs.

munificent 4y ago
It would be fascinating to instrument a programming text editor to capture the entire chain of user input. Then go through that log and determine: 1. How many edits don't preserve the AST structure of a program. Changing a leaf expression to a different expression does, as does, say, changing a `for` loop to a `while` loop. But cutting the `(cond)` part of `if (cond)` out (including the parentheses) and hoisting it to a separate expression does not because now those parentheses mean something different. 2. How often the program is in a grammatically invalid state across multiple edits. I suspect fans of always-valid visual editors massively underestimate the size of both of those.

runlaszlorun 4y ago

OK, cool. Thx for the all the inputs. If I get any where with it, it’ll hopefully pop up in HN. I know it may be a fools’ errand, but I’m always willing to step up to the plate… :)

imtringued 4y ago

You are completely ignoring that programs are built incrementally and taking them apart on the symbol layer is actually pretty common.