4 ms·

The problem becomes if the form you're used to working with the code in is very different from the form you need to communicate about it in. In practice every

by vidarh 2y ago

The problem becomes if the form you're used to working with the code in is very different from the form you need to communicate about it in. In practice every attempt I've looked at either become hard to communicate about the code in, or the visual aspect tends to end up just becoming a secondary visualisation of code that you still treat as textual first. In the latter case, turning it into a better UI is an unsolved problem, because round tripping reliably between something readable as text and a meaningful visual representation is really hard. As I said, I hope someone solves this, but most attempts aren't even pushing the boundaries into uncharted territory - a lot of attempts have been made over the years.

adastra22 2y ago
Better to round-trip to the AST, and have the textual representation be derivative of that (e.g. by a code formatted like Go). This also makes it easier to verify that you have all the same capabilities in both representations, as the ways of manipulating the AST are enumerable.
- vidarh 2y ago
  Many people have tried that. I built a language and UI around that way back, and many others have. I ditched mine because there were way too many unsolved problems I felt made it useless. The problem is that if your primary means of working with the code is visual, the textual representation of your code then tends to be foreign to you when you're trying to use it to communicate aspects of the code, and when you constrain yourself to something that can be represented in a readable manner in a textual form, it turns out to be really hard to get to a point where the visual form is easier to work with. E.g. something as basic as how you comment code in ways that roundtrips nicely is an unsolved problem. If I have code represented as a graph, I'd be inclined to want to label relationships and dataflows that would be hard to place textually in a way that is meaningful in a textual version and that would roundtrip back to labels in the right place in the visual version. I've not seen any attempts at visual code that gets even that right. I've not managed to get it right myself either. If you force users to use an editor built into this tool, and edit a textual representation where some information is hidden, you can do better, but then if people e.g. copy a textual representation of the code into another application and back in, you end up with a mess. Again, I want to be proven wrong about this. Badly. I love the idea. I've just seen enough failed attempts (and made enough failed attempts) to be disillusioned about it.
  - adastra22 2y ago
    Why would you want to work with the text representation, except when debugging or in the backend? I mean I get why you'd want the text representation to exist--we have mountains of infrastructure around text-based representations of code. Git for version control and LLM code models would work out of the box, for example. But that can all be handled on the backend by transpiling the AST to text as needed. Why would the user need to interact with the textual representation? Commenting needs to be solved at the language level, and there are many languages that have solved this exact problem. Python, newLISP, and Smalltalk IIRC all have methods for docstring commenting APIs such that the docstring is available as text to the running program / REPL. Use similar syntax to allow any statement to have comments attached, and use this instead of free-form /* */ comments.
    - vidarh 2y ago
      > Why would you want to work with the text representation, except when debugging or in the backend? How would I communicate about the project to others in e-mails, instant messenger, face to face, in blog posts, in articles, in books? How would I review diffs of code changes effectively? That is why. Find me a representation I can talk about and write about efficiently without screenshots or videos or requiring special software of every recipient on every platform, and you'll have advanced the state of the art in this field immensely. > that have solved this exact problem None of the ones you described have solved the problem of mapping between a visual and textual representation of the program seamlessly. Just attaching the comments from a textual version to an AST of the textual version is trivial. That's not the challenge. > Use similar syntax to allow any statement to have comments attached, and use this instead of free-form /* / comments. That doesn't get close to solving the issue. When I have a diagram showing the data flow of a piece of code, and I attach a comment to the edge* between the two nodes, in the textual representation where does that comment go? Does it go in the text version of the source node? In the destination node? What if I write a comment in the textual version right before a method call, and then switch to the visual version, does that stay in the source node? Does it become a label of the edge representing the method call? There are tons of edge cases there. The problem isn't finding a way to attach the comments in the right place, but finding a way that roundtrips perfectly without adding noise in either representation.
worldsayshi 2y ago
Part of the solution is probably to rethink the whole UX context. One interesting example to think about is ProtoFlux, a part of Resonite: https://youtu.be/70PH5cQQEdQ?si=y4YmhnimzferVpCD https://youtu.be/70PH5cQQEdQ?si=y4YmhnimzferVpCD Since you develop inside vr you can also talk about and show the code in the same vr environment. It's probably at this level we need to rethink stuff to make visual programming practical.
- vidarh 2y ago
  But then you want to write about it and explain it to someone else, and you're back to needing a way to represent it that fits on a page or can be explained verbally.
  - worldsayshi 2y ago
    ProtoFlux is largely explained through tutorial videos I think. And I'm not saying that is superior. But I can imagine a lot of quality of life features around explaining it through voice that could be added to make it potentially superior or at least competitive. You could for example explain every node verbally and visually to an AI bot that can then explain it to the next person, or selectively retrieve parts of your explanations on demand. (OK, I realize that sounds unnecessary complex.)
    - vidarh 2y ago
      Requiring me to watch videos to follow along is an absolute non-starter to me. It's way too slow. If it was just learning an environment maybe I could tolerate that, but the showstoppers is communicating about code for projects. An AI bot doesn't solve this - if you can't relay the information textually, there's no reason the AI bot will be able to.
      - worldsayshi 2y ago
        Are you saying visual programming languages can only work if they can be explained with text?
conartist6 2y ago
The trick is that you don't round trip. You choose one immutable data structure that captures both the textual source of the program and the semantic information captured by the parser at the same time.
- vidarh 2y ago
  That doesn't solve anything. The problem isn't how to represent the AST, but defining both a visual and textual version that can unambiguously represent the same thing without either or both representational becoming unusable.
  - conartist6 2y ago
    Right. I think it's already a solved problem though. HTML and the HTML DOM are a living solution to that exact problem. All that we have to do is take the patterns used to power general UI and develop a DOM for code.
    - vidarh 2y ago
      The internal representation isn't the problem, or even a problem. It's not even beginning to address what I described. E.g this is a real line of code: link = wf["links"].find{ _1["rel"] == "self" && _1["type"] == "application/activity+json" } Now consider I have a visual programming version representing that expression, and I want to ask someones opinion about it on Slack. Unless your visual programming environment has a solution for how I can post that to Slack, and have others respond with tweaked versions, it's a non-starter. Once you've solved Slack - maybe with a plugin -, you need to solve all our e-mail clients, and you need to solve Google Docs and Word for when we write documentation, and a multitude of other tools. You might be able to get part of the way there with a browser plugin, but you'll still have a wide variety of other tools and platforms to cover.
      - conartist6 2y ago
        Your assumption is that visual programming and textual programming are disjoint, but I have not observed this to be true (necessarily). Being able to edit a program in a semantic fashion does not preclude its having syntax. I don't see syntax as a boondoggle to be eliminated, but rather the highest-bandwidth way we know how to convey information! I think you're right that the end game of any successful attempt at standardization is integration with all those tools like Outlook, Slack, Discord, Signal, Word, Docs, Notion... The list goes on and on. It's strange how the presence or absence of political momentum behind a standard could change that from being "basically impossible" to "basically inevitable"

by vidarh 2y ago

adastra22 2y ago

Better to round-trip to the AST, and have the textual representation be derivative of that (e.g. by a code formatted like Go). This also makes it easier to verify that you have all the same capabilities in both representations, as the ways of manipulating the AST are enumerable.

vidarh 2y ago
Many people have tried that. I built a language and UI around that way back, and many others have. I ditched mine because there were way too many unsolved problems I felt made it useless. The problem is that if your primary means of working with the code is visual, the textual representation of your code then tends to be foreign to you when you're trying to use it to communicate aspects of the code, and when you constrain yourself to something that can be represented in a readable manner in a textual form, it turns out to be really hard to get to a point where the visual form is easier to work with. E.g. something as basic as how you comment code in ways that roundtrips nicely is an unsolved problem. If I have code represented as a graph, I'd be inclined to want to label relationships and dataflows that would be hard to place textually in a way that is meaningful in a textual version and that would roundtrip back to labels in the right place in the visual version. I've not seen any attempts at visual code that gets even that right. I've not managed to get it right myself either. If you force users to use an editor built into this tool, and edit a textual representation where some information is hidden, you can do better, but then if people e.g. copy a textual representation of the code into another application and back in, you end up with a mess. Again, I want to be proven wrong about this. Badly. I love the idea. I've just seen enough failed attempts (and made enough failed attempts) to be disillusioned about it.
- adastra22 2y ago
  Why would you want to work with the text representation, except when debugging or in the backend? I mean I get why you'd want the text representation to exist--we have mountains of infrastructure around text-based representations of code. Git for version control and LLM code models would work out of the box, for example. But that can all be handled on the backend by transpiling the AST to text as needed. Why would the user need to interact with the textual representation? Commenting needs to be solved at the language level, and there are many languages that have solved this exact problem. Python, newLISP, and Smalltalk IIRC all have methods for docstring commenting APIs such that the docstring is available as text to the running program / REPL. Use similar syntax to allow any statement to have comments attached, and use this instead of free-form /* */ comments.
  - vidarh 2y ago
    > Why would you want to work with the text representation, except when debugging or in the backend? How would I communicate about the project to others in e-mails, instant messenger, face to face, in blog posts, in articles, in books? How would I review diffs of code changes effectively? That is why. Find me a representation I can talk about and write about efficiently without screenshots or videos or requiring special software of every recipient on every platform, and you'll have advanced the state of the art in this field immensely. > that have solved this exact problem None of the ones you described have solved the problem of mapping between a visual and textual representation of the program seamlessly. Just attaching the comments from a textual version to an AST of the textual version is trivial. That's not the challenge. > Use similar syntax to allow any statement to have comments attached, and use this instead of free-form /* / comments. That doesn't get close to solving the issue. When I have a diagram showing the data flow of a piece of code, and I attach a comment to the edge* between the two nodes, in the textual representation where does that comment go? Does it go in the text version of the source node? In the destination node? What if I write a comment in the textual version right before a method call, and then switch to the visual version, does that stay in the source node? Does it become a label of the edge representing the method call? There are tons of edge cases there. The problem isn't finding a way to attach the comments in the right place, but finding a way that roundtrips perfectly without adding noise in either representation.

worldsayshi 2y ago

Part of the solution is probably to rethink the whole UX context. One interesting example to think about is ProtoFlux, a part of Resonite: https://youtu.be/70PH5cQQEdQ?si=y4YmhnimzferVpCD https://youtu.be/70PH5cQQEdQ?si=y4YmhnimzferVpCD Since you develop inside vr you can also talk about and show the code in the same vr environment. It's probably at this level we need to rethink stuff to make visual programming practical.

vidarh 2y ago
But then you want to write about it and explain it to someone else, and you're back to needing a way to represent it that fits on a page or can be explained verbally.
- worldsayshi 2y ago
  ProtoFlux is largely explained through tutorial videos I think. And I'm not saying that is superior. But I can imagine a lot of quality of life features around explaining it through voice that could be added to make it potentially superior or at least competitive. You could for example explain every node verbally and visually to an AI bot that can then explain it to the next person, or selectively retrieve parts of your explanations on demand. (OK, I realize that sounds unnecessary complex.)
  - vidarh 2y ago
    Requiring me to watch videos to follow along is an absolute non-starter to me. It's way too slow. If it was just learning an environment maybe I could tolerate that, but the showstoppers is communicating about code for projects. An AI bot doesn't solve this - if you can't relay the information textually, there's no reason the AI bot will be able to.
    - worldsayshi 2y ago
      Are you saying visual programming languages can only work if they can be explained with text?

conartist6 2y ago

The trick is that you don't round trip. You choose one immutable data structure that captures both the textual source of the program and the semantic information captured by the parser at the same time.

vidarh 2y ago
That doesn't solve anything. The problem isn't how to represent the AST, but defining both a visual and textual version that can unambiguously represent the same thing without either or both representational becoming unusable.
- conartist6 2y ago
  Right. I think it's already a solved problem though. HTML and the HTML DOM are a living solution to that exact problem. All that we have to do is take the patterns used to power general UI and develop a DOM for code.
  - vidarh 2y ago
    The internal representation isn't the problem, or even a problem. It's not even beginning to address what I described. E.g this is a real line of code: link = wf["links"].find{ _1["rel"] == "self" && _1["type"] == "application/activity+json" } Now consider I have a visual programming version representing that expression, and I want to ask someones opinion about it on Slack. Unless your visual programming environment has a solution for how I can post that to Slack, and have others respond with tweaked versions, it's a non-starter. Once you've solved Slack - maybe with a plugin -, you need to solve all our e-mail clients, and you need to solve Google Docs and Word for when we write documentation, and a multitude of other tools. You might be able to get part of the way there with a browser plugin, but you'll still have a wide variety of other tools and platforms to cover.
    - conartist6 2y ago
      Your assumption is that visual programming and textual programming are disjoint, but I have not observed this to be true (necessarily). Being able to edit a program in a semantic fashion does not preclude its having syntax. I don't see syntax as a boondoggle to be eliminated, but rather the highest-bandwidth way we know how to convey information! I think you're right that the end game of any successful attempt at standardization is integration with all those tools like Outlook, Slack, Discord, Signal, Word, Docs, Notion... The list goes on and on. It's strange how the presence or absence of political momentum behind a standard could change that from being "basically impossible" to "basically inevitable"