15 ms·

Graphical Programming: I Really Hope This Is the Future

44 points by pcmonk 13y ago

danmaz74 13y ago
Shouldn't an article about graphical programming show something graphical, instead of just ascii characters?
- Doublon 13y ago
  That was exactly my thought. This article is just plain text. It feels like something is missing.
  - CmonDev 13y ago
    It's a "wouldn't it be nice" kind of article. Concept art would be nice even if there is no intent to make a prototype.
    - pcmonk 13y ago
      Actually, I do have a prototype; I'm just not ready to release it yet. So, it's probably best to interpret this as a general discussion of the subject.
- mathattack 13y ago
  The precision needed for typing highlights why graphical programming is so tough.
- pcmonk 13y ago
  I've been spending all my time writing the code. Concept art isn't worth the time investment for me right now. I could post screenshots, but I don't think it's quite ready for that.
  - danmaz74 13y ago
    Well, it depends on what is your objective with the blog post. If it is to start creating an expectation and receiving feedback from people who would be interested, it would be very useful to spend some time for a couple of schematics or mock-ups, especially because your best target is quite likely very visual oriented. If you just want to let out your thoughts, then, no problem!
    - pcmonk 13y ago
      Yeah, I was really just letting my thoughts out in an almost stream-of-consciousness style. I posted to HN more just to see how the process works than because I thought people would actually read my post. After 9000 views (from both here and reddit) and some excellent feedback, though, I see that there is some interest in this type of thing. My next post should definitely have more substance to it.
      - danmaz74 13y ago
        Ok then, good luck with your project! I'll take a look again when there is something visual to see ;)
rtb 13y ago
What's the difference between this effort and the countless Graphical Programming efforts that have come before (many with large teams as well as many other single-author efforts)?
- Jare 13y ago
  This is still at the "idea" or "vision" level and hasn't failed yet. Sorry about the snark, but while I'd love someone to succeed in this type of quest, I believe this one is based more on ideology than rigor. Stuff like "WYSIWYG editors are basically evil" just doesn't give me confidence.
buro9 13y ago
Show, don't tell.
jdmitch 13y ago
Are you looking for others to test/collaborate on what you have done so far?
stinos 13y ago
I doubt this is THE future, and I also hope it is not: still too limited. As far as I can predict, the future is a proper direct brain-machine interface. Controversial yes, avoidable: I seriously doubt it. (would I use it: maybe if it could be implemented using one way communication, ie similar to how it's done now by analyzing EEG). I guess it would also be by far the fastest way. At least I can think code (both concepts and actual implementation) way faster than I can output it in any way.
- Houshalter 13y ago
  The rate of inputting information to the computer has never been the limiting factor for me. Displaying the information in more helpful format could be useful though.
- francis- 13y ago
  Good to know someone out there 'gets it'.
- Someone 13y ago
  You only need one way; evolution has provided plenty of ways for reverse communication.
- voyou 13y ago
  I have in front of me a system that uses electromagnetic signals to interface between my computer and my brain; part of it even works wirelessly - it's a high-tech system called "a screen and my eyes." In what way is your predicted "direct brain-machine interface" better than my actually-existing electromechanical brain-machine interface? : I can think code (both concepts and actual implementation) way faster than I can output it in any way. I doubt you can. You can have the psychological feeling of thinking you've developed the concepts much quicker, but in my experience it's only when I start doing the actual implementation that the full level of precision I need becomes clear, and its that level of precision that slows me down, not the need to physically press keys.
  - stinos 13y ago
    it's a high-tech system called "a screen and my eyes." I meant the interface the other way around. From your brain to the computer, without any mechanical inert pieces in between. As for the prediction part, note that it exists already in a very early stage and it's extremely likely to only grow. Can't find the exact artical atm, but it was used to move a mouse on screen by someone who couldn't move anything below their head anymore. I doubt you can Maybe not on all levels, 100% sure I can for some. Simple example: C++, I need a basic iteration over a sequence. I just think 'ok gimme a for_each on container Foo, calling function Bar'. No matter how advanced the intellisense or whatnot is, it will take me more time to properly type the needed line.
RivieraKid 13y ago
I have been thinking about this too recently and I'm thinking about writing a simple IDE to demostrate these ideas. If anyone would like to discuss this over mail (or here of course), let me know.
vidarh 13y ago
People have tried this. Over and over and over. I love the idea, and I too have tried (and scrapped my effort), but it is an immensely difficult problem to tackle for a number of reasons: - The moment you force a user to pick up the mouse, you have lost. - It seems very compelling to augment the display with more information, but a lot of it is simply distracting, or reduce the density of relevant information that will fit on the page. There's a reason that even in most IDE's very little additional information beyond syntax highlighting have "stuck". - Some weirdos (like me) won't touch tools that won't let me work over a dumb ssh connection and terminal when I need to, which is one of many reasons why... - ... everything needs to have a sane textual format. Another reason is that it is essential to communicating about the code. If you want to think about this, at least thoroughly review the long history of visual languages, as there are plenty of mistakes to repeat.
- seanmcdirmid 13y ago
  >There's a reason that even in most IDE's very little additional information beyond syntax highlighting have "stuck". We are working on it; e.g. http://research.microsoft.com/en-us/people/smcdirm/liveprogramming.aspx http://research.microsoft.com/en-us/people/smcdirm/liveprogr...
- splawn 13y ago
  I think visual programming works well if its geared towards a specific problem set. In the past I used Talend ETL to generate small java programs and it worked well. However I couldn't imagine trying to do something big with a general visual language.
- ArbitraryLimits 13y ago
  > People have tried this. Over and over and over. Seriously, it's the future and it has been for fifty years.
  - CmonDev 13y ago
    So it will probably be a part of final HTML5 specification. Worst case HTML5.1.
lnanek2 13y ago
Everything's solvable and everyone else is wrong, but he has none of his own work to show. Very funny post. Maybe he's just a student? These constant failures at trying to implement visual programming often seem to come from academia where the goal is to write papers rather than actually accomplish anything.
TeeWEE 13y ago
I worked with numerous tools for graphical programming, mostly for academic work. Such as graph-transformation based tools for software verification, in which your programs are graphs. http://wwwhome.ewi.utwente.nl/~rensink/papers/spin2006.pdf http://wwwhome.ewi.utwente.nl/~rensink/papers/spin2006.pdf Which is useful for software verification, but writing your actual program in it is cumbersome. Software is a lot about abstractions, and there is no reason that graphical programs are better at representing abstraction that text (or maybe im wrong here?). In theory, that what you can represent in graphical-form you can also represent in textual form. However the information density with an 2D graphical form is much higher than textual form. And the more information you put into the same space, the more difficult it is to reason about it. Thats why textual programming is still very much popular.
super-serial 13y ago
I've always thought that a visual interface for programming would help with my procrastination problems. It's so painless for me to open up a webpage and solve daily chess puzzles... on the other hand it seems more painful to open up an IDE and start solving code "puzzles." I wish I could visualize all the pieces of my software similar to chess.
- Kurtz79 13y ago
  I absolutely doubt that graphical programming, even in its best implementation, will be about solving "code" puzzles. Regardless of the interface, programming is "work", linking up different interfaces, checking edge cases, conform to protocols... It would feel more like glorified data entry than solving chess puzzles.
sesm 13y ago
Did some LabView programming as a student, didn't enjoy the experience. Graphical programs are similar to circuits without memory, and thus are less powerful then Turing machines. As a result, graphical programs quickly grow in size, and growth happens in 2 dimensions, which makes it very hard to read them.
- Houshalter 13y ago
  To be fair, text is merely a special case of graphical programming where the output is limited to a handful of characters and is completely linear. Is it impossible to improve upon that? Syntax highlighting and indentation were good improvements for example.
  - lmm 13y ago
    It's possible to improve, but I think the key is progressive enhancement. Look at how IDEs tend to offer typing and implicit information in scala: it's there, it's very useful, but it doesn't stop you from seeing the plain text source code.
    - deleted 13y ago
      [deleted]
- ansible 13y ago
  Ugh, LabView. Do not want. One of my main gripes is lack of version control integration. I want to use something like git, and I want to be able to easily and succinctly see what's been changed by who. You have to look at the "code" only using their own GUI tool. Large functions involve lots of scrolling, and at least with Labview, I was not aware of any means by which to jump around to specific parts using hotkeys or such. You'd also have pages of logic stacked on top of each other, where you can only view one page at a time. If you're implementing something that is going to closely resemble a circuit of some kind, I suppose it is OK, but when things start to get complicated, I found the system cumbersome to navigate. This impedes comprehension, and invites mistakes. I've also used this other GUI-driven code generation system, and that had serious issues too. It has this hilarious graphical view of all the components, showing how they are connected to each other. However, due to the limited size of the window, it will just stack all the components almost on top of each other. If you started the laborious process to move them around to see what is going on, your layout would not be saved. So, anyway, a modest proposal: Figure out how to display an existing programming language (maybe a simple one like Lua) with a graphical view. Allow me to easily navigate through the code, and let me see all the details easily.
CmonDev 13y ago
This is what you get when you lock yourself in the *nix Vim-niverse. Try Visual Studio 2012 Ultimate: http://msdn.microsoft.com/en-us/library/vstudio/jj739835.aspx http://msdn.microsoft.com/en-us/library/vstudio/jj739835.asp... http://msdn.microsoft.com/en-us/library/vstudio/dd409365.aspx http://msdn.microsoft.com/en-us/library/vstudio/dd409365.asp... PS: pricing is the sad bit :(, but there is no competition in this area. I hope JetBrains will work harder.
- lcedp 13y ago
  Diagrams look like a mess to me: virtually everything is connected to anything in unknown way (no method names or arguments specified). Text at the left is much clearer.
- rubberband 13y ago
  I sit in front of Visual Studio more hours in a day than I sleep, and by and large, I love it. But I can't for the life of me understand why the mapping diagram would help. I suppose in some unfamiliar code it may help me get up to speed faster, but this seems like a rehash of exporting UML to Visio? I cannot afford Ultimate, just Professional, so I admit I have no hands on experience with the tool. What JetBrains product is comparable?
  - CmonDev 13y ago
    There is no JetBrains product like that, I just wish they had, because they have a potential to build something lightweight. Best use cases for me personally are the visual dependency graph and method call diagrams when dealing with poor-quality code (huge classes with a lot of method dependencies). I would pay a hundred bucks for this, but not a thousand dollars.
- dman 13y ago
  If youre dependent on auto generated diagrams to understand the structure of your code you have bigger problems.
  - CmonDev 13y ago
lcedp 13y ago
The future is what? Nothing's been proposed so far.
smoyer 13y ago
In the '80s I was working on a (never released) system I called FlowPro, but EGA adapters and 8088 processors weren't really capable of the kind of DnD I wanted. The idea was that flow-charts would be turned into code, with some meta data added if you wanted specific variable names etc. (instead of the words used in the diagram's box). Then in the early-to-mid '90s, I came across a product called Prograph that ran on both Windows (I did a lot of programming for WFW with it) and Mac (OS 8 I believe). I loved that system, and realized it solved some of the syntax issues I had using flow-charts, but it still never really took off. The concept lives on with the Marten IDE though I'm not sure how many people are actively using it. https://en.wikipedia.org/wiki/Prograph https://en.wikipedia.org/wiki/Prograph http://andescotia.com/products/marten/ http://andescotia.com/products/marten/
collyw 13y ago
Users will use Excel. That is the closest thing we have to graphical programming that works. They don't need to understand data structures, just columns and rows. Anything more than that, and you need to understand new concepts. By the time you understand these, drag and drop doesn't gain you much over typing.
pizo 13y ago
The idea of graphical programming seems in a lot of way like graphical mathematics. Works in some cases, but not in all. I used graphics to represent databases, and their relationships, because it's more easily understood than several hundred lines of code. It fails to tell the whole story though, and thus only serves as a thousand foot view of what is going on.
- gshubert17 13y ago
  > Works in some cases, but not in all. Delphi (for Windows, 1995-on) had a very nice graphical IDE which was good for developing graphical user interfaces in an object-oriented programming style. The rest of the IDE was fine for editing and debugging. But it was great for GUI development.
- pcmonk 13y ago
  You're right about the similarity with graphical mathematics (and Bret Victor's Kill Math project was something of an inspiration for this (even though I have some fundamental disagreements with his methods), even though I didn't see it until after I'd started it). You've identified a critical problem with graphical programming today: it fails to tell the whole story. This is exactly what I referred to in the post, and that's not inherent in the medium; rather, it's just that people have been trying to solve this problem in the wrong way.
svantana 13y ago
Personally, I would prefer an IDE/debugger that allowed me to visualize my data really well. This ought to be a lot easier also - I've done some simple scripting with GnuPlot which has given me a lot of mileage. But imagine being able to put a breakpoint anywhere in your program and interactively browse your trees, dictionaries and arrays using OpenGL-accelerated real-time graphics. I don't understand why xCode and VS aren't putting more efforts into things like this.
- jamii 13y ago
  Sounds like this is what you want - https://github.com/Element84/vdd-core/wiki/What-is-Visualization-Driven-Development%3F https://github.com/Element84/vdd-core/wiki/What-is-Visualiza... EDIT: Or even http://www.youtube.com/watch?v=L6iUm_Cqx2s http://www.youtube.com/watch?v=L6iUm_Cqx2s
- goetz 13y ago
  A friend of mine is working precisely on this : a plugin for XCode for "visual debugging". What would you like to see in such a tool?
  - svantana 13y ago
    Sorry for the late reply. I'd love to be able to plot arrays of floats or ints as 2d or 3d graphs. If you've ever used Matlab, this is the sort of thing it excels at.
- CmonDev 13y ago
  Well actually Visual Studio team put efforts to address exactly this need: http://visualstudiogallery.msdn.microsoft.com/eedc48e7-5169-40d3-b602-ad451990a420 http://visualstudiogallery.msdn.microsoft.com/eedc48e7-5169-... http://blogs.msdn.com/b/vcblog/archive/2012/07/12/10329460.aspx http://blogs.msdn.com/b/vcblog/archive/2012/07/12/10329460.a... http://msdn.microsoft.com/en-us/library/zayyhzts.aspx http://msdn.microsoft.com/en-us/library/zayyhzts.aspx http://visualstudiogallery.msdn.microsoft.com/c7e02633-86d9-4262-b745-6cc647afb3a8 http://visualstudiogallery.msdn.microsoft.com/c7e02633-86d9-...
outworlder 13y ago
I can postulate that there is absolutely no way a purely graphical representation of code will be the future. The same way we do not talk using pictures. A graphical language would be best used to display structure. Bonus points if it allowed changing. The rest, modules if you will, can be better described by text.
- CmonDev 13y ago
  Don't want to argue with the spirit of your post, but Chinese hieroglyphics is derived from pictures and it's probably somewhat representative of their language.
drstrangevibes 13y ago
a wise man once said: "when evaluating a language examine its primitives, its means of combination and its means of abstraction". By these metrics I find graphics woefully inferior!
erock 13y ago
I've spent the past year "working" within graphical programming, and I have to say it is a slow form of torture. I have been "writing" IVR applications for a call center. The interface is far from intuitive, and it is jsut too easy to miss what is actually going on. The loops and if statements are all there, but you have to click through on the item to actually see what is going on. Forget trying to find an easy way to locate where something is. at least in my case (VXML) I was forced to use grep to at least give me an idea of where to look, then spend 10x the amount of time it would take in plain ol code to find where to make the change. ugh. give me ASCII code any day
brudgers 13y ago
[Terms of Service: Let me start by saying that I admire the author's passion and effort to create something wonderful, and that this is not a crap on their project, but rather a reaction to their essay. And the fact that their essay raised a reaction to the point where I wrote something about it is, in this case, a testament to its quality.] Lisp programs can be represented as trees because they are trees. They can also be represented as lists, and this is because they are also lists. The important difference between these representations is that trees the primary representation by which computers unravel the intent of a Lisp program and lists are the primary representation by which programmers express intent in their programs. Sure some people sometimes picture their lisp program as a tree, but usually when a programmer pictures something as a tree, it is when viewing it as a data structure. Picturing a Lisp program as a tree is easier than picturing the tree represented by code in most other languages because of Lisps simple parsing process, but the way in which a programming language is parsed is an arbitrary feature of the language [though one worth design consideration]. Although it is easy to draw a tree. Often in Lisp and other languages, the verbal diagram "a list of lists of ..." is adequate to represent a tree along with a description of its interesting properties. Again, while the interesting properties can be represented graphically, what makes them interesting are the maths underpinning their structure not the aesthetics of their spatial representation. "The purpose of this project is to assist the coder in developing more complex code faster and easier" This is orthogonal to Rich Hickey's goal of making it easier and faster to produce simple code. http://www.infoq.com/presentations/Simple-Made-Easy http://www.infoq.com/presentations/Simple-Made-Easy We already have a great tool which allows graphic methods to be used easily to create complex code. It is called Excel.
- pcmonk 13y ago
  Hm, interesting. I've always pictured my Lisp programs as trees. I think it's a more useful representation, but I see what you mean. You're right that this is orthogonal to Rich Hickey's goal. That's because I think his goal is great for writing simple programs. His goal is to make more programs simple, which is great. My goal is to make it easier to write complex code. As Fred Brooks talks about in No Silver Bullet, there are some problems that are accidental and some that are essential. Hickey is trying to solve the accidental problemms, and I'm trying to make it easier to solve the essential problems. Some logic is just inherently complex -- I want to create tools to make it easier to reason about. And Excel is not nearly programmable enough to be better than a textual interface to solve the same problems. It's benefit is that it can be used (relatively) easily by non-programmers.
bane 13y ago
http://en.wikipedia.org/wiki/DRAKON http://en.wikipedia.org/wiki/DRAKON
Zigurd 13y ago
There are some general problems with the way people think about graphical programming. First, they should stop thinking about it as executable flowcharts and take a step back: There must be a better way than typing characters and pseudo-words to create programs. But that way isn't necessarily any of the graphical programming systems anyone has yet come up with. This article is a good example of thinking that's not even near the walls of the box, much less outside. And what's that crazy digression about wysiwyg editors? And then the author drops this one: "Thus, this isn’t really about graphical programming." Derrrrrr. We are barely at a point where we admit that evaluating a language syntax in a vacuum is silly, so people are still surprised that "verbose" Java is popular, despite the advantages static types and an intermediate language confer on the editor, compiler, static analyzer, debugger, etc. A less than holistic approach to graphical languages is probably doomed.
taeric 13y ago
Again I look at how our industry completely ignores every other industry out there. Some things do lend themselves to graphical viewing. Some things do not. Many things lend themselves to a series of graphical representations. The vast majority of things benefit from both graphical representations and textual elaborations. One of the things I wish programming tools offered, was a way to "zoom in" details wise on a solution. More than just offering an outline, I want there to be some sort of representation of how things went from conception to realization. Consider, the story boards for movies/games. Those story boards are lost in the final product. Worse, when you are looking at a piece of code, you don't really see the "why" of what it was made for. Sure, you get some of this with descriptive names, but that really doesn't help with what I'm wanting to say here. An example, that I really want to explore some more is graphviz. It can go from text to graphics. This process is fairly static, though. If it doesn't turn out how you want it, you have to edit the text and fire it again. I want something that can possibly go from text, to initial graphics where I manipulate it some more, to final representation all in a repeatable way. (I think this is possible, but I am not familiar with the toolchain.) More, there is no reason I shouldn't be able to start from graphics, go to a textual details portion, at varying levels of detail, make edits, and view what effect they have in the viewer. Imagine, something similar to developer tools in chrome/firefox, where in addition to being able to look over at the boxmodel for details, you can physically move elements around. Of course, I'm a bit of a heretic. I would kind of like to be able to export absolute positioned style sheets for forms designed this way. :) Still not sure on all of the reasons that would be terrible.
- MetaCosm 13y ago
  ... wanting Visual Basic doesn't make you a heretic, it just makes you aching for a foregone era a bit. Visual coding has been pushed for ... at least 30 years. The thing you are looking for is all over the place, in various IDEs, in QT MOC, in literally DOZENS of VPL (Visual Programming Languages) and even to a lesser degree in UML (Unified Modeling Language). Since we have QT MOC in the mix, we could even through in ALL the meta compilation tools, which adds in hundreds of others. What you want isn't so much unique, as already done and discarded. You end up with tightly bound IDEs and code, you end up with hard to manage binary nonsense, you end up with ridiculously hard peer review and change tracking. As for tracking the "why" -- use any method you want -- my team uses -- story boards! They work perfect for software.
kyllo 13y ago
Good sell, I agree with all the points the author makes, but as Linus says, "Talk is cheap, show me the code."
- pcmonk 13y ago
  I'm working on it...
  - kyllo 13y ago
    Thanks, it sounds like a cool project. Please do post it here when you have an alpha version, I'd be interested to see it.
seanmcdirmid 13y ago
Was the article flagged?
vinceguidry 13y ago
I find myself looking for graphical interfaces to things I already know fairly well but don't really care to drop into shell every time I need to use it. I have an SSH tunnel manager and an rsync manager. I use Sequel Pro instead of the mysql command line. In fact, I really only find myself using the command line for, you know, running commands. If I find myself there a lot, I'll check myself and start to think, "can I start putting all these things I'm typing into a Rake task or shell script? After all, the more I'm there, the more chance for user error. I think the only major task I still use a terminal for is looking at log files. I intend to eliminate that soon enough when I can 12-factorify the app I'm responsible for and find a decent graphical log viewer.
chipsy 13y ago
I don't see graphics as a silver bullet, but I do think they're underused or misused. Graphics provide a wider variety of feedback; they let you see graph structures and summaries of data, and allow connections to be spatially represented. All of which means that they excel at connecting the big chunks of the system that build on top of some model of service provision and protocol definition, but poor at engaging in the "direct dialogue" of imperative or applicative computation. The part that makes people say "it never works" is that we're almost always thinking from an inverted model, where we know how things scale up and down with text, and apply that learned experience to graphical systems. Subsequently, our engineering instinct leads us to put a similar degree of granular agency into graphical coding and allow the standard constructs (looping, branching, subroutines) to appear in it, which massively increases the complexity of the system and brings us away from problem domain thinking. My recommendation now is - figure out a data structure that could be interesting to visually edit. Make a good editor for that. Allow imperative code a runtime model for accessing it. Allow imperative code to be called from it. Then it stands at least some chance of not being a bad wheel reinvention.
- pcmonk 13y ago
  I understand what you're saying. My hypothesis is that (some version of) Lisp itself, being a tree, could be interesting to visually edit. All my code, by the way, can be directly accessed from Racket (and vice-versa), so it's definitely not a requirement that the whole program be written graphically. I want to make it a smooth transition from textual to graphical programming.

44 points by pcmonk 13y ago

danmaz74 13y ago

Shouldn't an article about graphical programming show something graphical, instead of just ascii characters?

Doublon 13y ago
That was exactly my thought. This article is just plain text. It feels like something is missing.
- CmonDev 13y ago
  It's a "wouldn't it be nice" kind of article. Concept art would be nice even if there is no intent to make a prototype.
  - pcmonk 13y ago
    Actually, I do have a prototype; I'm just not ready to release it yet. So, it's probably best to interpret this as a general discussion of the subject.
mathattack 13y ago
The precision needed for typing highlights why graphical programming is so tough.
pcmonk 13y ago
I've been spending all my time writing the code. Concept art isn't worth the time investment for me right now. I could post screenshots, but I don't think it's quite ready for that.
- danmaz74 13y ago
  Well, it depends on what is your objective with the blog post. If it is to start creating an expectation and receiving feedback from people who would be interested, it would be very useful to spend some time for a couple of schematics or mock-ups, especially because your best target is quite likely very visual oriented. If you just want to let out your thoughts, then, no problem!
  - pcmonk 13y ago
    Yeah, I was really just letting my thoughts out in an almost stream-of-consciousness style. I posted to HN more just to see how the process works than because I thought people would actually read my post. After 9000 views (from both here and reddit) and some excellent feedback, though, I see that there is some interest in this type of thing. My next post should definitely have more substance to it.
    - danmaz74 13y ago
      Ok then, good luck with your project! I'll take a look again when there is something visual to see ;)

rtb 13y ago

What's the difference between this effort and the countless Graphical Programming efforts that have come before (many with large teams as well as many other single-author efforts)?

Jare 13y ago
This is still at the "idea" or "vision" level and hasn't failed yet. Sorry about the snark, but while I'd love someone to succeed in this type of quest, I believe this one is based more on ideology than rigor. Stuff like "WYSIWYG editors are basically evil" just doesn't give me confidence.

buro9 13y ago

Show, don't tell.

jdmitch 13y ago

Are you looking for others to test/collaborate on what you have done so far?

stinos 13y ago

I doubt this is THE future, and I also hope it is not: still too limited. As far as I can predict, the future is a proper direct brain-machine interface. Controversial yes, avoidable: I seriously doubt it. (would I use it: maybe if it could be implemented using one way communication, ie similar to how it's done now by analyzing EEG). I guess it would also be by far the fastest way. At least I can think code (both concepts and actual implementation) way faster than I can output it in any way.

Houshalter 13y ago
The rate of inputting information to the computer has never been the limiting factor for me. Displaying the information in more helpful format could be useful though.
francis- 13y ago
Good to know someone out there 'gets it'.
Someone 13y ago
You only need one way; evolution has provided plenty of ways for reverse communication.
voyou 13y ago
I have in front of me a system that uses electromagnetic signals to interface between my computer and my brain; part of it even works wirelessly - it's a high-tech system called "a screen and my eyes." In what way is your predicted "direct brain-machine interface" better than my actually-existing electromechanical brain-machine interface? : I can think code (both concepts and actual implementation) way faster than I can output it in any way. I doubt you can. You can have the psychological feeling of thinking you've developed the concepts much quicker, but in my experience it's only when I start doing the actual implementation that the full level of precision I need becomes clear, and its that level of precision that slows me down, not the need to physically press keys.
- stinos 13y ago
  it's a high-tech system called "a screen and my eyes." I meant the interface the other way around. From your brain to the computer, without any mechanical inert pieces in between. As for the prediction part, note that it exists already in a very early stage and it's extremely likely to only grow. Can't find the exact artical atm, but it was used to move a mouse on screen by someone who couldn't move anything below their head anymore. I doubt you can Maybe not on all levels, 100% sure I can for some. Simple example: C++, I need a basic iteration over a sequence. I just think 'ok gimme a for_each on container Foo, calling function Bar'. No matter how advanced the intellisense or whatnot is, it will take me more time to properly type the needed line.

RivieraKid 13y ago

I have been thinking about this too recently and I'm thinking about writing a simple IDE to demostrate these ideas. If anyone would like to discuss this over mail (or here of course), let me know.

vidarh 13y ago

People have tried this. Over and over and over. I love the idea, and I too have tried (and scrapped my effort), but it is an immensely difficult problem to tackle for a number of reasons: - The moment you force a user to pick up the mouse, you have lost. - It seems very compelling to augment the display with more information, but a lot of it is simply distracting, or reduce the density of relevant information that will fit on the page. There's a reason that even in most IDE's very little additional information beyond syntax highlighting have "stuck". - Some weirdos (like me) won't touch tools that won't let me work over a dumb ssh connection and terminal when I need to, which is one of many reasons why... - ... everything needs to have a sane textual format. Another reason is that it is essential to communicating about the code. If you want to think about this, at least thoroughly review the long history of visual languages, as there are plenty of mistakes to repeat.

seanmcdirmid 13y ago
>There's a reason that even in most IDE's very little additional information beyond syntax highlighting have "stuck". We are working on it; e.g. http://research.microsoft.com/en-us/people/smcdirm/liveprogramming.aspx http://research.microsoft.com/en-us/people/smcdirm/liveprogr...
splawn 13y ago
I think visual programming works well if its geared towards a specific problem set. In the past I used Talend ETL to generate small java programs and it worked well. However I couldn't imagine trying to do something big with a general visual language.
ArbitraryLimits 13y ago
> People have tried this. Over and over and over. Seriously, it's the future and it has been for fifty years.
- CmonDev 13y ago
  So it will probably be a part of final HTML5 specification. Worst case HTML5.1.

lnanek2 13y ago

Everything's solvable and everyone else is wrong, but he has none of his own work to show. Very funny post. Maybe he's just a student? These constant failures at trying to implement visual programming often seem to come from academia where the goal is to write papers rather than actually accomplish anything.

TeeWEE 13y ago

I worked with numerous tools for graphical programming, mostly for academic work. Such as graph-transformation based tools for software verification, in which your programs are graphs. http://wwwhome.ewi.utwente.nl/~rensink/papers/spin2006.pdf http://wwwhome.ewi.utwente.nl/~rensink/papers/spin2006.pdf Which is useful for software verification, but writing your actual program in it is cumbersome. Software is a lot about abstractions, and there is no reason that graphical programs are better at representing abstraction that text (or maybe im wrong here?). In theory, that what you can represent in graphical-form you can also represent in textual form. However the information density with an 2D graphical form is much higher than textual form. And the more information you put into the same space, the more difficult it is to reason about it. Thats why textual programming is still very much popular.

super-serial 13y ago

I've always thought that a visual interface for programming would help with my procrastination problems. It's so painless for me to open up a webpage and solve daily chess puzzles... on the other hand it seems more painful to open up an IDE and start solving code "puzzles." I wish I could visualize all the pieces of my software similar to chess.

Kurtz79 13y ago
I absolutely doubt that graphical programming, even in its best implementation, will be about solving "code" puzzles. Regardless of the interface, programming is "work", linking up different interfaces, checking edge cases, conform to protocols... It would feel more like glorified data entry than solving chess puzzles.

sesm 13y ago

Did some LabView programming as a student, didn't enjoy the experience. Graphical programs are similar to circuits without memory, and thus are less powerful then Turing machines. As a result, graphical programs quickly grow in size, and growth happens in 2 dimensions, which makes it very hard to read them.

Houshalter 13y ago
To be fair, text is merely a special case of graphical programming where the output is limited to a handful of characters and is completely linear. Is it impossible to improve upon that? Syntax highlighting and indentation were good improvements for example.
- lmm 13y ago
  It's possible to improve, but I think the key is progressive enhancement. Look at how IDEs tend to offer typing and implicit information in scala: it's there, it's very useful, but it doesn't stop you from seeing the plain text source code.
  - deleted 13y ago
    [deleted]
ansible 13y ago
Ugh, LabView. Do not want. One of my main gripes is lack of version control integration. I want to use something like git, and I want to be able to easily and succinctly see what's been changed by who. You have to look at the "code" only using their own GUI tool. Large functions involve lots of scrolling, and at least with Labview, I was not aware of any means by which to jump around to specific parts using hotkeys or such. You'd also have pages of logic stacked on top of each other, where you can only view one page at a time. If you're implementing something that is going to closely resemble a circuit of some kind, I suppose it is OK, but when things start to get complicated, I found the system cumbersome to navigate. This impedes comprehension, and invites mistakes. I've also used this other GUI-driven code generation system, and that had serious issues too. It has this hilarious graphical view of all the components, showing how they are connected to each other. However, due to the limited size of the window, it will just stack all the components almost on top of each other. If you started the laborious process to move them around to see what is going on, your layout would not be saved. So, anyway, a modest proposal: Figure out how to display an existing programming language (maybe a simple one like Lua) with a graphical view. Allow me to easily navigate through the code, and let me see all the details easily.

CmonDev 13y ago

This is what you get when you lock yourself in the *nix Vim-niverse. Try Visual Studio 2012 Ultimate: http://msdn.microsoft.com/en-us/library/vstudio/jj739835.aspx http://msdn.microsoft.com/en-us/library/vstudio/jj739835.asp... http://msdn.microsoft.com/en-us/library/vstudio/dd409365.aspx http://msdn.microsoft.com/en-us/library/vstudio/dd409365.asp... PS: pricing is the sad bit :(, but there is no competition in this area. I hope JetBrains will work harder.

lcedp 13y ago
Diagrams look like a mess to me: virtually everything is connected to anything in unknown way (no method names or arguments specified). Text at the left is much clearer.
rubberband 13y ago
I sit in front of Visual Studio more hours in a day than I sleep, and by and large, I love it. But I can't for the life of me understand why the mapping diagram would help. I suppose in some unfamiliar code it may help me get up to speed faster, but this seems like a rehash of exporting UML to Visio? I cannot afford Ultimate, just Professional, so I admit I have no hands on experience with the tool. What JetBrains product is comparable?
- CmonDev 13y ago
  There is no JetBrains product like that, I just wish they had, because they have a potential to build something lightweight. Best use cases for me personally are the visual dependency graph and method call diagrams when dealing with poor-quality code (huge classes with a lot of method dependencies). I would pay a hundred bucks for this, but not a thousand dollars.
dman 13y ago
If youre dependent on auto generated diagrams to understand the structure of your code you have bigger problems.
- CmonDev 13y ago

lcedp 13y ago

The future is what? Nothing's been proposed so far.

smoyer 13y ago

In the '80s I was working on a (never released) system I called FlowPro, but EGA adapters and 8088 processors weren't really capable of the kind of DnD I wanted. The idea was that flow-charts would be turned into code, with some meta data added if you wanted specific variable names etc. (instead of the words used in the diagram's box). Then in the early-to-mid '90s, I came across a product called Prograph that ran on both Windows (I did a lot of programming for WFW with it) and Mac (OS 8 I believe). I loved that system, and realized it solved some of the syntax issues I had using flow-charts, but it still never really took off. The concept lives on with the Marten IDE though I'm not sure how many people are actively using it. https://en.wikipedia.org/wiki/Prograph https://en.wikipedia.org/wiki/Prograph http://andescotia.com/products/marten/ http://andescotia.com/products/marten/

collyw 13y ago

Users will use Excel. That is the closest thing we have to graphical programming that works. They don't need to understand data structures, just columns and rows. Anything more than that, and you need to understand new concepts. By the time you understand these, drag and drop doesn't gain you much over typing.

pizo 13y ago

The idea of graphical programming seems in a lot of way like graphical mathematics. Works in some cases, but not in all. I used graphics to represent databases, and their relationships, because it's more easily understood than several hundred lines of code. It fails to tell the whole story though, and thus only serves as a thousand foot view of what is going on.

gshubert17 13y ago
> Works in some cases, but not in all. Delphi (for Windows, 1995-on) had a very nice graphical IDE which was good for developing graphical user interfaces in an object-oriented programming style. The rest of the IDE was fine for editing and debugging. But it was great for GUI development.
pcmonk 13y ago
You're right about the similarity with graphical mathematics (and Bret Victor's Kill Math project was something of an inspiration for this (even though I have some fundamental disagreements with his methods), even though I didn't see it until after I'd started it). You've identified a critical problem with graphical programming today: it fails to tell the whole story. This is exactly what I referred to in the post, and that's not inherent in the medium; rather, it's just that people have been trying to solve this problem in the wrong way.

svantana 13y ago

Personally, I would prefer an IDE/debugger that allowed me to visualize my data really well. This ought to be a lot easier also - I've done some simple scripting with GnuPlot which has given me a lot of mileage. But imagine being able to put a breakpoint anywhere in your program and interactively browse your trees, dictionaries and arrays using OpenGL-accelerated real-time graphics. I don't understand why xCode and VS aren't putting more efforts into things like this.

jamii 13y ago
Sounds like this is what you want - https://github.com/Element84/vdd-core/wiki/What-is-Visualization-Driven-Development%3F https://github.com/Element84/vdd-core/wiki/What-is-Visualiza... EDIT: Or even http://www.youtube.com/watch?v=L6iUm_Cqx2s http://www.youtube.com/watch?v=L6iUm_Cqx2s
goetz 13y ago
A friend of mine is working precisely on this : a plugin for XCode for "visual debugging". What would you like to see in such a tool?
- svantana 13y ago
  Sorry for the late reply. I'd love to be able to plot arrays of floats or ints as 2d or 3d graphs. If you've ever used Matlab, this is the sort of thing it excels at.
CmonDev 13y ago
Well actually Visual Studio team put efforts to address exactly this need: http://visualstudiogallery.msdn.microsoft.com/eedc48e7-5169-40d3-b602-ad451990a420 http://visualstudiogallery.msdn.microsoft.com/eedc48e7-5169-... http://blogs.msdn.com/b/vcblog/archive/2012/07/12/10329460.aspx http://blogs.msdn.com/b/vcblog/archive/2012/07/12/10329460.a... http://msdn.microsoft.com/en-us/library/zayyhzts.aspx http://msdn.microsoft.com/en-us/library/zayyhzts.aspx http://visualstudiogallery.msdn.microsoft.com/c7e02633-86d9-4262-b745-6cc647afb3a8 http://visualstudiogallery.msdn.microsoft.com/c7e02633-86d9-...

outworlder 13y ago

I can postulate that there is absolutely no way a purely graphical representation of code will be the future. The same way we do not talk using pictures. A graphical language would be best used to display structure. Bonus points if it allowed changing. The rest, modules if you will, can be better described by text.

CmonDev 13y ago
Don't want to argue with the spirit of your post, but Chinese hieroglyphics is derived from pictures and it's probably somewhat representative of their language.

drstrangevibes 13y ago

a wise man once said: "when evaluating a language examine its primitives, its means of combination and its means of abstraction". By these metrics I find graphics woefully inferior!

erock 13y ago

I've spent the past year "working" within graphical programming, and I have to say it is a slow form of torture. I have been "writing" IVR applications for a call center. The interface is far from intuitive, and it is jsut too easy to miss what is actually going on. The loops and if statements are all there, but you have to click through on the item to actually see what is going on. Forget trying to find an easy way to locate where something is. at least in my case (VXML) I was forced to use grep to at least give me an idea of where to look, then spend 10x the amount of time it would take in plain ol code to find where to make the change. ugh. give me ASCII code any day

brudgers 13y ago

[Terms of Service: Let me start by saying that I admire the author's passion and effort to create something wonderful, and that this is not a crap on their project, but rather a reaction to their essay. And the fact that their essay raised a reaction to the point where I wrote something about it is, in this case, a testament to its quality.] Lisp programs can be represented as trees because they are trees. They can also be represented as lists, and this is because they are also lists. The important difference between these representations is that trees the primary representation by which computers unravel the intent of a Lisp program and lists are the primary representation by which programmers express intent in their programs. Sure some people sometimes picture their lisp program as a tree, but usually when a programmer pictures something as a tree, it is when viewing it as a data structure. Picturing a Lisp program as a tree is easier than picturing the tree represented by code in most other languages because of Lisps simple parsing process, but the way in which a programming language is parsed is an arbitrary feature of the language [though one worth design consideration]. Although it is easy to draw a tree. Often in Lisp and other languages, the verbal diagram "a list of lists of ..." is adequate to represent a tree along with a description of its interesting properties. Again, while the interesting properties can be represented graphically, what makes them interesting are the maths underpinning their structure not the aesthetics of their spatial representation. "The purpose of this project is to assist the coder in developing more complex code faster and easier" This is orthogonal to Rich Hickey's goal of making it easier and faster to produce simple code. http://www.infoq.com/presentations/Simple-Made-Easy http://www.infoq.com/presentations/Simple-Made-Easy We already have a great tool which allows graphic methods to be used easily to create complex code. It is called Excel.

pcmonk 13y ago
Hm, interesting. I've always pictured my Lisp programs as trees. I think it's a more useful representation, but I see what you mean. You're right that this is orthogonal to Rich Hickey's goal. That's because I think his goal is great for writing simple programs. His goal is to make more programs simple, which is great. My goal is to make it easier to write complex code. As Fred Brooks talks about in No Silver Bullet, there are some problems that are accidental and some that are essential. Hickey is trying to solve the accidental problemms, and I'm trying to make it easier to solve the essential problems. Some logic is just inherently complex -- I want to create tools to make it easier to reason about. And Excel is not nearly programmable enough to be better than a textual interface to solve the same problems. It's benefit is that it can be used (relatively) easily by non-programmers.

bane 13y ago

http://en.wikipedia.org/wiki/DRAKON http://en.wikipedia.org/wiki/DRAKON

Zigurd 13y ago

There are some general problems with the way people think about graphical programming. First, they should stop thinking about it as executable flowcharts and take a step back: There must be a better way than typing characters and pseudo-words to create programs. But that way isn't necessarily any of the graphical programming systems anyone has yet come up with. This article is a good example of thinking that's not even near the walls of the box, much less outside. And what's that crazy digression about wysiwyg editors? And then the author drops this one: "Thus, this isn’t really about graphical programming." Derrrrrr. We are barely at a point where we admit that evaluating a language syntax in a vacuum is silly, so people are still surprised that "verbose" Java is popular, despite the advantages static types and an intermediate language confer on the editor, compiler, static analyzer, debugger, etc. A less than holistic approach to graphical languages is probably doomed.

taeric 13y ago

Again I look at how our industry completely ignores every other industry out there. Some things do lend themselves to graphical viewing. Some things do not. Many things lend themselves to a series of graphical representations. The vast majority of things benefit from both graphical representations and textual elaborations. One of the things I wish programming tools offered, was a way to "zoom in" details wise on a solution. More than just offering an outline, I want there to be some sort of representation of how things went from conception to realization. Consider, the story boards for movies/games. Those story boards are lost in the final product. Worse, when you are looking at a piece of code, you don't really see the "why" of what it was made for. Sure, you get some of this with descriptive names, but that really doesn't help with what I'm wanting to say here. An example, that I really want to explore some more is graphviz. It can go from text to graphics. This process is fairly static, though. If it doesn't turn out how you want it, you have to edit the text and fire it again. I want something that can possibly go from text, to initial graphics where I manipulate it some more, to final representation all in a repeatable way. (I think this is possible, but I am not familiar with the toolchain.) More, there is no reason I shouldn't be able to start from graphics, go to a textual details portion, at varying levels of detail, make edits, and view what effect they have in the viewer. Imagine, something similar to developer tools in chrome/firefox, where in addition to being able to look over at the boxmodel for details, you can physically move elements around. Of course, I'm a bit of a heretic. I would kind of like to be able to export absolute positioned style sheets for forms designed this way. :) Still not sure on all of the reasons that would be terrible.

MetaCosm 13y ago
... wanting Visual Basic doesn't make you a heretic, it just makes you aching for a foregone era a bit. Visual coding has been pushed for ... at least 30 years. The thing you are looking for is all over the place, in various IDEs, in QT MOC, in literally DOZENS of VPL (Visual Programming Languages) and even to a lesser degree in UML (Unified Modeling Language). Since we have QT MOC in the mix, we could even through in ALL the meta compilation tools, which adds in hundreds of others. What you want isn't so much unique, as already done and discarded. You end up with tightly bound IDEs and code, you end up with hard to manage binary nonsense, you end up with ridiculously hard peer review and change tracking. As for tracking the "why" -- use any method you want -- my team uses -- story boards! They work perfect for software.

kyllo 13y ago

Good sell, I agree with all the points the author makes, but as Linus says, "Talk is cheap, show me the code."

pcmonk 13y ago
I'm working on it...
- kyllo 13y ago
  Thanks, it sounds like a cool project. Please do post it here when you have an alpha version, I'd be interested to see it.

seanmcdirmid 13y ago

Was the article flagged?

vinceguidry 13y ago

I find myself looking for graphical interfaces to things I already know fairly well but don't really care to drop into shell every time I need to use it. I have an SSH tunnel manager and an rsync manager. I use Sequel Pro instead of the mysql command line. In fact, I really only find myself using the command line for, you know, running commands. If I find myself there a lot, I'll check myself and start to think, "can I start putting all these things I'm typing into a Rake task or shell script? After all, the more I'm there, the more chance for user error. I think the only major task I still use a terminal for is looking at log files. I intend to eliminate that soon enough when I can 12-factorify the app I'm responsible for and find a decent graphical log viewer.

chipsy 13y ago

I don't see graphics as a silver bullet, but I do think they're underused or misused. Graphics provide a wider variety of feedback; they let you see graph structures and summaries of data, and allow connections to be spatially represented. All of which means that they excel at connecting the big chunks of the system that build on top of some model of service provision and protocol definition, but poor at engaging in the "direct dialogue" of imperative or applicative computation. The part that makes people say "it never works" is that we're almost always thinking from an inverted model, where we know how things scale up and down with text, and apply that learned experience to graphical systems. Subsequently, our engineering instinct leads us to put a similar degree of granular agency into graphical coding and allow the standard constructs (looping, branching, subroutines) to appear in it, which massively increases the complexity of the system and brings us away from problem domain thinking. My recommendation now is - figure out a data structure that could be interesting to visually edit. Make a good editor for that. Allow imperative code a runtime model for accessing it. Allow imperative code to be called from it. Then it stands at least some chance of not being a bad wheel reinvention.

pcmonk 13y ago
I understand what you're saying. My hypothesis is that (some version of) Lisp itself, being a tree, could be interesting to visually edit. All my code, by the way, can be directly accessed from Racket (and vice-versa), so it's definitely not a requirement that the whole program be written graphically. I want to make it a smooth transition from textual to graphical programming.