12 ms·

An editor for composed programs

91 points by ltratt 12y ago

Guvante 12y ago
Interesting idea, and any research into getting away from text editors is positive. However in order to compete with text, which has a bountiful number of editors available. You need to beat the programmers favorite editor at writing the program. Doing this given the competition size is incredibly hard. While I think the has merit at a high level, poor implementation isn't the biggest thing holding back non-text implementations. Being the best editor for everyone has been. On a related note, is there a reason they didn't go with some form of escaping? Similar to CDATA in XML.
- jamii 12y ago
  It is effectively just a form of magic bracket that never needs to be escaped.
  - mtdewcmu 12y ago
    XML is about as good a format as any could be for this kind of application. XML, after all, stands for eXtensible Markup Language, and brackets that provide side-channel data on programming language are nothing if not markup. It would be a mistake to abandon text as a storage format when suitable formats are not only popular but ubiquitous.
- rtpg 12y ago
  The issue with escaping is that the inner language might have escape tokens of a similar form. Though if they're mechanically inserted, I'm sure _some_ sequence of characters will be obscure enough to be confident of lack of conflict (except maybe for APL). Something like $@#/></!* might work.
  - sparkie 12y ago
    The issue isn't just which characters are obscure enough to use - it's that they must be unique for each combination of languages and we could have potentially infinite combinations. Using the CDATA example, ]>> is pretty rare and we're unlikely to see it used in other languages - the problem is it used in one language - XML itself. If you happen to have some XML data embedded in a CDATA, and that XML also has CDATA of its own, you've broken the parser, because it will match the inner ]>> as the closing tag for the outer CDATA. To solve this, we'd either need to modify the parser for the inner XML to use something other than ]>> for it's CDATA, or we'd need a parser which ensures these escaping delimiters are evenly balanced - and our regular parsing algorithms are unsuitable, so we'd need SDE. Of interest are languages like Nemerle, which allow you to define additional syntax for the language through macro systems - however, there are dedicated delimiters which can be used to enclose sequences to ensure unambiguity. (Nemerle uses a PEG based parser). Another interesting approach is taken by Wyvern, which was just posted recently - which uses different indentation levels to disambiguate different languages in the same text file. What's interesting about Tratt and Diekmann's model (Language Boxes), is they do not specify a storage format - although they use a tree based format in the implementation of eco, one could in theory, spit out plain text in languages like Nemerle and Wyvern - where the editor can do the job of selecting the right escape delimiters for each embedded language - and the result is just a plain text file which can be accepted by the normal compilers, but that's still open for research.
    - Guvante 12y ago
      You could try supporting a subset of languages to fit in your requirements. For instance disallowing tab characters and using them for control. Additionally problem character replacement isn't so bad if it is machine controlled, since it can be more complex and making it human editable isn't a problem since your current format isn't human editable.
- mtdewcmu 12y ago
  >> any research into getting away from text editors is positive. Any research at all? I'm curious: what it is that's unworkable about text in your view?
  - Guvante 12y ago
    > Any research at all? Other people spending money to figure out if an alternative idea will work is good for me. > what it is that's unworkable about text in your view? Nothing, it just feels like a local maximum. Typing is incredibly efficient, so I don't think it is going anywhere. There are plenty of ways of showing that in a certain situation non-text can be more efficient, so I believe there is a way to introduce non-text that can make the general case more efficient.
kazinator 12y ago
> "In a sense, when programming in a text editor I think in trees; mentally flatten that tree structure into sequences of ASCII characters that I can type into the text editor; and then have the parser recreate the tree structure I had in my head when I wrote the program." This is quite an amazing article in that the author really "gets" Lisp and yet at the same time doesn't get it. (It isn't mentioned; don't waste time looking.) If you admit you're already thinking in terms of that tree structure already, just freakin' use it instead of this charade of encoding into in some completely rearranged way that has to be unraveled back to what you were originally thinking of anyway (or something else).
- jamii 12y ago
  The same thing is true for lisp. Writing a tree-based interaction mode like paredit that doesn't die horribly on malformed text is a non-trivial problem (http://mumble.net/~campbell/emacs/paredit.el http://mumble.net/~campbell/emacs/paredit.el - 2625 lines of emacs lisp just to edit a tree).
  - taeric 12y ago
    I think the point is that there is only the tree structure in lisp. Even though you can embed things with the reader macro, most of the time this is frowned upon specifically because it hides the actual structure. There are some difficulties in this, of course. Sometimes I don't think in terms of trees of information. This is relatively rare, though.
  - abecedarius 12y ago
    The editor in http://en.wikipedia.org/wiki/Interlisp http://en.wikipedia.org/wiki/Interlisp did work directly on list structure instead of on top of a character-string editor. (I never studied it carefully since at least at first it worked through a tty interface and that's pretty inherently a pain, like Unix ed.) I think I read somewhere that Deutsch's PDP-1 Lisp pioneered that editing model, though I may be completely misremembering.
    - agumonkey 12y ago
pfraze 12y ago
I will almost certainly be giving this a try tonight.
tikhonj 12y ago
Emacs's structured editing, in the style of paredit[1], sounds very similar to a text-based version of syntax-directed editing (SDE). Designed for Lisp and S-expressions, it's a system that lets you edit and navigate around nested expressions at a syntactic rather than textual language. As you type, the correct level of nesting is maintained and there are commands for moving and editing code in terms of s-expressions rather than characters. In practice, it's very easy to start using and very powerful... but requires using Lisp. It'll seem odd if you're not regularly a Lisp user. Happily, this idea can generalize well enough to distinctly non-Lisp languages. Structured Haskell mode[2] does this for Haskell, which has a less regular and sparser syntax than s-expressions. The documentation has a bunch of handy animations showing the various features, giving you an idea of how it works without having to install it yourself. This doesn't address the meat of the blog post—composing languages and grammars by taking advantage of structured editing—but it shows that the necessary foundation is possible with a perfectly normal text editor. It would be interesting to see if some of the ideas from Eco could be added to structured editing modes like this without requiring you to use an external, self-contained editor. [1]: http://www.emacswiki.org/emacs/ParEdit http://www.emacswiki.org/emacs/ParEdit [2]: https://github.com/chrisdone/structured-haskell-mode https://github.com/chrisdone/structured-haskell-mode Also, here's a great video of Paredit in action: https://www.youtube.com/watch?v=D6h5dFyyUX0 https://www.youtube.com/watch?v=D6h5dFyyUX0
- kazinator 12y ago
  ... and of course, in Lisp you can embed other languages in terms of their syntax trees, so the composed language problem goes away. Your HTML just looks like (html (head ...) (body (h1 ...))) or whatever; no "language box for HTML" needed.
  - mtdewcmu 12y ago
    >> Your HTML just looks like (html (head ...) (body (h1 ...))) or whatever; no "language box for HTML" needed. In your example, the language composition is smooth because there's only one language there: Lisp. HTML has capitulated.
al2o3cr 12y ago
Interesting idea, but the code example used completely puts me off - it's literally the opposite of well-factored code, with HTML, Python and SQL all smeared together.
- deleted 12y ago
  [deleted]
jimbobimbo 12y ago
Here's how Visual Studio 2013 handles HTML, C# and JavaScript syntax coloring in one Razor view: http://i.imgur.com/he1ieG9.png http://i.imgur.com/he1ieG9.png
rch 12y ago
nice.
tephra 12y ago
Here are the slides [1] from ECOOP 2014 presenting his research. It was a really interesting talk. He also did a demo which was cool. [1] http://ecoop14.it.uu.se/programme/ecoop_ss_tratt.pdf http://ecoop14.it.uu.se/programme/ecoop_ss_tratt.pdf
- sparkie 12y ago
  Additionally, they published this paper last year with an early demo of the editor linked on Youtube. I've posted it on here a few times but it hasn't had much response. http://soft-dev.org/pubs/pdf/diekmann_tratt__parsing_composed_grammars_with_language_boxes.pdf http://soft-dev.org/pubs/pdf/diekmann_tratt__parsing_compose... https://www.youtube.com/watch?v=LMzrTb22Ot8 https://www.youtube.com/watch?v=LMzrTb22Ot8
npatrick04 12y ago
I've been using Org Mode with emacs for writing literate programs. You could write a simple front to Org Mode to decrease the clutter...but the hierarchical nature of the Org Mode files tends to reduce the issue of clutter already, so it may not be worthwhile. http://orgmode.org/manual/Working-With-Source-Code.html#Working-With-Source-Code http://orgmode.org/manual/Working-With-Source-Code.html#Work...

91 points by ltratt 12y ago

Interesting idea, and any research into getting away from text editors is positive. However in order to compete with text, which has a bountiful number of editors available. You need to beat the programmers favorite editor at writing the program. Doing this given the competition size is incredibly hard. While I think the has merit at a high level, poor implementation isn't the biggest thing holding back non-text implementations. Being the best editor for everyone has been. On a related note, is there a reason they didn't go with some form of escaping? Similar to CDATA in XML.

jamii 12y ago
It is effectively just a form of magic bracket that never needs to be escaped.
- mtdewcmu 12y ago
  XML is about as good a format as any could be for this kind of application. XML, after all, stands for eXtensible Markup Language, and brackets that provide side-channel data on programming language are nothing if not markup. It would be a mistake to abandon text as a storage format when suitable formats are not only popular but ubiquitous.
rtpg 12y ago
The issue with escaping is that the inner language might have escape tokens of a similar form. Though if they're mechanically inserted, I'm sure _some_ sequence of characters will be obscure enough to be confident of lack of conflict (except maybe for APL). Something like $@#/></!* might work.
- sparkie 12y ago
  The issue isn't just which characters are obscure enough to use - it's that they must be unique for each combination of languages and we could have potentially infinite combinations. Using the CDATA example, ]>> is pretty rare and we're unlikely to see it used in other languages - the problem is it used in one language - XML itself. If you happen to have some XML data embedded in a CDATA, and that XML also has CDATA of its own, you've broken the parser, because it will match the inner ]>> as the closing tag for the outer CDATA. To solve this, we'd either need to modify the parser for the inner XML to use something other than ]>> for it's CDATA, or we'd need a parser which ensures these escaping delimiters are evenly balanced - and our regular parsing algorithms are unsuitable, so we'd need SDE. Of interest are languages like Nemerle, which allow you to define additional syntax for the language through macro systems - however, there are dedicated delimiters which can be used to enclose sequences to ensure unambiguity. (Nemerle uses a PEG based parser). Another interesting approach is taken by Wyvern, which was just posted recently - which uses different indentation levels to disambiguate different languages in the same text file. What's interesting about Tratt and Diekmann's model (Language Boxes), is they do not specify a storage format - although they use a tree based format in the implementation of eco, one could in theory, spit out plain text in languages like Nemerle and Wyvern - where the editor can do the job of selecting the right escape delimiters for each embedded language - and the result is just a plain text file which can be accepted by the normal compilers, but that's still open for research.
  - Guvante 12y ago
    You could try supporting a subset of languages to fit in your requirements. For instance disallowing tab characters and using them for control. Additionally problem character replacement isn't so bad if it is machine controlled, since it can be more complex and making it human editable isn't a problem since your current format isn't human editable.
mtdewcmu 12y ago
>> any research into getting away from text editors is positive. Any research at all? I'm curious: what it is that's unworkable about text in your view?
- Guvante 12y ago
  > Any research at all? Other people spending money to figure out if an alternative idea will work is good for me. > what it is that's unworkable about text in your view? Nothing, it just feels like a local maximum. Typing is incredibly efficient, so I don't think it is going anywhere. There are plenty of ways of showing that in a certain situation non-text can be more efficient, so I believe there is a way to introduce non-text that can make the general case more efficient.

kazinator 12y ago

> "In a sense, when programming in a text editor I think in trees; mentally flatten that tree structure into sequences of ASCII characters that I can type into the text editor; and then have the parser recreate the tree structure I had in my head when I wrote the program." This is quite an amazing article in that the author really "gets" Lisp and yet at the same time doesn't get it. (It isn't mentioned; don't waste time looking.) If you admit you're already thinking in terms of that tree structure already, just freakin' use it instead of this charade of encoding into in some completely rearranged way that has to be unraveled back to what you were originally thinking of anyway (or something else).

jamii 12y ago
The same thing is true for lisp. Writing a tree-based interaction mode like paredit that doesn't die horribly on malformed text is a non-trivial problem (http://mumble.net/~campbell/emacs/paredit.el http://mumble.net/~campbell/emacs/paredit.el - 2625 lines of emacs lisp just to edit a tree).
- taeric 12y ago
  I think the point is that there is only the tree structure in lisp. Even though you can embed things with the reader macro, most of the time this is frowned upon specifically because it hides the actual structure. There are some difficulties in this, of course. Sometimes I don't think in terms of trees of information. This is relatively rare, though.
- abecedarius 12y ago
  The editor in http://en.wikipedia.org/wiki/Interlisp http://en.wikipedia.org/wiki/Interlisp did work directly on list structure instead of on top of a character-string editor. (I never studied it carefully since at least at first it worked through a tty interface and that's pretty inherently a pain, like Unix ed.) I think I read somewhere that Deutsch's PDP-1 Lisp pioneered that editing model, though I may be completely misremembering.
  - agumonkey 12y ago

pfraze 12y ago

I will almost certainly be giving this a try tonight.

tikhonj 12y ago

Emacs's structured editing, in the style of paredit[1], sounds very similar to a text-based version of syntax-directed editing (SDE). Designed for Lisp and S-expressions, it's a system that lets you edit and navigate around nested expressions at a syntactic rather than textual language. As you type, the correct level of nesting is maintained and there are commands for moving and editing code in terms of s-expressions rather than characters. In practice, it's very easy to start using and very powerful... but requires using Lisp. It'll seem odd if you're not regularly a Lisp user. Happily, this idea can generalize well enough to distinctly non-Lisp languages. Structured Haskell mode[2] does this for Haskell, which has a less regular and sparser syntax than s-expressions. The documentation has a bunch of handy animations showing the various features, giving you an idea of how it works without having to install it yourself. This doesn't address the meat of the blog post—composing languages and grammars by taking advantage of structured editing—but it shows that the necessary foundation is possible with a perfectly normal text editor. It would be interesting to see if some of the ideas from Eco could be added to structured editing modes like this without requiring you to use an external, self-contained editor. [1]: http://www.emacswiki.org/emacs/ParEdit http://www.emacswiki.org/emacs/ParEdit [2]: https://github.com/chrisdone/structured-haskell-mode https://github.com/chrisdone/structured-haskell-mode Also, here's a great video of Paredit in action: https://www.youtube.com/watch?v=D6h5dFyyUX0 https://www.youtube.com/watch?v=D6h5dFyyUX0

kazinator 12y ago
... and of course, in Lisp you can embed other languages in terms of their syntax trees, so the composed language problem goes away. Your HTML just looks like (html (head ...) (body (h1 ...))) or whatever; no "language box for HTML" needed.
- mtdewcmu 12y ago
  >> Your HTML just looks like (html (head ...) (body (h1 ...))) or whatever; no "language box for HTML" needed. In your example, the language composition is smooth because there's only one language there: Lisp. HTML has capitulated.

al2o3cr 12y ago

Interesting idea, but the code example used completely puts me off - it's literally the opposite of well-factored code, with HTML, Python and SQL all smeared together.

deleted 12y ago
[deleted]

jimbobimbo 12y ago

Here's how Visual Studio 2013 handles HTML, C# and JavaScript syntax coloring in one Razor view: http://i.imgur.com/he1ieG9.png http://i.imgur.com/he1ieG9.png

rch 12y ago

nice.

tephra 12y ago

Here are the slides [1] from ECOOP 2014 presenting his research. It was a really interesting talk. He also did a demo which was cool. [1] http://ecoop14.it.uu.se/programme/ecoop_ss_tratt.pdf http://ecoop14.it.uu.se/programme/ecoop_ss_tratt.pdf

sparkie 12y ago
Additionally, they published this paper last year with an early demo of the editor linked on Youtube. I've posted it on here a few times but it hasn't had much response. http://soft-dev.org/pubs/pdf/diekmann_tratt__parsing_composed_grammars_with_language_boxes.pdf http://soft-dev.org/pubs/pdf/diekmann_tratt__parsing_compose... https://www.youtube.com/watch?v=LMzrTb22Ot8 https://www.youtube.com/watch?v=LMzrTb22Ot8

npatrick04 12y ago

I've been using Org Mode with emacs for writing literate programs. You could write a simple front to Org Mode to decrease the clutter...but the hierarchical nature of the Org Mode files tends to reduce the issue of clutter already, so it may not be worthwhile. http://orgmode.org/manual/Working-With-Source-Code.html#Working-With-Source-Code http://orgmode.org/manual/Working-With-Source-Code.html#Work...