5 ms·

The GNU extension language

48 points by fogus 15y ago

dkersten 15y ago
This quote from the comments summarizes my thoughts on this: "Robert says: 30 August 2011 11:55 PM It isn't whether Guile is good, bad, usable, or not usable. It is whether people will and do USE it. I have not seen that as the case for Guile or Scheme in general." I don't know anybody anywhere who has even tried to use Guile. I don't even know if anybody I know have ever looked at it. I don't know how many people even know it exists. I personally love Scheme as a language, but I honestly can't see myself ever investing enough time into Guile to be able to write program extensions in it. I don't see my friends (some of whom I believe would write extensions to popular open source software if they could do so in a language that is sufficiently similar to one they already know - I certainly would be a lot more likely to do so and I'm not even Lisp averse) doing so either. "First of all, what does it mean to have a GNU system based on a non-GNU technology? What would be left of GNU?" So pick a non-GNU language and create a GNU implementation. Personally, I think the best choice would be Lua. Its mature, a lot of people already use it as an extension language and its very similar to a lot of popular languages, so resistance for people to learn it is low (it is reasonably similar to Javascript and to a lesser extent, Ruby and Python; also, as has been proven by the game industry, it is easy to learn for C, C++ and Java programmers. Finally, while not GNU, it has some exceptionally good implementations).
- wollw 15y ago
  While it's true that Guile isn't exactly pervasive as an extension language I can name two projects off the top of my head that do use it as their scripting language: LilyPond[1] and gEDA[2]. I haven't actually done anything with that capability but Guile is in use as an extension language to some degree. [1] http://lilypond.org/doc/v2.14/Documentation/extending/scheme-tutorial http://lilypond.org/doc/v2.14/Documentation/extending/scheme... [2] http://www.geda.seul.org/wiki/geda:hse_howto http://www.geda.seul.org/wiki/geda:hse_howto
  - dkersten 15y ago
    Oh, I'm not arguing that its not in use - I'm arguing from the point of view of the users of the software itself (not its authors) who may want to write extensions. I feel users are unlikely to jump through hoops to write extensions to software, yet I feel that Guile creates hoops for a number of reasons, including (but certainly not limited to) an aversion to lisp (i have personally observed this), unfamiliarity compared to the users every day language, the difficulty of a functional-focused vs imperative focused language for quickly hacking an extension together and ease of obtaining documentation.
    - wollw 15y ago
      I think we agree then. The (to most people) unfamiliar syntax and programming paradigm are certainly barriers to entry. It's not what people expect and I doubt most people writing plugins for programs want to do so in unfamiliar territory.
      - dkersten 15y ago
        It's not what people expect and I doubt most people writing plugins for programs want to do so in unfamiliar territory. This summarizes it nicely.
- tikhonj 15y ago
  Well, I've never used Guile, but I have used Elisp, to great success. It allowed me--with little previous Elisp experience--to easily write a minor and a major mode without having to spend too much time in the manual. Given that Guile is very similar--just more modern and polished--to Elisp, I definitely see it as a reasonable extension language. I think the issue with the limited number of extensions written in Guile is more with the projects that can be extended than with the language--I'm not aware of any that I use, and certainly none that I would like to extend.
  - dkersten 15y ago
    I get the impression that the majority of emacs users are not averse to learning seemingly weird and dificult new things (otherwise you probably would not have learned to use emacs in the first place). The majority of users are, in my experience anyway, put off by all sorts of things (and often quite unrelated to the merits (or lack of) the language itself). I have also experienced quite a bit of lisp hostility. All of these things add up for me to feel there are better choices as an extension language if you want the average[1] user to write extensions. [1] by average I mean that they have the technical skill to do so, but aren't terribly invested in learning lots of new things. I imagine (I'm a vim user myself and I find its true for me) that an emacs user has already invested in the tool enough that learning a little more to make it even more useful is not a big deal.
    - tikhonj 15y ago
      I can't help thinking that your average user is much less likely to write an extension at all. Also, I suspect a disproportionate number of GNU program users sufficiently invested in them to write extensions are either Emacs users already (it is Gnu Emacs, after all) or at least fairly familiar with Lisp. The sort of person who doesn't go out of his way to learn a new tool is also the sort of person who probably doesn't go out of his way to extend an existing tool either. I think targeting advanced users for writing extensions is completely reasonable. Plenty of "average" (by your definition) programmers who use Eclipse know Java well enough to write Eclispe plugins. How many of them do? I suspect a large portion of those who do would have written the plugins in Guile as well.
- LeafStorm 15y ago
  I think that one big advantage of Lua (also Python and Ruby) over Guile for extensions is that the techniques you use in Lua programming are a lot more generally applicable to writing one's own scripts and programs in the future than Guile's - not just in Lua, but also in Ruby, Python, JavaScript, Java, C#, C, and so on.
  - dkersten 15y ago
    Exactly. Its more like what most people use already, so the barrier to entry is lower.
- microtherion 15y ago
  I agree. The language advocacy in the linked article is fine as far as that goes, but 17 years after the Tcl wars, it should be possible to evaluate an extension language strategy by whether it WORKS. Sure, guile is powerful and it COULD be used to build all sorts of powerful extensions, but IS it actually used that way? I would argue that overall[1], guile has had pretty much no meaningful adoption in GNU projects. Emacs is (of course) still using its own flavor of lisp. gdb, which badly needs a powerful scripting language (and which I think was scheduled to have Tcl support when the wars started), still has no good scripting integration. I cannot think of any other major GNU project using guile. [1] There are certainly some exceptions. Lilypond uses guile extensively and successfully.
jwatte 15y ago
Quote: " Scheme's macros have no comparison in any other language." I argue that Erlang parse transforms are at least as powerful. Interesting that he mentions Erlang, but doesn't mention that! (Erlang probably isn't ideal as an /extension/ language for other reasons, though)
- zaphar 15y ago
  erlangs parse transforms are indeed just as powerful but no where near as easy to use. They allow you to modify your source file in pretty much the same way but the ast you are modifying is way more complicated than lisps and much more difficult to properly modify. If Lisps macros are difficult for the average user to learn then Erlangs Parse Transforms are even more so and a much larger barrier to usage than lisps. It's rare for an erlang developer to reach for a Parse Transform it's rare for a lisp developer not to reach for one.
mindslight 15y ago
Although treasonous to say, I don't think a sexp-based language is appropriate for a wide community of programmers. (I'm currently enjoying coming back to PLT Racket from Clojure, so please don't write me off as someone who can't get over the initial shock.) Macros make it too easy to define new syntax, to the point where things that should be core language features are neglected, only to be continually re-half-implemented, etc. Take a look at what passes for 'good enough' for destructuring in Clojure, both the older let-based and the newer core.match (which still lacks the extensibility of Racket's match). The deficiencies can always be fixed 'later', right? No syntactic difference between function application and special forms means that one requires an editor that can somehow differentiate (slime/drracket approaches) to even have a shot at quickly scanning unfamiliar code (say, quickly looking at what an implementation actually does). It's pointed out that Guile will handle other languages (imho this is tacitly admitting uncertainty about Scheme). But multiple languages only stand to make the code-reading problem even worse! Never mind that Scheme will be the only first-class language (other languages can't consume macros) so alternatives are at best advertisements to get people hooked and having to learn the 'real language'. The little bit of Ruby I've written opened my eyes to how much you can do with just first-class code blocks, and made me understand why something like SLIB would opt for explicit lambdas everywhere instead of macros. Macros are so enticing to cut down on boilerplate, but they're visually indistinguishable, and without plenty of forethought they're not composable. Clearly many skin syntaxes for Lisps have been attempted and never caught on. I don't currently have much to add on that front besides a suggestion of not being afraid of well thought out concrete syntax (how many base special forms are there, really?). But really, I think my whole post shows the main problem with Lisp, at least for me. It's so easy to do things and it's such a blank slate that no matter what I sit down to do, I always end up trying to create a new language.
- jrockway 15y ago
  Although treasonous to say, I don't think a sexp-based language will ever be adopted by a wide community of programmers. The problem with this argument is that a sexp-based configuration language is already adopted by a community of programmers: Emacs Lisp. Emacs itself ships with over a million lines of Lisp code, and there are thousands of extensions available from around the Internet. The problems you worry about with respect to macros have simply not happened; most people name macros the same way (starting with def or define-), and most macros are written "cleanly" in such a way as to not surprise the programmer. (I'll be honest: I can't think of one time I've ever hit a but in a Lisp program caused by a strange macro.) Emacs Lisp shows that if you give someone a turing-complete programming language on top of a powerful piece of software with a clean and well-documented API, programmers will ignore the warts and enhance the software in ways you never thought possible. Guile is not a perfect programming langauge, and I would never use it by choice. But if I started using some piece of software that used it for configuration, I know I would love it because it would stay out of the way and let me make things. Once you are making cool things, you don't care about the syntax or programming language features at your disposal: you just make it work. Some other examples of this include JavaScript for Firefox extensions and Vimscript for Vpim extensions. JavaScript is very ... minimal compared to other languages like Perl, Python, Ruby, Haskell, and so on, but people write great browser extensions anyway. Vimscript is the weirdest thing I've ever seen but Vim is able to do a lot of cool stuff despite this. The only thing a successful extension language needs is something worth extending. If you use Guile as your extension language, you can concentrate doing that instead of writing your own programming langauge toolchain.
gruseom 15y ago
When it comes to extensibility, Emacs and Emacs Lisp have one of the most compelling stories of all time. (In fact, is there anything that beats them?) If Andy is right that Guile can do for GNU what Elisp did for Emacs, only better, it would be downright amazing. I wonder, though, whether some things that made Emacs/Elisp so successful would get lost in the switchover. For example, while dynamic scoping is bad for writing robust complex software, it does make Emacs easier to extend. Emacs is not really an application or a platform, it is a framework: a framework for a text editor, one that comes with a well-thought-through model and a flagship application (Emacs itself) that doubles as a development environment. I don't mean "model" as in MVC, but in the sense of a consistent set of concepts that can be composed to do things. In Emacs those concepts are things like buffers and windows and text properties. Emacs has one of the best models I've ever seen; it should be taught as an example of how to do this kind of design. Emacs on the surface may be clunky and unlovable, but dive underneath and its world is beautiful. When people extend Emacs, they're building programs out of that model using tools designed for building programs out of the model (which themselves were built out of the model). If you shift from Emacs to GNU as a whole, you're moving from that single -- self-consistent and composable, not to mention interactive -- design context into a hugely diverse environment with no common design context at all. If Emacs is a planet, GNU is a universe. It's not clear how well Emacs' extensibility magic can be reproduced at that level. If the benefits come mostly from the language, Andy may be right. I hope he is. But if the benefits come mostly from Emacs' framework and interactive environment (both of which are inseparably bound up with the language), then the magic is likely to be left behind with them. Still, it's hard to argue against his case for choosing the most powerful possible extension language so as to maximize the extending that people will be able to do with it. To argue against this in favor of familiarity is particularly to miss the point.
- Peaker 15y ago
  How does dynamic scoping make Emacs easier to extend?
  - yoklov 15y ago
    In a system with dynamic scoping, you have access to variables which otherwise you might not. If it were written in scheme, it might be more difficult to know how many buffers are currently open or some other text-editory value than it is in emacs lisp. Interestingly, from what I've seen dynamic scoping hasn't hindered emacs lispers at all. They have some sort of variable naming system that is specific enough that confusion is fairly rare (I've never seen anybody talk about variable naming conflicts, at least). For example, the value of the (chosen randomly out of my .emacs) ``ido-auto-merge-work-directories-length'' variable seems unlikely to be changed accidentally. That being said, Scheme/Guile have dynamic fluids which allow for the same sorts of things to be done. Personally, I feel like these would have to be used heavily in order to for a hypothetical lexically-scoped guile-based emacs to work, but maybe thats just because I feel like dynamic scope makes things easier to extend :p (at least, in this sort of program).

48 points by fogus 15y ago

This quote from the comments summarizes my thoughts on this: "Robert says: 30 August 2011 11:55 PM It isn't whether Guile is good, bad, usable, or not usable. It is whether people will and do USE it. I have not seen that as the case for Guile or Scheme in general." I don't know anybody anywhere who has even tried to use Guile. I don't even know if anybody I know have ever looked at it. I don't know how many people even know it exists. I personally love Scheme as a language, but I honestly can't see myself ever investing enough time into Guile to be able to write program extensions in it. I don't see my friends (some of whom I believe would write extensions to popular open source software if they could do so in a language that is sufficiently similar to one they already know - I certainly would be a lot more likely to do so and I'm not even Lisp averse) doing so either. "First of all, what does it mean to have a GNU system based on a non-GNU technology? What would be left of GNU?" So pick a non-GNU language and create a GNU implementation. Personally, I think the best choice would be Lua. Its mature, a lot of people already use it as an extension language and its very similar to a lot of popular languages, so resistance for people to learn it is low (it is reasonably similar to Javascript and to a lesser extent, Ruby and Python; also, as has been proven by the game industry, it is easy to learn for C, C++ and Java programmers. Finally, while not GNU, it has some exceptionally good implementations).

wollw 15y ago
While it's true that Guile isn't exactly pervasive as an extension language I can name two projects off the top of my head that do use it as their scripting language: LilyPond[1] and gEDA[2]. I haven't actually done anything with that capability but Guile is in use as an extension language to some degree. [1] http://lilypond.org/doc/v2.14/Documentation/extending/scheme-tutorial http://lilypond.org/doc/v2.14/Documentation/extending/scheme... [2] http://www.geda.seul.org/wiki/geda:hse_howto http://www.geda.seul.org/wiki/geda:hse_howto
- dkersten 15y ago
  Oh, I'm not arguing that its not in use - I'm arguing from the point of view of the users of the software itself (not its authors) who may want to write extensions. I feel users are unlikely to jump through hoops to write extensions to software, yet I feel that Guile creates hoops for a number of reasons, including (but certainly not limited to) an aversion to lisp (i have personally observed this), unfamiliarity compared to the users every day language, the difficulty of a functional-focused vs imperative focused language for quickly hacking an extension together and ease of obtaining documentation.
  - wollw 15y ago
    I think we agree then. The (to most people) unfamiliar syntax and programming paradigm are certainly barriers to entry. It's not what people expect and I doubt most people writing plugins for programs want to do so in unfamiliar territory.
    - dkersten 15y ago
      It's not what people expect and I doubt most people writing plugins for programs want to do so in unfamiliar territory. This summarizes it nicely.
tikhonj 15y ago
Well, I've never used Guile, but I have used Elisp, to great success. It allowed me--with little previous Elisp experience--to easily write a minor and a major mode without having to spend too much time in the manual. Given that Guile is very similar--just more modern and polished--to Elisp, I definitely see it as a reasonable extension language. I think the issue with the limited number of extensions written in Guile is more with the projects that can be extended than with the language--I'm not aware of any that I use, and certainly none that I would like to extend.
- dkersten 15y ago
  I get the impression that the majority of emacs users are not averse to learning seemingly weird and dificult new things (otherwise you probably would not have learned to use emacs in the first place). The majority of users are, in my experience anyway, put off by all sorts of things (and often quite unrelated to the merits (or lack of) the language itself). I have also experienced quite a bit of lisp hostility. All of these things add up for me to feel there are better choices as an extension language if you want the average[1] user to write extensions. [1] by average I mean that they have the technical skill to do so, but aren't terribly invested in learning lots of new things. I imagine (I'm a vim user myself and I find its true for me) that an emacs user has already invested in the tool enough that learning a little more to make it even more useful is not a big deal.
  - tikhonj 15y ago
    I can't help thinking that your average user is much less likely to write an extension at all. Also, I suspect a disproportionate number of GNU program users sufficiently invested in them to write extensions are either Emacs users already (it is Gnu Emacs, after all) or at least fairly familiar with Lisp. The sort of person who doesn't go out of his way to learn a new tool is also the sort of person who probably doesn't go out of his way to extend an existing tool either. I think targeting advanced users for writing extensions is completely reasonable. Plenty of "average" (by your definition) programmers who use Eclipse know Java well enough to write Eclispe plugins. How many of them do? I suspect a large portion of those who do would have written the plugins in Guile as well.
LeafStorm 15y ago
I think that one big advantage of Lua (also Python and Ruby) over Guile for extensions is that the techniques you use in Lua programming are a lot more generally applicable to writing one's own scripts and programs in the future than Guile's - not just in Lua, but also in Ruby, Python, JavaScript, Java, C#, C, and so on.
- dkersten 15y ago
  Exactly. Its more like what most people use already, so the barrier to entry is lower.
microtherion 15y ago
I agree. The language advocacy in the linked article is fine as far as that goes, but 17 years after the Tcl wars, it should be possible to evaluate an extension language strategy by whether it WORKS. Sure, guile is powerful and it COULD be used to build all sorts of powerful extensions, but IS it actually used that way? I would argue that overall[1], guile has had pretty much no meaningful adoption in GNU projects. Emacs is (of course) still using its own flavor of lisp. gdb, which badly needs a powerful scripting language (and which I think was scheduled to have Tcl support when the wars started), still has no good scripting integration. I cannot think of any other major GNU project using guile. [1] There are certainly some exceptions. Lilypond uses guile extensively and successfully.

jwatte 15y ago

Quote: " Scheme's macros have no comparison in any other language." I argue that Erlang parse transforms are at least as powerful. Interesting that he mentions Erlang, but doesn't mention that! (Erlang probably isn't ideal as an /extension/ language for other reasons, though)

zaphar 15y ago
erlangs parse transforms are indeed just as powerful but no where near as easy to use. They allow you to modify your source file in pretty much the same way but the ast you are modifying is way more complicated than lisps and much more difficult to properly modify. If Lisps macros are difficult for the average user to learn then Erlangs Parse Transforms are even more so and a much larger barrier to usage than lisps. It's rare for an erlang developer to reach for a Parse Transform it's rare for a lisp developer not to reach for one.

mindslight 15y ago

Although treasonous to say, I don't think a sexp-based language is appropriate for a wide community of programmers. (I'm currently enjoying coming back to PLT Racket from Clojure, so please don't write me off as someone who can't get over the initial shock.) Macros make it too easy to define new syntax, to the point where things that should be core language features are neglected, only to be continually re-half-implemented, etc. Take a look at what passes for 'good enough' for destructuring in Clojure, both the older let-based and the newer core.match (which still lacks the extensibility of Racket's match). The deficiencies can always be fixed 'later', right? No syntactic difference between function application and special forms means that one requires an editor that can somehow differentiate (slime/drracket approaches) to even have a shot at quickly scanning unfamiliar code (say, quickly looking at what an implementation actually does). It's pointed out that Guile will handle other languages (imho this is tacitly admitting uncertainty about Scheme). But multiple languages only stand to make the code-reading problem even worse! Never mind that Scheme will be the only first-class language (other languages can't consume macros) so alternatives are at best advertisements to get people hooked and having to learn the 'real language'. The little bit of Ruby I've written opened my eyes to how much you can do with just first-class code blocks, and made me understand why something like SLIB would opt for explicit lambdas everywhere instead of macros. Macros are so enticing to cut down on boilerplate, but they're visually indistinguishable, and without plenty of forethought they're not composable. Clearly many skin syntaxes for Lisps have been attempted and never caught on. I don't currently have much to add on that front besides a suggestion of not being afraid of well thought out concrete syntax (how many base special forms are there, really?). But really, I think my whole post shows the main problem with Lisp, at least for me. It's so easy to do things and it's such a blank slate that no matter what I sit down to do, I always end up trying to create a new language.

jrockway 15y ago
Although treasonous to say, I don't think a sexp-based language will ever be adopted by a wide community of programmers. The problem with this argument is that a sexp-based configuration language is already adopted by a community of programmers: Emacs Lisp. Emacs itself ships with over a million lines of Lisp code, and there are thousands of extensions available from around the Internet. The problems you worry about with respect to macros have simply not happened; most people name macros the same way (starting with def or define-), and most macros are written "cleanly" in such a way as to not surprise the programmer. (I'll be honest: I can't think of one time I've ever hit a but in a Lisp program caused by a strange macro.) Emacs Lisp shows that if you give someone a turing-complete programming language on top of a powerful piece of software with a clean and well-documented API, programmers will ignore the warts and enhance the software in ways you never thought possible. Guile is not a perfect programming langauge, and I would never use it by choice. But if I started using some piece of software that used it for configuration, I know I would love it because it would stay out of the way and let me make things. Once you are making cool things, you don't care about the syntax or programming language features at your disposal: you just make it work. Some other examples of this include JavaScript for Firefox extensions and Vimscript for Vpim extensions. JavaScript is very ... minimal compared to other languages like Perl, Python, Ruby, Haskell, and so on, but people write great browser extensions anyway. Vimscript is the weirdest thing I've ever seen but Vim is able to do a lot of cool stuff despite this. The only thing a successful extension language needs is something worth extending. If you use Guile as your extension language, you can concentrate doing that instead of writing your own programming langauge toolchain.

gruseom 15y ago

When it comes to extensibility, Emacs and Emacs Lisp have one of the most compelling stories of all time. (In fact, is there anything that beats them?) If Andy is right that Guile can do for GNU what Elisp did for Emacs, only better, it would be downright amazing. I wonder, though, whether some things that made Emacs/Elisp so successful would get lost in the switchover. For example, while dynamic scoping is bad for writing robust complex software, it does make Emacs easier to extend. Emacs is not really an application or a platform, it is a framework: a framework for a text editor, one that comes with a well-thought-through model and a flagship application (Emacs itself) that doubles as a development environment. I don't mean "model" as in MVC, but in the sense of a consistent set of concepts that can be composed to do things. In Emacs those concepts are things like buffers and windows and text properties. Emacs has one of the best models I've ever seen; it should be taught as an example of how to do this kind of design. Emacs on the surface may be clunky and unlovable, but dive underneath and its world is beautiful. When people extend Emacs, they're building programs out of that model using tools designed for building programs out of the model (which themselves were built out of the model). If you shift from Emacs to GNU as a whole, you're moving from that single -- self-consistent and composable, not to mention interactive -- design context into a hugely diverse environment with no common design context at all. If Emacs is a planet, GNU is a universe. It's not clear how well Emacs' extensibility magic can be reproduced at that level. If the benefits come mostly from the language, Andy may be right. I hope he is. But if the benefits come mostly from Emacs' framework and interactive environment (both of which are inseparably bound up with the language), then the magic is likely to be left behind with them. Still, it's hard to argue against his case for choosing the most powerful possible extension language so as to maximize the extending that people will be able to do with it. To argue against this in favor of familiarity is particularly to miss the point.

Peaker 15y ago
How does dynamic scoping make Emacs easier to extend?
- yoklov 15y ago
  In a system with dynamic scoping, you have access to variables which otherwise you might not. If it were written in scheme, it might be more difficult to know how many buffers are currently open or some other text-editory value than it is in emacs lisp. Interestingly, from what I've seen dynamic scoping hasn't hindered emacs lispers at all. They have some sort of variable naming system that is specific enough that confusion is fairly rare (I've never seen anybody talk about variable naming conflicts, at least). For example, the value of the (chosen randomly out of my .emacs) ``ido-auto-merge-work-directories-length'' variable seems unlikely to be changed accidentally. That being said, Scheme/Guile have dynamic fluids which allow for the same sorts of things to be done. Personally, I feel like these would have to be used heavily in order to for a hypothetical lexically-scoped guile-based emacs to work, but maybe thats just because I feel like dynamic scope makes things easier to extend :p (at least, in this sort of program).