8 ms·

Lisp is Abstract Syntax

35 points by mrbbk 12y ago

PaulHoule 12y ago
Well, yeah. I think the popularity of lisp comes down to the hostility the mit ai lab had towards noam chomsky who in the 1960s spoke out about the vietnam war and thus bit the hand that funds cs research. All other computer languages (except for forth) embraced formal grammar concepts but mit rejected them out of spite. If mit had stopped burning the dragon book and embraced algol syntax we might have been spared the horrors of c, c++ and java but no, they just gave us 1 more reason rt 128 is a boulevard ofbroken dreams.
- ScottBurson 12y ago
  Huh. I was around the AI Lab in the late 1970s -- as an undergrad -- and I don't recall hearing anyone talk about Chomsky in the way you imagine. DARPA funding was still flowing rather freely, it seemed to me, well into the mid-1980s. I don't recall anyone suggesting that there would have been yet more DoD money if it hadn't been for Chomsky. And the grad student I did some work for, who was working on natural language parsing, was clearly intimately familiar with Chomsky's ideas on transformational grammar and referred to them throughout his dissertation. So this isn't ringing true for me. I also recall Gerry Sussman speaking quite highly of Algol 60. No, I think you're quite wrong: the reasons for the popularity of Lisp at the AI Lab have to do with its actual advantages over other languages for the kinds of problems they were working on. And there is an example of a language in the MIT tradition with infix syntax: Dylan. Great language, but wasn't in the right place at the right time to catch on.
  - hga 12y ago
    Was Sussman talking about the syntax, or things like lexical scope, which he and Steele introduced to the LISP world with Scheme in the mid-70s? About the only relevant negative thing I remember being said about Chomsky in a period just after your's was Marvin Minsky saying he saved someone from going down a rabbit hole following Chomsky's work. But that was a judgement of merit of his linguistics work, not his politics.
    - ScottBurson 12y ago
      Well, the lexical scope, certainly, but I don't recall him disparaging the syntax. I don't recall any burning of Dragon Books. Nor was Vaughan Pratt tarred and feathered for releasing CGOL, which allowed one to write Lisp in infix syntax. People were aware of it, and a few people used it, but it didn't catch on in a big way. I don't think this was for ideological reasons; I think it was simply that most people writing Lisp (myself included) found that with proper editor support, editing fully parenthesized code is easy and pleasant -- more so, in many cases, than editing infix code. That is the fact that many non-Lispers cannot believe could possibly be true. And yet over and over again, people who learn Lisp report that it is.
- sparkie 12y ago
  Poe's law? Honestly can't tell. Perhaps lisp programmers like the idea of composition, and decades later we still have no useful method for composing formal grammars? When we instead use a uniform tree structure and replace syntax with vocabulary, the problem disappears, and we only need to be concerned with composing the semantics of different "languages". Or maybe they also like being able to quote some tree structure and treat it as code, or quote some code an treat it as data too - without the need for some genius to create an "API" to serialize and deserialize the data, and re-implement the compiler/interpreter. Oh, if you need a feature, you could always ask the developers of your formal syntax to add it to the language, then wait a few years for the next revision to come around.
joe_the_user 12y ago
The abstract syntax was intended to be used writing programs until designers could get around to creating a concrete syntax with human-readable punctuation (instead of Lots of Irritating Silly Parentheses), but programmers soon got used to programming directly in abstract syntax. That's some programmers who got used to the syntax. It may be absolutely true that once you get used to LISP's syntax, it is better than any other. The problem is that since one is dealing with human beings, and with human beings you have the power of first impression and the tendency to make snap decisions based on these, people initial difficulty in grokking LISP's syntax is always going to be a hurdle for the language.
- Crito 12y ago
  Is grokking Lisp's parenthesized s-expressions actually difficult though? I think that the real problem is that developers believe that they will find it difficult before they try it, rather than actually finding it difficult when they eventually do try it. I don't think I've ever picked up another class of languages faster; even python took more effort from me. Basically, I think it's a marketing problem.
  - dimovich 12y ago
    We're hard-wired for Lisp. Broca's area 44,45 and 47 are responsible for processing hierarchical structures.[1] That's why it's so easy to reason about Lisp code. [1] http://www.letras.ufrj.br/poslinguistica/recursion/papers/17-friederici.pdf http://www.letras.ufrj.br/poslinguistica/recursion/papers/17...
    - thinkpad20 12y ago
      I don't see what that specifically has to do with Lisp. Virtually every programming language in use has a recursive hierarchical structure.
      - dimovich 12y ago
        The code itself is hierarchical in nature.
endlessvoid94 12y ago
"preventing advancement" is a pretty bold assumption.
tluyben2 12y ago
What most people have with Lisp syntax, I have with Objective-c. I am mostly blind to syntax (I skip from language to language fast reading/writing) except for some esoteric languages (like http://www.hakank.org/k/ http://www.hakank.org/k/) AND Objective-c. I find it extremely annoying to read while I do it every day. Writing is better for some reason, but far removed from other languages. So I see why some people would not like Lisp syntax while I find it very pleasant/enjoyable to read.
GeneralMayhem 12y ago
I have yet to hear a good explanation why homoiconicity is a good thing or in any way easier to learn. My main problem is that you have to know all the same things, but they become unknown unknowns. Not being able to tell at a glance that the argument list in a lambda expression, as one of the most common examples, is doing something other than execute code is very confusing if you're expecting everything to follow the (func arg1 arg2) pattern. On the other hand, when you encounter different syntax, it's immediately obvious that you need to know something else - a known unknown, which is easier to correct.
- roryokane 12y ago
  The main advantage of homoiconicity is that it lets you write macros more easily and safely. This Stack Overflow answer lists some reasons why this is so: http://stackoverflow.com/a/9146303/578288 http://stackoverflow.com/a/9146303/578288. Macros are good because they let you refactor repetitive bits of code that functions can’t refactor. With the extra refactoring, your program is simpler and easier to modify, because its abstractions are more apt.
- adwf 12y ago
  I think part of the problem is that Lisp is an incredibly broad language in terms of its capabilities. The macro system can be extremely powerful and when abused, is a nightmare to decipher. But. In the real world, I've rarely ever encountered someone who has actually programmed in such a bad style. A macro to me is almost no different than a function declaration (ie. (func arg1 arg2)), I just have to bear in mind whether it's being executed at compile or run time. Frequently I find a macro unnecessary and replace it with a straight function definition. My golden rule is to avoid nesting macros in my own code (libraries are a different matter - wrap 'em in error handlers). If I stick to that, it becomes very simple and very powerful. It's not a very strict rule, but it is a helpful one as it reduces the general level of abstraction to a manageable level, while still being useful. Due to Lisp's flexibility, I think you have to maintain stricter coding standards than you perhaps would otherwise (but also know when to break them). If you create the right macros, you can save hundreds of lines of code and make the codebase far more maintainable. A good example I had recently was for a website; I had a template macro for all the <doctype>/<head>/css/js and had the navbar, footer, general layout in the template as well. Then I had each webpage defined as a function that called the template macro and then inserted the rest of the <body> into the page. (This saved a ton of code in terms of templating; it could probably be done in other languages too, but you'd have to squeeze it into OO format or something) This was all well and good, I had about 20 pages done before I ran into a bug. In some other languages, you would then go round adding a debug line/condition handler to every function, but instead I added it to the main template macro just once and it handled all the functions. Even better, the macro system and condition handler allowed me to deal with the code both before and after it had been compiled. I then passed the error up the stack from the function to the macro's handler, had the handler choose a fix (recompile the page function with different params), then send it back to the function (without ever exiting the stack) and carry on as if nothing happened. The users see nothing. I don't use a lot of the things Lisp can do (eg. I can't remember the last time I made a function generate another function and return it), but it works exceptionally well for what I do use (currying, composing, macros and more).
cjo 12y ago
What I think this is missing which was a huge part of the SICP course is that Lisp is probably the best language to write your own language in with a custom syntax tailored for the problem space you're working on. Even if it isn't trivial to do. An example is Clojure's Hiccup[0] where there's a whole new syntax for dynamic HTML generation. It's not a full new Turing-complete language, but it's a custom syntax and vocabulary that illustrates Lisp's extensibility (a sample of Hiccup in action is linked below, though it isn't mine.) [0] - https://github.com/yokolet/hiccup-samples/blob/master/src/hiccup_templating/views/contents.clj https://github.com/yokolet/hiccup-samples/blob/master/src/hi...
- dreamcompiler 12y ago
  This one of the reasons why, as a Lisp programmer, I don't care much for Clojure. Three different bracketing constructs when one would do just fine. Now I have to go look up the difference between () and {} and [] just to emit some HTML.
  - gfodor 12y ago
    No, you wouldn't. Idiomatically [] is used for declarative structures for consumption by macros (like argument lists) and {} is obviously a map, and is used for any key/value pairs. So the HTML DSL linked above is the most intuitive one to a clojure programmer with regard to the data structures used. Adding maps and vectors to be first-level syntax to clojure is one of my favorite things about the language.
    - hga 12y ago
      As an old Lisp hand, while I'd prefer those declarative structures to be in old fashioned parens, it's not too obnoxious and has some clarity value in adding it as language syntax. But I agree 100% with the value of adding to the first-level/first-class syntax, it's both clear and very convenient.
  - wonderzombie 12y ago
    That's funny. This is one of the reason why I don't care much for Common Lisp. Hashtables in particular are common as mud in programming, but I have to work with a clunky interface specific to each abstraction. Ironically this puts CL closer to Java and C++ than the likes of Python and Ruby.
lcedp 12y ago
The abstract syntax was intended to be used writing programs until designers could get around to creating a concrete syntax with human-readable punctuation (instead of Lots of Irritating Silly Parentheses), but programmers soon got used to programming directly in abstract syntax. Perhaps it was, but.. 1) The nature of research is so that you can't always predict in which way your invention will be useful. To me and many others it seems lisp syntax is quite handy. 2) If we take Clojure for the sake of example, we'd see that it comes with some neat macros which are important part of the language. Those can be viewed as mentioned concrete syntax elements built upon the abstract syntax. Punctuation's been improved as well.
hajile 12y ago
The biggest proof of the overwhelming preference for parens can be found in the dylan programming language. Despite having M-expression syntax (in fact, the original spec was prefix lisp), the same CLOS object system, and even a macro system, the language has gone nowhere with lispers. More interesting is that dylan was also unpopular with ALGOL programmers. The existence of "sweet expressions" (M-expressions) allowing infix syntax in existing lisp languages further removes this as the main issue. The worst answer as to why people hate lisp is the "too many parenthesis" argument. C-style programmers are just as happy to mash 5-6 paren/curly-brace sets together in one line. The only major difference is location. For example, (if test (<result>) (<alternate>)) vs if(test){<result>}else{<alternate>}. In addition, the only reason the closing parens are on a separate line in C and the same line in lisp is convention. I suspect the biggest barrier to ANY language is having a non-C syntax style rather than infix vs prefix or parenthesis. The inertia is simply too great.

35 points by mrbbk 12y ago

Well, yeah. I think the popularity of lisp comes down to the hostility the mit ai lab had towards noam chomsky who in the 1960s spoke out about the vietnam war and thus bit the hand that funds cs research. All other computer languages (except for forth) embraced formal grammar concepts but mit rejected them out of spite. If mit had stopped burning the dragon book and embraced algol syntax we might have been spared the horrors of c, c++ and java but no, they just gave us 1 more reason rt 128 is a boulevard ofbroken dreams.

ScottBurson 12y ago
Huh. I was around the AI Lab in the late 1970s -- as an undergrad -- and I don't recall hearing anyone talk about Chomsky in the way you imagine. DARPA funding was still flowing rather freely, it seemed to me, well into the mid-1980s. I don't recall anyone suggesting that there would have been yet more DoD money if it hadn't been for Chomsky. And the grad student I did some work for, who was working on natural language parsing, was clearly intimately familiar with Chomsky's ideas on transformational grammar and referred to them throughout his dissertation. So this isn't ringing true for me. I also recall Gerry Sussman speaking quite highly of Algol 60. No, I think you're quite wrong: the reasons for the popularity of Lisp at the AI Lab have to do with its actual advantages over other languages for the kinds of problems they were working on. And there is an example of a language in the MIT tradition with infix syntax: Dylan. Great language, but wasn't in the right place at the right time to catch on.
- hga 12y ago
  Was Sussman talking about the syntax, or things like lexical scope, which he and Steele introduced to the LISP world with Scheme in the mid-70s? About the only relevant negative thing I remember being said about Chomsky in a period just after your's was Marvin Minsky saying he saved someone from going down a rabbit hole following Chomsky's work. But that was a judgement of merit of his linguistics work, not his politics.
  - ScottBurson 12y ago
    Well, the lexical scope, certainly, but I don't recall him disparaging the syntax. I don't recall any burning of Dragon Books. Nor was Vaughan Pratt tarred and feathered for releasing CGOL, which allowed one to write Lisp in infix syntax. People were aware of it, and a few people used it, but it didn't catch on in a big way. I don't think this was for ideological reasons; I think it was simply that most people writing Lisp (myself included) found that with proper editor support, editing fully parenthesized code is easy and pleasant -- more so, in many cases, than editing infix code. That is the fact that many non-Lispers cannot believe could possibly be true. And yet over and over again, people who learn Lisp report that it is.
sparkie 12y ago
Poe's law? Honestly can't tell. Perhaps lisp programmers like the idea of composition, and decades later we still have no useful method for composing formal grammars? When we instead use a uniform tree structure and replace syntax with vocabulary, the problem disappears, and we only need to be concerned with composing the semantics of different "languages". Or maybe they also like being able to quote some tree structure and treat it as code, or quote some code an treat it as data too - without the need for some genius to create an "API" to serialize and deserialize the data, and re-implement the compiler/interpreter. Oh, if you need a feature, you could always ask the developers of your formal syntax to add it to the language, then wait a few years for the next revision to come around.

joe_the_user 12y ago

The abstract syntax was intended to be used writing programs until designers could get around to creating a concrete syntax with human-readable punctuation (instead of Lots of Irritating Silly Parentheses), but programmers soon got used to programming directly in abstract syntax. That's some programmers who got used to the syntax. It may be absolutely true that once you get used to LISP's syntax, it is better than any other. The problem is that since one is dealing with human beings, and with human beings you have the power of first impression and the tendency to make snap decisions based on these, people initial difficulty in grokking LISP's syntax is always going to be a hurdle for the language.

Crito 12y ago
Is grokking Lisp's parenthesized s-expressions actually difficult though? I think that the real problem is that developers believe that they will find it difficult before they try it, rather than actually finding it difficult when they eventually do try it. I don't think I've ever picked up another class of languages faster; even python took more effort from me. Basically, I think it's a marketing problem.
- dimovich 12y ago
  We're hard-wired for Lisp. Broca's area 44,45 and 47 are responsible for processing hierarchical structures.[1] That's why it's so easy to reason about Lisp code. [1] http://www.letras.ufrj.br/poslinguistica/recursion/papers/17-friederici.pdf http://www.letras.ufrj.br/poslinguistica/recursion/papers/17...
  - thinkpad20 12y ago
    I don't see what that specifically has to do with Lisp. Virtually every programming language in use has a recursive hierarchical structure.
    - dimovich 12y ago
      The code itself is hierarchical in nature.

endlessvoid94 12y ago

"preventing advancement" is a pretty bold assumption.

tluyben2 12y ago

What most people have with Lisp syntax, I have with Objective-c. I am mostly blind to syntax (I skip from language to language fast reading/writing) except for some esoteric languages (like http://www.hakank.org/k/ http://www.hakank.org/k/) AND Objective-c. I find it extremely annoying to read while I do it every day. Writing is better for some reason, but far removed from other languages. So I see why some people would not like Lisp syntax while I find it very pleasant/enjoyable to read.

GeneralMayhem 12y ago

I have yet to hear a good explanation why homoiconicity is a good thing or in any way easier to learn. My main problem is that you have to know all the same things, but they become unknown unknowns. Not being able to tell at a glance that the argument list in a lambda expression, as one of the most common examples, is doing something other than execute code is very confusing if you're expecting everything to follow the (func arg1 arg2) pattern. On the other hand, when you encounter different syntax, it's immediately obvious that you need to know something else - a known unknown, which is easier to correct.

roryokane 12y ago
The main advantage of homoiconicity is that it lets you write macros more easily and safely. This Stack Overflow answer lists some reasons why this is so: http://stackoverflow.com/a/9146303/578288 http://stackoverflow.com/a/9146303/578288. Macros are good because they let you refactor repetitive bits of code that functions can’t refactor. With the extra refactoring, your program is simpler and easier to modify, because its abstractions are more apt.
adwf 12y ago
I think part of the problem is that Lisp is an incredibly broad language in terms of its capabilities. The macro system can be extremely powerful and when abused, is a nightmare to decipher. But. In the real world, I've rarely ever encountered someone who has actually programmed in such a bad style. A macro to me is almost no different than a function declaration (ie. (func arg1 arg2)), I just have to bear in mind whether it's being executed at compile or run time. Frequently I find a macro unnecessary and replace it with a straight function definition. My golden rule is to avoid nesting macros in my own code (libraries are a different matter - wrap 'em in error handlers). If I stick to that, it becomes very simple and very powerful. It's not a very strict rule, but it is a helpful one as it reduces the general level of abstraction to a manageable level, while still being useful. Due to Lisp's flexibility, I think you have to maintain stricter coding standards than you perhaps would otherwise (but also know when to break them). If you create the right macros, you can save hundreds of lines of code and make the codebase far more maintainable. A good example I had recently was for a website; I had a template macro for all the <doctype>/<head>/css/js and had the navbar, footer, general layout in the template as well. Then I had each webpage defined as a function that called the template macro and then inserted the rest of the <body> into the page. (This saved a ton of code in terms of templating; it could probably be done in other languages too, but you'd have to squeeze it into OO format or something) This was all well and good, I had about 20 pages done before I ran into a bug. In some other languages, you would then go round adding a debug line/condition handler to every function, but instead I added it to the main template macro just once and it handled all the functions. Even better, the macro system and condition handler allowed me to deal with the code both before and after it had been compiled. I then passed the error up the stack from the function to the macro's handler, had the handler choose a fix (recompile the page function with different params), then send it back to the function (without ever exiting the stack) and carry on as if nothing happened. The users see nothing. I don't use a lot of the things Lisp can do (eg. I can't remember the last time I made a function generate another function and return it), but it works exceptionally well for what I do use (currying, composing, macros and more).

cjo 12y ago

What I think this is missing which was a huge part of the SICP course is that Lisp is probably the best language to write your own language in with a custom syntax tailored for the problem space you're working on. Even if it isn't trivial to do. An example is Clojure's Hiccup[0] where there's a whole new syntax for dynamic HTML generation. It's not a full new Turing-complete language, but it's a custom syntax and vocabulary that illustrates Lisp's extensibility (a sample of Hiccup in action is linked below, though it isn't mine.) [0] - https://github.com/yokolet/hiccup-samples/blob/master/src/hiccup_templating/views/contents.clj https://github.com/yokolet/hiccup-samples/blob/master/src/hi...

dreamcompiler 12y ago
This one of the reasons why, as a Lisp programmer, I don't care much for Clojure. Three different bracketing constructs when one would do just fine. Now I have to go look up the difference between () and {} and [] just to emit some HTML.
- gfodor 12y ago
  No, you wouldn't. Idiomatically [] is used for declarative structures for consumption by macros (like argument lists) and {} is obviously a map, and is used for any key/value pairs. So the HTML DSL linked above is the most intuitive one to a clojure programmer with regard to the data structures used. Adding maps and vectors to be first-level syntax to clojure is one of my favorite things about the language.
  - hga 12y ago
    As an old Lisp hand, while I'd prefer those declarative structures to be in old fashioned parens, it's not too obnoxious and has some clarity value in adding it as language syntax. But I agree 100% with the value of adding to the first-level/first-class syntax, it's both clear and very convenient.
- wonderzombie 12y ago
  That's funny. This is one of the reason why I don't care much for Common Lisp. Hashtables in particular are common as mud in programming, but I have to work with a clunky interface specific to each abstraction. Ironically this puts CL closer to Java and C++ than the likes of Python and Ruby.

lcedp 12y ago

The abstract syntax was intended to be used writing programs until designers could get around to creating a concrete syntax with human-readable punctuation (instead of Lots of Irritating Silly Parentheses), but programmers soon got used to programming directly in abstract syntax. Perhaps it was, but.. 1) The nature of research is so that you can't always predict in which way your invention will be useful. To me and many others it seems lisp syntax is quite handy. 2) If we take Clojure for the sake of example, we'd see that it comes with some neat macros which are important part of the language. Those can be viewed as mentioned concrete syntax elements built upon the abstract syntax. Punctuation's been improved as well.

hajile 12y ago

The biggest proof of the overwhelming preference for parens can be found in the dylan programming language. Despite having M-expression syntax (in fact, the original spec was prefix lisp), the same CLOS object system, and even a macro system, the language has gone nowhere with lispers. More interesting is that dylan was also unpopular with ALGOL programmers. The existence of "sweet expressions" (M-expressions) allowing infix syntax in existing lisp languages further removes this as the main issue. The worst answer as to why people hate lisp is the "too many parenthesis" argument. C-style programmers are just as happy to mash 5-6 paren/curly-brace sets together in one line. The only major difference is location. For example, (if test (<result>) (<alternate>)) vs if(test){<result>}else{<alternate>}. In addition, the only reason the closing parens are on a separate line in C and the same line in lisp is convention. I suspect the biggest barrier to ANY language is having a non-C syntax style rather than infix vs prefix or parenthesis. The inertia is simply too great.