16 ms·

Just what does “code as data” mean anyway? (2014)

149 points by jxub 9y ago

mpweiher 9y ago
The Elixir getting started link is now at: https://elixir-lang.org/getting-started/introduction.html https://elixir-lang.org/getting-started/introduction.html
divs1210 9y ago
PigPen is an amazing wrapper over Pig that shows why and how macros are important and useful. https://github.com/Netflix/PigPen https://github.com/Netflix/PigPen
RaycatRakittra 9y ago
(2014) "Code as data" is a wonderful thing but I also enjoy writing Lisp, so take that with a grain of salt.
- ZenoArrow 9y ago
  > "(2014)" Thought it was odd when I saw the name "Nimrod", that explains why.
PaulHoule 9y ago
One of the failures of conventional programming libraries is that parsing libraries such as yacc work in only one direction. In 2018 we could easily have libraries that work both ways by default, but we don't. Bidirectional parsing is great for code generation and opens up a lot of things you could do easily, but because common parsing libraries are unidirectional, people aren't aware of what you can do and don't clamor for bidirectional parsing. It is very possible and practical to parse conventional languages down to an AST tree, work on the tree, and run that code. See https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy Sometimes I wonder if the LISP cult is just trying to pretend Noam Chomsky was never born.
- mmjaa 9y ago
  > clamor for bidirectional parsing I get this from Lua. "Everything to a bytecode and back again" ..
- pjc50 9y ago
  What do you actually mean by "bidirectional parsing" - re-generating the original input, or something equivalent to it, from the AST?
  - PaulHoule 9y ago
    Yep. "Something equivalent" is pretty easy. To be able to regenerate the original input (spacing and other foibles) is a bit harder but also possible. Commercial vendors developed tools that can do this in the 00's to support UML and UML-like tooling. Related to this is getting sensible output structures. For instance, many parsing libraries based on parsing combinators output a "lispy" combination of lists/dicts/scalars which is dependent on the details of the grammar, for ex. https://github.com/neogeny/tatsu https://github.com/neogeny/tatsu it is particularly obnoxious when operator precedence is implemented by creating extra nodes in the parse tree, and even more obnoxious when you have to copy those already over-complicated structures in your grammar to deal with complex context. Even in 2018 people are still using some compiler compilers that use a "callback hell" interface that dates back to yacc. That might have been OK in 1978, but it is one of the many unergonomic things that put compiler compilers out of reach for many application programmers.
    - Steltek 9y ago
      You want a decompiler that produces the original language, not just marked up assembly. For pure VM or interpreted languages, you're right this doesn't seem impossible but has unacceptable tradeoffs for most people. For starters, most companies like that compilation is a one-way action. Whitespace is easy, btw. Just enforce code style in the language like gofmt. You don't need to preserve what you can derive programmatically. As a bonus, gofmt is awesome.
      - 9y ago
jbrennan 9y ago
See also Mark Miller’s excellent blog post on the same topic, which really turned on a few lightbulbs for me: https://tekkie.wordpress.com/2010/07/05/sicp-what-is-meant-by-data/ https://tekkie.wordpress.com/2010/07/05/sicp-what-is-meant-b...
peatmoss 9y ago
And, as I wrote in another thread yesterday, it allows you to work with your text editor at a higher level of abstraction than non-lisp-coding people may have seen before (https://news.ycombinator.com/item?id=16386380 https://news.ycombinator.com/item?id=16386380). Rather than thinking in letters, words, lines, or paragraphs, editor modes like paredit let you edit in terms of the structure of your code. I find this really hard to give up after lispy sessions.
- jstimpfle 9y ago
  I really value C syntax. I'm convinced it's much easier to navigate with the eyes than "oatmeal with fingerclips mixed in" (quoting Larry Wall). And >90% of the time the relevant syntactic unit is on its own line or ranges of lines, and that's really super easy to handle in vim. I don't think there is much to improve by building more complex abstractions on top. If you use vim as a C programmer, you should know the basic line operations like dd, p, and Shift+V (line range select). Also, I often use just { and } to navigate to the next/previous empty line. These keyboard shortcuts get those >90% covered. Beyond that, you can use Control+V (block select), and XaY where X can be c (change) or d (delete) or v (visual select), and Y can be things like " (string literals), w (identifier), { (braced block) or ( (parenthesized expression). I don't disagree that if all you have is parens, then you want something like paredit. But for a C programmer there is no need for such a thing.
  - peatmoss 9y ago
    I find C syntax awkward, bloated, inconsistent, and gross compared to lisp syntax, but that’s an aesthetic preference. Beyond the personal preferences of individuals (individuals who probably started with one kind of syntax or another) there are characteristics that have some objective utility. I appreciate that a lot of people start with C-ish syntax, and a similarly large number of people prefer that syntax, but I’ve never heard anything that demonstrated an intrinsic benefit of C syntax.
    - jstimpfle 9y ago
      It's obviously a lot shorter and more distinctive. It's highly optimized for assignments, pointer and array dereferences, address-of operations and data definitions. If that's what you do most of the day, you want C syntax.
tombert 9y ago
When I was first learning C, I wondered the point of macros; especially with inline functions. It wasn't until I taught myself Racket and Clojure a few years later that I realized the utility, and it was immediately one of those "the world is different now" moments; I could actually augment the language without having to contribute to the core compiler.
- carlmr 9y ago
  It's very nice, but with great power comes great responsibility. Some people program so many things with C macros that it becomes very hard to read their code.
  - zeveb 9y ago
    That's the beauty of Lisp: it's all very readable, because it's a real language, not a string-substitution preprocessor. Yes, any idiot can write unreadable Lisp — but good Lisp is a work of art. The Art of the Metaobject Protocol should be required reading: it builds an entire object system, complete with classes and generic functions (multimethods), written in the language itself. It's a tour de force.
    - metaobject 9y ago
      And I'm just pointing this out for those who aren't already aware -- C macros and Lisp macros aren't really comparable. C macros are implemented as string substitutions in a separate step before compilation. Lisp macros can make use of the rest of the language features and allow for operations on the abstract syntax tree of the code, opening up things like adding new control constructs, generating boilerplate code, string substitution, and the creation of domain specific languages.
      - deleted 9y ago
        [deleted]
- yetanotheruser 9y ago
  I'm still wrapping my head around macros, but I have been able to come up with a couple use cases that I quickly realized were already covered by existing macros, the most recent one being a less flexible -> macro. Anyways, I kept reading places that lisp macros and c macros were completely distinct, which seems untrue after a couple years of trying clojure.
quadcore 9y ago
See what I did there? Despite the good intention, no, because you're trying to explain lisp using lisp. If I knew lisp, I wouldnt need you to explain what code-as-data means. Maybe it's time for a different strategy. As a matter of fact, I've tried to explain code-as-data using OOP just this morning: https://news.ycombinator.com/item?id=16389514 https://news.ycombinator.com/item?id=16389514
- 3pt14159 9y ago
  Your linked comment is great but it sweeps one thing under the rug. There is a reason lisp didn't win. We put up with it (I use emacs, so I have to write it at times) but in general it's harder to reason about transformable lists of code than it is about plain old objects and their methods. Almost every problem elegantly solved in lisp has a parallel elegant solution in Ruby (using blocks, dynamic method definition, or otherwise) and I generally find the resulting code both more maintainable as well as more readable. Sometimes with lisp things get "spooky" in a way that just doesn't happen in practice with Ruby, even if it theoretically possible to say redefine String#inspect. Though I completely agree that inheritance gets clunky and I strongly favour composition.
  - quadcore 9y ago
    I agree, OO is ultimately good enough. Problem is, coding using OOP is like printing ideas on paper: you dont wanna do that because it's hard to discard ideas that's been printed on paper. You wanna use post-its instead because post-its are easy to discard. Refactoring is hard using OO and refactoring is the key difference between waterfall and good software development. Refactoring is easier, even fun, using functional programming because the medium is more malleable.
    - flavio81 9y ago
      >I agree, OO is ultimately good enough. And Common Lisp has arguably the most flexible and powerful OOP system around, CLOS. Why don't write OOP in Lisp? It is done successfully by lispers around the world.
dmitriid 9y ago
One of the greatest lies that Lispers have is: "Lisp has no syntax". Syntax is defined as "the structure of statements in a computer language." What Lisp has, and is, is a syntax to describe an AST. If you get the syntax wrong, your program won't run. And even that syntax isn't uniform across the various Lisps (some will throw in weird chars and constructs here and there to make dealing with common structures easier etc.) It does make it somewhat easier to manipulate code as data if you wish to. And it does somewhat make reasoning about some parts of your code somewhat easier. Is it that good as it's glorified to be? The author links to Korma as an example of the power of code as data, and macros: (select users (where {:active true}) (order :created) (limit 5) (offset 3)) :-\ To me, it's a chain of 5 functions which are neither shorter to write nor better than Java's jOOQ [1]: create.select(a.FIRST_NAME, a.LAST_NAME, countDistinct(s.NAME)) .from(a) .join(b).on(b.AUTHOR_ID.eq(a.ID)) .join(t).on(t.BOOK_ID.eq(b.ID)) .join(s).on(t.BOOK_STORE_NAME.eq(s.NAME)) .groupBy(a.FIRST_NAME, a.LAST_NAME) .orderBy(countDistinct(s.NAME).desc()) .fetch(); Look, it even has a similar number of parentheses ;) [1] Example from https://www.jooq.org/doc/3.10/manual-single-page/#sql-building https://www.jooq.org/doc/3.10/manual-single-page/#sql-buildi...
- kazinator 9y ago
  That particular select instance could indeed just be an ordinary function, whereby the (where ...) and (order ...) are just evaluated argument expressions, also calling ordinary constructors for objects that influence the query. It doesn't really demonstrate the ability to manipulate syntax. Things start to get more interesting when some of the inputs need to be lambdas. Still, the sugaring of those can be in the arguments, and select can remain a function. Now suppose that a clause can, say, bind a lexical variable that a later clause can somehow usefully refer to; things like that.
  - dmitriid 9y ago
    Unless I’m missing something everything you said applies to jOOQ code.
- 9y ago
gameswithgo 9y ago
Does anyone have some examples of great products that have been built largely by leveraging lisp macros? emacs?
- flavio81 9y ago
  >Does anyone have some examples of great products that have been built largely by leveraging lisp macros? Common Lisp itself. The Common Lisp language is not only made up by data types, functions and control constructs: Many of the typical keywords a Common Lisp programmer would use, like defun (for defining a function), are macros themselves. Many control constructs are macros as well. So a good part of the language is built using macros as well. The compiler itself (compiling Lisp to machine language) is also mostly made up of macros in many Lisp implementations. Thus, the answer is: Common Lisp implementations are built largely by leveraging lisp macros. And i'd say those are "great products."
- erikj 9y ago
  One of my favorite examples: https://franz.com/success/customer_apps/animation_graphics/izware.lhtml https://franz.com/success/customer_apps/animation_graphics/i...
scottmsul 9y ago
Would it be possible to do the same thing in Haskell using partial function application, where the first argument is the name of the tag and the second argument makes the "<" ">" tags and applies string concatenation?
- andolanra 9y ago
  Absolutely. However, I will note that, while Haskell makes partial application very easy, there's also nothing Haskell-specific about it: you can write curried functions in many languages, possibly by using helper functions or just manually transforming your code: def tag(tag): def render(*body): return '<{tag}>{body}</{tag}>'.format( tag=tag, body=''.join(body)) return render html = tag('html') body = tag('body') h1 = tag('h1') p = tag('p') I also should note that, while the macros in this case could be obviated by approaching the problem slightly differently, I don't think that's true of Lisp macros in general. I don't want to be be too harsh to this particular blog post—coming up with a two-or-three line motivating example that's accessible to a general audience without being contrived is very hard! But one consequence of the slightly-contrived nature of the example in this post is that it's very easy to come up with non-macro ways of solving the same problem, as demonstrated above.
- yorwba 9y ago
  You could get most of the value in Haskell using higher-order functions, but note that the macro-based code generates a complete function definition from (deftag tagname). Since definitions aren't values in Haskell (code isn't data), you can't just write deftag "html" but instead would have to write something like html = tag "html" I'm not sure whether you could get around that limitation using Template Haskell (never tried it).
- kamaal 9y ago
  Haskell comes from a family of languages called ML. And ML was known as Lisp without parentheses. So its again Lisp all over again.
  - andolanra 9y ago
junke 9y ago
Why call `(map eval inner)`?
segmondy 9y ago
Best language to understand this is prolog. dog(fido). This is data and it's also code. It's data and code that state's that there's a dog called fido. I could have written it as exists_dog(fido).
- draw_down 9y ago
  Uh, thanks.
accatyyc 9y ago
I think there’s a much simpler example of how code is data: (hello world) That’s a list containing two symbols. So it’s data. However, if I evaluate that list, it will call the function “hello” with the argument “world”. So it’s also code. Incidentally, lists and function calls are identical in lisp, hence code is data. I think the article just complicates it by introducing macros.
- rootlocus 9y ago
  '(hello world) is a list (hello world) is a function call
  - Karunamon 9y ago
    Well, technically they're both lists, it's just that the ' prefix makes it a list literal rather than having the compiler try to macro expand it. A bit pedantic, but of massive importance when macros are in play.
    - tincholio 9y ago
      Evaluate it, not macro expand it ;)
  - junke 9y ago
    Typically when writing about lisp code, it is also correct to talk about how "read" sees the expression, to avoid some confusion about what happens after "eval". That's how I interpret the parent comment when it says that (hello world) is a list of two elements.
platz 9y ago
Code is data does not come from lisp or macros, those are simply affordances. It comes from computer hardware architecture. The von Neumann Architecture is named after the mathematician and early computer scientist John von Neumann. von Neumann machines have shared signals and memory for code and data. Thus, the program can be easily modified by itself since it is stored in read-write memory. Any self-modifying program, even if it is written in Borland Delphi, expresses this code is data capability.
- AnimalMuppet 9y ago
  It's even deeper than that. It's not just how code is handled in a von Neumann architecture, it's how code is created. Data is just a sequence of bytes. Source code is also just a sequence of bytes - it's just a data file with a particular structure and intent. And then an executable is just a sequence of bytes - it's another data file, though on Unix it's one with the executable permission set. A compiler reads in a data file and writes a data file - they just happen to be source code and executable programs, respectively. (Now, when Lisp people say "code is data", they mean much more than compile time. For other languages, that is less true, other than self-modifying code...)
  - platz 9y ago
    The hardware is more fundamental. (Unless you want to invoke mathematics e.g. term rewriting) The compiler/seq of bytes example requires von neumann, because in that case it is the behavior of the whole system that is self-modifying. You could not have the beautiful executable and interchangable files of bytes without von neumann.
    - AnimalMuppet 9y ago
      I don't think that's right. There have been systems that were Harvard architecture (Motorola 88000, I think, and definitely others). But the code to run on them still started out as a sequence of bytes in a text file, that is, as data, and a compiler still took that data file and wrote another data file that was the executable image.
DannyB2 9y ago
Code is data. In Java, I could use a bytecode manipulation library to read in a class file, manipulate it by using the library's API to add, modify, delete members of the class. Then write the class back out as a file. Code becomes data and is manipulated as date and written back out. In Lisp 'code is data' is so much more natural because code IS the most elementary data structures of the language. No need for a library to manipulate code behind an API. The API is the language primitives. The code is the data structure you manipulate using language primitives.
marknadal 9y ago
This is exactly what we mean as "code as data" and why it is so profound and has such exciting potential: vimeo.com/208899228/b9bc9eaaa4#t=13m50s "The Future of Programming and Databases" ^ JSRemote Conf / NodeJS Italy
whitten 9y ago
For me, "code as data" means that the code I write provides a particular structure (whether to create a report, or go through an editing task or find particular information in the computer). I then use a table-driven "data" approach to finding the particulars that are significant for this particular class of information. That might mean that I have the unique code stored in a table which I lookup using a relevant key, hence "table-driven code", or I have "setup code", "one iteration", and "teardown code" accessible through an indirect call because the structure of the code to do the task can stay the same. It is also common to have a general "event" fire at the end of the code to inform any other programs that need to know this data structure or real-world event has been accounted for. I think this approach is amentable to better testing, and can be viewed as a higher-level abstraction similar to LISP macros. (Even though my language of choice - MUMPS) doesn't have LISP-style syntactic macros.
err4nt 9y ago
That example was great - I've been inspired by LISP and have been playing around with my own idea of an HTML-building set of functions already this week: http://staticresource.com/html-js-4.html http://staticresource.com/html-js-4.html It's great to see such a relevant example :D
sbov 9y ago
As someone who uses Clojure for only side projects, macros seem to get a lot of attention, both good and bad, for something I very rarely write. Maybe I'm missing something though, and programmers using these languages professionally resort to them more often than I do.
- dustingetz 9y ago
  code-is-data also enables structural editing–instead of typing strings we can transform data–, there are gifs of that here: https://cursive-ide.com/userguide/paredit.html https://cursive-ide.com/userguide/paredit.html
didibus 9y ago
In Java, you have code as string. That is, your code is represented using a big string. In Lisp, your code is represented using the list data structure. So its really code-as-data-structure. A string is difficult to parse and modify, inserting things in the middle, removing elements, changing the order of the words, that's all really difficult with a string. So if you want to transform Java code, its going to be hard and error prone. A list is easy to manipulate in contrast. Inserting elements in the middle is trivial, so are deletes and swaps. So in Lisp, if you want to transform code its pretty easy. Meta-programming is when you write a program that writes a program. An example is say you want to you wanted to add a semi-colon at the end of all your lines of code. You need a macro to do it for you. A macro is a program that acts upon your code to transform it. Eclipse has them. So now its really easy to add a semi-colon at the end of each line. What if you wanted to add a comma between all words in a selection? Now its trickier if your macro operates over a big string, you might need a regex for example. This is meta programming though. Instead of adding the commas yourself, which are required for the program to run, you write another program to add them for you. Now if all the words were elements in a list, that macro would be a lot easier to write. This is in essence what code-as-data(structure) means. Its in contrast with code-as-string. You don't have to choose lists as your datastructure either, as long as its something that allows you to represent a turing complete program and is easy to manipulate. Now, homoiconicity is the fact that your text of code looks like a data-structure too. Making it trivial to parse it into one. So back to my example, you could parse the java code string into a list, and then add commas, and then convert it back to a string. But Java code doesn't map logically into a list. Some construct don't nest like lists, and how do you define what goes into each node of the list? Do you group public and String together? In Lisp, the syntax is an unambiguous AST already, its thus trivial to parse into a list of lists. So you can easily get that AST you need to easily add commas where it make sense. Finally, there's a third aspect. Code-as-data also implies that your language can accept code as argument in the form of raw data. The best way to think of it is, how would you send a function over the wire to a program and have that program run the code I sent? You need a way to serialize that function, which is code, into raw data that can be transmitted over the wire. The receiving program doesn't have that function defined, so it needs to know how to deserialize it, but also at runtime it must be able to take this raw data, which represents code, and be able to parse it, compile/interpet it and execute it. Think of SQL, SQL is often used in a code-as-data way. You want "select * from %s". Now you'd take this as a string, and you'd use a string replace, and replace %s with something the user picked in a drop down. At runtime, you are dynamically creating the SQL code, and once you have it, running it. You might have methods that accept SQL and return SQL. Now again, your SQL is a big string, which isn't ideal. But this is still an example of code as data. Now in Java, you can not do that with Java code. Java does not have this concept of code-as-data. In other words, there's no eval. So when you combine all these three aspects, a homoiconic syntax that parses easily and logically into a data-structure which is easy to manipulate, and where you can then execute data which represent code at runtime and pass it around to other functions, even over the wire, you get a very powerful combo that turns into a Meta-programming powerhouse. This is the strength of Lisps, the one strength all Lisps share.
dogfishbar 9y ago
M-LISP: a representation-independent dialect of LISP with reduction semantics In this paper we introduce M-LISP, a dialect of LISP designed with an eye toward reconciling LISP's metalinguistic power with the structural style of operational semantics advocated by Plotkin [28]. We begin by reviewing the original definition of LISP [20] in an attempt to clarify the source of its metalinguistic power. We find that it arises from a problematic clause in this definition. We then define the abstract syntax and operational semantics of M-LISP, essentially a hybrid of M-expression LISP and Scheme. Next, we tie the operational semantics to the corresponding equational logic. As usual, provable equality in the logic implies operational equality. Having established this framework we then extend M-LISP with the metalinguistic eval and reify operators (the latter is a nonstrict operator that converts its argument to its metalanguage representation). These operators encapsulate the metalinguistic representation conversions that occur globally in S-expression LISP. We show that the naive versions of these operators render LISP's equational logic inconsistent. On the positive side, we show that a naturally restricted form of the eval operator is confluent and therefore a conservative extension of M-LISP. Unfortunately, we must weaken the logic considerably to obtain a consistent theory of reification.

149 points by jxub 9y ago

The Elixir getting started link is now at: https://elixir-lang.org/getting-started/introduction.html https://elixir-lang.org/getting-started/introduction.html

divs1210 9y ago

PigPen is an amazing wrapper over Pig that shows why and how macros are important and useful. https://github.com/Netflix/PigPen https://github.com/Netflix/PigPen

RaycatRakittra 9y ago

(2014) "Code as data" is a wonderful thing but I also enjoy writing Lisp, so take that with a grain of salt.

ZenoArrow 9y ago
> "(2014)" Thought it was odd when I saw the name "Nimrod", that explains why.

PaulHoule 9y ago

One of the failures of conventional programming libraries is that parsing libraries such as yacc work in only one direction. In 2018 we could easily have libraries that work both ways by default, but we don't. Bidirectional parsing is great for code generation and opens up a lot of things you could do easily, but because common parsing libraries are unidirectional, people aren't aware of what you can do and don't clamor for bidirectional parsing. It is very possible and practical to parse conventional languages down to an AST tree, work on the tree, and run that code. See https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy Sometimes I wonder if the LISP cult is just trying to pretend Noam Chomsky was never born.

mmjaa 9y ago
> clamor for bidirectional parsing I get this from Lua. "Everything to a bytecode and back again" ..
pjc50 9y ago
What do you actually mean by "bidirectional parsing" - re-generating the original input, or something equivalent to it, from the AST?
- PaulHoule 9y ago
  Yep. "Something equivalent" is pretty easy. To be able to regenerate the original input (spacing and other foibles) is a bit harder but also possible. Commercial vendors developed tools that can do this in the 00's to support UML and UML-like tooling. Related to this is getting sensible output structures. For instance, many parsing libraries based on parsing combinators output a "lispy" combination of lists/dicts/scalars which is dependent on the details of the grammar, for ex. https://github.com/neogeny/tatsu https://github.com/neogeny/tatsu it is particularly obnoxious when operator precedence is implemented by creating extra nodes in the parse tree, and even more obnoxious when you have to copy those already over-complicated structures in your grammar to deal with complex context. Even in 2018 people are still using some compiler compilers that use a "callback hell" interface that dates back to yacc. That might have been OK in 1978, but it is one of the many unergonomic things that put compiler compilers out of reach for many application programmers.
  - Steltek 9y ago
    You want a decompiler that produces the original language, not just marked up assembly. For pure VM or interpreted languages, you're right this doesn't seem impossible but has unacceptable tradeoffs for most people. For starters, most companies like that compilation is a one-way action. Whitespace is easy, btw. Just enforce code style in the language like gofmt. You don't need to preserve what you can derive programmatically. As a bonus, gofmt is awesome.
    - 9y ago

jbrennan 9y ago

See also Mark Miller’s excellent blog post on the same topic, which really turned on a few lightbulbs for me: https://tekkie.wordpress.com/2010/07/05/sicp-what-is-meant-by-data/ https://tekkie.wordpress.com/2010/07/05/sicp-what-is-meant-b...

peatmoss 9y ago

And, as I wrote in another thread yesterday, it allows you to work with your text editor at a higher level of abstraction than non-lisp-coding people may have seen before (https://news.ycombinator.com/item?id=16386380 https://news.ycombinator.com/item?id=16386380). Rather than thinking in letters, words, lines, or paragraphs, editor modes like paredit let you edit in terms of the structure of your code. I find this really hard to give up after lispy sessions.

jstimpfle 9y ago
I really value C syntax. I'm convinced it's much easier to navigate with the eyes than "oatmeal with fingerclips mixed in" (quoting Larry Wall). And >90% of the time the relevant syntactic unit is on its own line or ranges of lines, and that's really super easy to handle in vim. I don't think there is much to improve by building more complex abstractions on top. If you use vim as a C programmer, you should know the basic line operations like dd, p, and Shift+V (line range select). Also, I often use just { and } to navigate to the next/previous empty line. These keyboard shortcuts get those >90% covered. Beyond that, you can use Control+V (block select), and XaY where X can be c (change) or d (delete) or v (visual select), and Y can be things like " (string literals), w (identifier), { (braced block) or ( (parenthesized expression). I don't disagree that if all you have is parens, then you want something like paredit. But for a C programmer there is no need for such a thing.
- peatmoss 9y ago
  I find C syntax awkward, bloated, inconsistent, and gross compared to lisp syntax, but that’s an aesthetic preference. Beyond the personal preferences of individuals (individuals who probably started with one kind of syntax or another) there are characteristics that have some objective utility. I appreciate that a lot of people start with C-ish syntax, and a similarly large number of people prefer that syntax, but I’ve never heard anything that demonstrated an intrinsic benefit of C syntax.
  - jstimpfle 9y ago
    It's obviously a lot shorter and more distinctive. It's highly optimized for assignments, pointer and array dereferences, address-of operations and data definitions. If that's what you do most of the day, you want C syntax.

tombert 9y ago

When I was first learning C, I wondered the point of macros; especially with inline functions. It wasn't until I taught myself Racket and Clojure a few years later that I realized the utility, and it was immediately one of those "the world is different now" moments; I could actually augment the language without having to contribute to the core compiler.

carlmr 9y ago
It's very nice, but with great power comes great responsibility. Some people program so many things with C macros that it becomes very hard to read their code.
- zeveb 9y ago
  That's the beauty of Lisp: it's all very readable, because it's a real language, not a string-substitution preprocessor. Yes, any idiot can write unreadable Lisp — but good Lisp is a work of art. The Art of the Metaobject Protocol should be required reading: it builds an entire object system, complete with classes and generic functions (multimethods), written in the language itself. It's a tour de force.
  - metaobject 9y ago
    And I'm just pointing this out for those who aren't already aware -- C macros and Lisp macros aren't really comparable. C macros are implemented as string substitutions in a separate step before compilation. Lisp macros can make use of the rest of the language features and allow for operations on the abstract syntax tree of the code, opening up things like adding new control constructs, generating boilerplate code, string substitution, and the creation of domain specific languages.
    - deleted 9y ago
      [deleted]
yetanotheruser 9y ago
I'm still wrapping my head around macros, but I have been able to come up with a couple use cases that I quickly realized were already covered by existing macros, the most recent one being a less flexible -> macro. Anyways, I kept reading places that lisp macros and c macros were completely distinct, which seems untrue after a couple years of trying clojure.

quadcore 9y ago

See what I did there? Despite the good intention, no, because you're trying to explain lisp using lisp. If I knew lisp, I wouldnt need you to explain what code-as-data means. Maybe it's time for a different strategy. As a matter of fact, I've tried to explain code-as-data using OOP just this morning: https://news.ycombinator.com/item?id=16389514 https://news.ycombinator.com/item?id=16389514

3pt14159 9y ago
Your linked comment is great but it sweeps one thing under the rug. There is a reason lisp didn't win. We put up with it (I use emacs, so I have to write it at times) but in general it's harder to reason about transformable lists of code than it is about plain old objects and their methods. Almost every problem elegantly solved in lisp has a parallel elegant solution in Ruby (using blocks, dynamic method definition, or otherwise) and I generally find the resulting code both more maintainable as well as more readable. Sometimes with lisp things get "spooky" in a way that just doesn't happen in practice with Ruby, even if it theoretically possible to say redefine String#inspect. Though I completely agree that inheritance gets clunky and I strongly favour composition.
- quadcore 9y ago
  I agree, OO is ultimately good enough. Problem is, coding using OOP is like printing ideas on paper: you dont wanna do that because it's hard to discard ideas that's been printed on paper. You wanna use post-its instead because post-its are easy to discard. Refactoring is hard using OO and refactoring is the key difference between waterfall and good software development. Refactoring is easier, even fun, using functional programming because the medium is more malleable.
  - flavio81 9y ago
    >I agree, OO is ultimately good enough. And Common Lisp has arguably the most flexible and powerful OOP system around, CLOS. Why don't write OOP in Lisp? It is done successfully by lispers around the world.

dmitriid 9y ago

One of the greatest lies that Lispers have is: "Lisp has no syntax". Syntax is defined as "the structure of statements in a computer language." What Lisp has, and is, is a syntax to describe an AST. If you get the syntax wrong, your program won't run. And even that syntax isn't uniform across the various Lisps (some will throw in weird chars and constructs here and there to make dealing with common structures easier etc.) It does make it somewhat easier to manipulate code as data if you wish to. And it does somewhat make reasoning about some parts of your code somewhat easier. Is it that good as it's glorified to be? The author links to Korma as an example of the power of code as data, and macros: (select users (where {:active true}) (order :created) (limit 5) (offset 3)) :-\ To me, it's a chain of 5 functions which are neither shorter to write nor better than Java's jOOQ [1]: create.select(a.FIRST_NAME, a.LAST_NAME, countDistinct(s.NAME)) .from(a) .join(b).on(b.AUTHOR_ID.eq(a.ID)) .join(t).on(t.BOOK_ID.eq(b.ID)) .join(s).on(t.BOOK_STORE_NAME.eq(s.NAME)) .groupBy(a.FIRST_NAME, a.LAST_NAME) .orderBy(countDistinct(s.NAME).desc()) .fetch(); Look, it even has a similar number of parentheses ;) [1] Example from https://www.jooq.org/doc/3.10/manual-single-page/#sql-building https://www.jooq.org/doc/3.10/manual-single-page/#sql-buildi...

kazinator 9y ago
That particular select instance could indeed just be an ordinary function, whereby the (where ...) and (order ...) are just evaluated argument expressions, also calling ordinary constructors for objects that influence the query. It doesn't really demonstrate the ability to manipulate syntax. Things start to get more interesting when some of the inputs need to be lambdas. Still, the sugaring of those can be in the arguments, and select can remain a function. Now suppose that a clause can, say, bind a lexical variable that a later clause can somehow usefully refer to; things like that.
- dmitriid 9y ago
  Unless I’m missing something everything you said applies to jOOQ code.
9y ago

gameswithgo 9y ago

Does anyone have some examples of great products that have been built largely by leveraging lisp macros? emacs?

flavio81 9y ago
>Does anyone have some examples of great products that have been built largely by leveraging lisp macros? Common Lisp itself. The Common Lisp language is not only made up by data types, functions and control constructs: Many of the typical keywords a Common Lisp programmer would use, like defun (for defining a function), are macros themselves. Many control constructs are macros as well. So a good part of the language is built using macros as well. The compiler itself (compiling Lisp to machine language) is also mostly made up of macros in many Lisp implementations. Thus, the answer is: Common Lisp implementations are built largely by leveraging lisp macros. And i'd say those are "great products."
erikj 9y ago
One of my favorite examples: https://franz.com/success/customer_apps/animation_graphics/izware.lhtml https://franz.com/success/customer_apps/animation_graphics/i...

scottmsul 9y ago

Would it be possible to do the same thing in Haskell using partial function application, where the first argument is the name of the tag and the second argument makes the "<" ">" tags and applies string concatenation?

andolanra 9y ago
Absolutely. However, I will note that, while Haskell makes partial application very easy, there's also nothing Haskell-specific about it: you can write curried functions in many languages, possibly by using helper functions or just manually transforming your code: def tag(tag): def render(*body): return '<{tag}>{body}</{tag}>'.format( tag=tag, body=''.join(body)) return render html = tag('html') body = tag('body') h1 = tag('h1') p = tag('p') I also should note that, while the macros in this case could be obviated by approaching the problem slightly differently, I don't think that's true of Lisp macros in general. I don't want to be be too harsh to this particular blog post—coming up with a two-or-three line motivating example that's accessible to a general audience without being contrived is very hard! But one consequence of the slightly-contrived nature of the example in this post is that it's very easy to come up with non-macro ways of solving the same problem, as demonstrated above.
yorwba 9y ago
You could get most of the value in Haskell using higher-order functions, but note that the macro-based code generates a complete function definition from (deftag tagname). Since definitions aren't values in Haskell (code isn't data), you can't just write deftag "html" but instead would have to write something like html = tag "html" I'm not sure whether you could get around that limitation using Template Haskell (never tried it).
kamaal 9y ago
Haskell comes from a family of languages called ML. And ML was known as Lisp without parentheses. So its again Lisp all over again.
- andolanra 9y ago

junke 9y ago

Why call `(map eval inner)`?

segmondy 9y ago

Best language to understand this is prolog. dog(fido). This is data and it's also code. It's data and code that state's that there's a dog called fido. I could have written it as exists_dog(fido).

draw_down 9y ago
Uh, thanks.

accatyyc 9y ago

I think there’s a much simpler example of how code is data: (hello world) That’s a list containing two symbols. So it’s data. However, if I evaluate that list, it will call the function “hello” with the argument “world”. So it’s also code. Incidentally, lists and function calls are identical in lisp, hence code is data. I think the article just complicates it by introducing macros.

rootlocus 9y ago
'(hello world) is a list (hello world) is a function call
- Karunamon 9y ago
  Well, technically they're both lists, it's just that the ' prefix makes it a list literal rather than having the compiler try to macro expand it. A bit pedantic, but of massive importance when macros are in play.
  - tincholio 9y ago
    Evaluate it, not macro expand it ;)
- junke 9y ago
  Typically when writing about lisp code, it is also correct to talk about how "read" sees the expression, to avoid some confusion about what happens after "eval". That's how I interpret the parent comment when it says that (hello world) is a list of two elements.

platz 9y ago

Code is data does not come from lisp or macros, those are simply affordances. It comes from computer hardware architecture. The von Neumann Architecture is named after the mathematician and early computer scientist John von Neumann. von Neumann machines have shared signals and memory for code and data. Thus, the program can be easily modified by itself since it is stored in read-write memory. Any self-modifying program, even if it is written in Borland Delphi, expresses this code is data capability.

AnimalMuppet 9y ago
It's even deeper than that. It's not just how code is handled in a von Neumann architecture, it's how code is created. Data is just a sequence of bytes. Source code is also just a sequence of bytes - it's just a data file with a particular structure and intent. And then an executable is just a sequence of bytes - it's another data file, though on Unix it's one with the executable permission set. A compiler reads in a data file and writes a data file - they just happen to be source code and executable programs, respectively. (Now, when Lisp people say "code is data", they mean much more than compile time. For other languages, that is less true, other than self-modifying code...)
- platz 9y ago
  The hardware is more fundamental. (Unless you want to invoke mathematics e.g. term rewriting) The compiler/seq of bytes example requires von neumann, because in that case it is the behavior of the whole system that is self-modifying. You could not have the beautiful executable and interchangable files of bytes without von neumann.
  - AnimalMuppet 9y ago
    I don't think that's right. There have been systems that were Harvard architecture (Motorola 88000, I think, and definitely others). But the code to run on them still started out as a sequence of bytes in a text file, that is, as data, and a compiler still took that data file and wrote another data file that was the executable image.

DannyB2 9y ago

Code is data. In Java, I could use a bytecode manipulation library to read in a class file, manipulate it by using the library's API to add, modify, delete members of the class. Then write the class back out as a file. Code becomes data and is manipulated as date and written back out. In Lisp 'code is data' is so much more natural because code IS the most elementary data structures of the language. No need for a library to manipulate code behind an API. The API is the language primitives. The code is the data structure you manipulate using language primitives.

marknadal 9y ago

This is exactly what we mean as "code as data" and why it is so profound and has such exciting potential: vimeo.com/208899228/b9bc9eaaa4#t=13m50s "The Future of Programming and Databases" ^ JSRemote Conf / NodeJS Italy

whitten 9y ago

For me, "code as data" means that the code I write provides a particular structure (whether to create a report, or go through an editing task or find particular information in the computer). I then use a table-driven "data" approach to finding the particulars that are significant for this particular class of information. That might mean that I have the unique code stored in a table which I lookup using a relevant key, hence "table-driven code", or I have "setup code", "one iteration", and "teardown code" accessible through an indirect call because the structure of the code to do the task can stay the same. It is also common to have a general "event" fire at the end of the code to inform any other programs that need to know this data structure or real-world event has been accounted for. I think this approach is amentable to better testing, and can be viewed as a higher-level abstraction similar to LISP macros. (Even though my language of choice - MUMPS) doesn't have LISP-style syntactic macros.

err4nt 9y ago

That example was great - I've been inspired by LISP and have been playing around with my own idea of an HTML-building set of functions already this week: http://staticresource.com/html-js-4.html http://staticresource.com/html-js-4.html It's great to see such a relevant example :D

sbov 9y ago

As someone who uses Clojure for only side projects, macros seem to get a lot of attention, both good and bad, for something I very rarely write. Maybe I'm missing something though, and programmers using these languages professionally resort to them more often than I do.

dustingetz 9y ago
code-is-data also enables structural editing–instead of typing strings we can transform data–, there are gifs of that here: https://cursive-ide.com/userguide/paredit.html https://cursive-ide.com/userguide/paredit.html

didibus 9y ago

In Java, you have code as string. That is, your code is represented using a big string. In Lisp, your code is represented using the list data structure. So its really code-as-data-structure. A string is difficult to parse and modify, inserting things in the middle, removing elements, changing the order of the words, that's all really difficult with a string. So if you want to transform Java code, its going to be hard and error prone. A list is easy to manipulate in contrast. Inserting elements in the middle is trivial, so are deletes and swaps. So in Lisp, if you want to transform code its pretty easy. Meta-programming is when you write a program that writes a program. An example is say you want to you wanted to add a semi-colon at the end of all your lines of code. You need a macro to do it for you. A macro is a program that acts upon your code to transform it. Eclipse has them. So now its really easy to add a semi-colon at the end of each line. What if you wanted to add a comma between all words in a selection? Now its trickier if your macro operates over a big string, you might need a regex for example. This is meta programming though. Instead of adding the commas yourself, which are required for the program to run, you write another program to add them for you. Now if all the words were elements in a list, that macro would be a lot easier to write. This is in essence what code-as-data(structure) means. Its in contrast with code-as-string. You don't have to choose lists as your datastructure either, as long as its something that allows you to represent a turing complete program and is easy to manipulate. Now, homoiconicity is the fact that your text of code looks like a data-structure too. Making it trivial to parse it into one. So back to my example, you could parse the java code string into a list, and then add commas, and then convert it back to a string. But Java code doesn't map logically into a list. Some construct don't nest like lists, and how do you define what goes into each node of the list? Do you group public and String together? In Lisp, the syntax is an unambiguous AST already, its thus trivial to parse into a list of lists. So you can easily get that AST you need to easily add commas where it make sense. Finally, there's a third aspect. Code-as-data also implies that your language can accept code as argument in the form of raw data. The best way to think of it is, how would you send a function over the wire to a program and have that program run the code I sent? You need a way to serialize that function, which is code, into raw data that can be transmitted over the wire. The receiving program doesn't have that function defined, so it needs to know how to deserialize it, but also at runtime it must be able to take this raw data, which represents code, and be able to parse it, compile/interpet it and execute it. Think of SQL, SQL is often used in a code-as-data way. You want "select * from %s". Now you'd take this as a string, and you'd use a string replace, and replace %s with something the user picked in a drop down. At runtime, you are dynamically creating the SQL code, and once you have it, running it. You might have methods that accept SQL and return SQL. Now again, your SQL is a big string, which isn't ideal. But this is still an example of code as data. Now in Java, you can not do that with Java code. Java does not have this concept of code-as-data. In other words, there's no eval. So when you combine all these three aspects, a homoiconic syntax that parses easily and logically into a data-structure which is easy to manipulate, and where you can then execute data which represent code at runtime and pass it around to other functions, even over the wire, you get a very powerful combo that turns into a Meta-programming powerhouse. This is the strength of Lisps, the one strength all Lisps share.

dogfishbar 9y ago

M-LISP: a representation-independent dialect of LISP with reduction semantics In this paper we introduce M-LISP, a dialect of LISP designed with an eye toward reconciling LISP's metalinguistic power with the structural style of operational semantics advocated by Plotkin [28]. We begin by reviewing the original definition of LISP [20] in an attempt to clarify the source of its metalinguistic power. We find that it arises from a problematic clause in this definition. We then define the abstract syntax and operational semantics of M-LISP, essentially a hybrid of M-expression LISP and Scheme. Next, we tie the operational semantics to the corresponding equational logic. As usual, provable equality in the logic implies operational equality. Having established this framework we then extend M-LISP with the metalinguistic eval and reify operators (the latter is a nonstrict operator that converts its argument to its metalanguage representation). These operators encapsulate the metalinguistic representation conversions that occur globally in S-expression LISP. We show that the naive versions of these operators render LISP's equational logic inconsistent. On the positive side, we show that a naturally restricted form of the eval operator is confluent and therefore a conservative extension of M-LISP. Unfortunately, we must weaken the logic considerably to obtain a consistent theory of reification.