5 ms·

Lispy - Code-as-data in Ruby, without the metaprogramming madness.

90 points by chrislloyd 15y ago

bguthrie 15y ago
I really, really like this idea. It's difficult enough to employ Ruby's metaprogramming constructs without making a mess, so the idea of instantly decomposing into nested lists is really appealing. I realize the library is only, like, three days old, but questions/feedback: - Do the DSL methods get turned into into proper Ruby methods eventually? I'd hate to see this whole thing built on the back of method_missing. - Picking apart nested arrays can lead to its own kind of spaghetti code. When people build DSLs, I recommend they immediately turn their high-level method calls into nicely testable objects. Can we see an example of that here? - The author specifically calls out libraries like ActiveRecord, which would suggest that the primary use case for this library is for integration as a DSL layer into your own libraries, but he doesn't provide any examples of that usage--just Lispy.new. I'd like a way to extend Lispy into my own class or module and get that same functionality.
- ryan-allen 15y ago
  G'day, Thanks for the feedback! I was hoping to do a little more work on it before it being posted to somewhere like HN, but oh well! A lot of the work I've done with this kind of Ruby, it often only is run once, so I tend to use method_missing. The reason why you'd want it converted to methods is for performance, right? I agree with your testing comments, there's a lot left out so far, specifically a guide on how you might use such a library, I'm working on that. And yes, you're right about AR being not the best example, it's use is for what you've described. Integration with your own code would be an ideal use. Implementing it as a module would be trivial.
  - pi18n 15y ago
    Is there a specific reason you chose to do arrays of arrays instead of hashes for subelements? I'd want the lists to come out as (maybe using a pluralizer or maybe not) :items => [{:priority => :normal, :desc => '...'}, ...] Then you could do more intuitive things like todo_list.items.sort{|a,b| a[:priority] <=> b[:priority] } which is somewhat stupid for symbols but not for numbers.
    - ryan-allen 15y ago
      No, no reason why I chose arrays, it's just a first cut. I was thinking of converting the representation for objects as the meaning of the three possible elements isn't explicit at the moment.
      - alttab 15y ago
        One thing to keep in mind is different Ruby implementations, versions, and platforms run differently. An array is currently the safest way to maintain order when it matters. If order didn't matter, than a hash would be fine, but iterating over the data could cause unwanted behavior if order is important and hash keys are served to an iterator in a different order than was laid out by the DSL code.
  - bguthrie 15y ago
    Performance and improved introspection. Once they're first-class they get included in that class's list of methods, which is helpful sometimes. Mostly I just like to avoid m_m when I can. AR is actually an okay example - it's as good an illustration as any of a place that employs these kinds of techniques, and the source of a lot of complexity in Rails.
sleight42 15y ago
The man has a point. We should consider using sexps more often when developing internal DSLs. That's not to say (stupid iPad) that we dont still need define_method and instance_eval; however, decoupling the language definition/API from the language impl is good sauce.
raganwald 15y ago
I like it, and I speak as someone who is slowly recovering from a year in a sanatorium gibbering about how I killed Mozart by using a library to decompile Ruby into sexps, then rearranged the sexps before reconstructing Ruby code: https://github.com/raganwald/rewrite_rails https://github.com/raganwald/rewrite_rails I'd still be in a padded cell if I hadn't run into a doctor who eschewed the more complex forms of therapy and simply told me to stop doing that.
extension 15y ago
I called this a Domain Agnostic Language: http://code.extension.ws/post/169602795/dal-rb http://code.extension.ws/post/169602795/dal-rb I don't actually like the approach though. Ruby makes DSLs as easy as "normal" interfaces. Why not take advantage of that? Generating and parsing an AST is exactly the kind of extra complexity that Ruby lets you avoid.
- raganwald 15y ago
  Here's a use case: Imagine that you want a form validation library, something like active record's domain-specific language, but it is not attached to models, just to forms being submitted. But anyhow, you write your DSL. Now you have two options: 1. Execute it directly in the controller of your Ruby server 2. Convert it to sexps, export those as JSON, and write a Javascript validator that validates forms on the client-side.
  - extension 15y ago
    That is indeed a nifty use case, but also kind of a special case in which a serialized AST is the required output of your program. For the general case where the DSL is just a sexy API, I wouldn't recommend building a parse tree. EDIT: Actually, for your case I don't think I would use sexps. I would write some primitive validators in Ruby that can also generate equivalent JavaScript code, then use those to build higher order validators in Ruby alone. Compare that to maintaining complete validator engines in two different languages.
- regularfry 15y ago
  One advantage of this approach is that it lets you not care about the order of definitions if you don't want to. Doing things the usual way, ruby's execution model can have you tied in knots if you're not careful.
jamesbritt 15y ago
How does this compare to the 'sexp' library? http://www.artima.com/rubycs/articles/patterns_sexp_dslsP.html http://www.artima.com/rubycs/articles/patterns_sexp_dslsP.ht...
JonnieCache 15y ago
Ruby metaprogramming hell (it is a real place). But meta-sin is so much fun! Nice lib with a lot of potential. It will definitely come in handy and may well prompt me to take some unfinished projects off the back burner and get them in the oven as it were. Also, I enjoy your README writing style. It walks the fine line between amusing irreverence and technical specificity very well.
- ryan-allen 15y ago
  I'm glad you enjoyed the README, I had fun writing it (aided by some Jameson).
- jamesbritt 15y ago
  Also, I enjoy your README writing style. It walks the fine line between amusing irreverence and technical specificity very well. I found the README distracting and hard to follow. I'm still unclear just what the library does or why one might need it, possibly because of an over-emphasis on jokes over technical detail and sufficient examples. However, there seems to be an under-current of "Ruby is hard to use correctly, so use this." Do people really find Ruby meta-programming so hard?
- bitwize 15y ago
  Ruby metaprogramming hell is a myth used to frighten school children into doing the dishes. C++ metaprogramming hell -- ohhhh, that's real.
jcapote 15y ago
1000 internets to this man. This is excellent!
zwp 15y ago
From the Examples: fart => [ :fart, [] ] fart 1 => [ :fart, 1 ] fart 1, 2 => [ :fart, [ 1, 2 ] ] Why isn't the single argument call: fart 1 => [ :fart, [ 1 ] ] ? ps. this is also fun: http://parsetree.rubyforge.org/ruby_parser/ http://parsetree.rubyforge.org/ruby_parser/
- dustmop 15y ago
  That's funny, I was expecting the opposite behavior: fart => [ :fart ] fart 1 => [ :fart, 1 ] fart 1, 2 => [ :fart, 1, 2 ] Seems more lispy to me, and from a quick glance I can't see where it would break.
  - zwp 15y ago
    Yes, I prefer this, definitely more lispy. I blame seeing too much of clojure's "defn fart [ args ]" syntax ;-)
KirinDave 15y ago
I'm a bit confused what this has to do with lisp. Seems more like a classic Smalltalk building pattern. If it's a reference to "code as data" from the Lisp tradition (which is a friendlier colloquialism for "homoiconicity"), then they bear only a tenuous relationship. Which is not to say that this library wouldn't be useful. But it is to say that it doesn't really bring any of the power of homoiconicity or consequential techniques thereof to Ruby.
deleted 15y ago
[deleted]
moron4hire 15y ago
"Code-as-data in Ruby, without the metaprogramming madness. — Read more http://www.youtube.com/watch?v=aD4bn5pp32w http://www.youtube.com/watch?v=aD4bn5pp32w link is a rickroll. What was the point of that? I don't think we need to be super serious about everything, but at least don't be anti-productive. I was hoping to have a link to a talk or screen-cast demonstrating what Lispy was about. Instead, backwards rickroll.
- ryan-allen 15y ago
  I'm sorry! I've updated the link to the RubyGem. I'm also working on a tutorial for intended usage, I'll post it when it's done.
- bluedanieru 15y ago
  It seems this hit the public consciousness a little earlier than the author expected is all. That might have just been a placeholder.
mardigan 15y ago
There was a recent ruby-talk announcement for live parse trees being available in 1.9 now. Presumably gives the functionality of ParseTree. http://quix.github.com/live_ast http://quix.github.com/live_ast
helium 15y ago
This is very close to something I built earlier this year...just nicer. https://github.com/michael-erasmus/Flippant https://github.com/michael-erasmus/Flippant
pavelludiq 15y ago
OT: I'm very tempted to be an asshole smug lisp weenie, and make some Greenspun's rule joke, but instead I'm going to try to be helpful. Ruby hackers pride themselves for being early adopters of new and cool technology, learning new skills, and libraries. What I've failed to notice is any interest in the old technologies, from which ruby got most of its good ideas(lisp, smalltalk). There is tremendous value in learning from the past(a lot of really good ideas didn't make it to the "cool kids languages" of today). This isn't specific to ruby, but its a good example. So my attempt at useful advice is: do learn lisp, instead of reinventing the wheel badly. Anyway, I've been thinking about this kind of stuff, and it seemed like a good opportunity to mention it.
- erikpukinskis 15y ago
  learn lisp, instead of reinventing the wheel badly When someone says something like this, my typical response is "That's true, I should dig more into Lisp. Oh, look! A bug in my software I need to fix. * forgets Lisp entirely *" If you were to say something like "the difference between Lispy and Lisp is that in Lisp you can do X", then my likelihood of actually going out and learning about X, pulling out the Lisp interpreter and playing around goes up about 1000x.
  - pavelludiq 15y ago
    Yes, finding the time and motivation to educate ourselves is sometimes hard, especially when we have to deal with other crap, I've been meaning to look into smalltalk for ages now, myself. If you want an example of something cool you can do with lisp, in principal the common lisp object system is built with macros on top of the non-object oriented lisp. Could i use Lispy to add a CLOS style object system on top of ruby? What about a prototype based one? This isn't a rhetorical question, I'm actually curious if ruby can do these kinds of things.
    - erikpukinskis 15y ago
      Could i use Lispy to add a CLOS style object system on top of ruby What features of CLOS are you interested in? I just whipped up some CLOS-style parameter-matching method invocation in Ruby: https://gist.github.com/926156 https://gist.github.com/926156 I'm not sure how you'd do that with Lispy. But it's definitely possible in Ruby. If you Google "prototype-based object model in Ruby" you get at least one solution: http://www.google.com/search?hl=en&q=prototype-based+object+in+ruby#sclient=psy&hl=en&safe=off&source=hp&q=prototype-based+object+model+in+ruby&aq=f&aqi=&aql=f&oq=&pbx=1&bav=on.2,or.r_gc.r_pw.&fp=3826ddc6bde629ca http://www.google.com/search?hl=en&q=prototype-based+obj...
samps 15y ago
This looks pretty cool, but as a non-Rubyist, I remain confused about the purpose of all these "DSL" libraries that lispy aims to simplify. In the examples, at least, the language was only used to declaratively build hierarchical list/dictionary structures. The only constructs I can see used are blocks and method invocations; there aren't any loops or conditionals or anything else imperative. Can someone more experienced with Ruby DSLs explain what lispy accomplishes that something like YAML or a simple custom parser doesn't? (Other than a bunch of extra "do"s and ":"s.)
- erikpukinskis 15y ago
  There aren't any loops or conditionals in the example, but since it's using blocks, there could be arbitrary code. But yeah, if you're not doing any of that you might as well just use YAML, or a hash.
kleiba 15y ago
Saying that, I've written about 10 lines of LISP in my life. Pete keeps telling me to go read SICP and I keep meaning to but then I end up at the local bar listening to some rock band and I'm like "crap, it's 3am". Man, you're such a rebel!!
soamv 15y ago
I see that lispy is using instance_exec -- which means that the scope of code inside that code is the lispy instance, which might not be what the programmer expects. For example if he does this inside a method he'd find his instance variables missing. (Is that right, or am I misreading the code?) You could use some sort of global or thread local variable to maintain the context, and then you wouldn't lose the instance scope, but then this code will not mix well with yield (and continuations and fibers etc.) So then it looks like the only way to solve this problem really neatly is in a language that has macros. (I made some attempts at this problem too, at https://github.com/soam/blox https://github.com/soam/blox . It is slightly more complex than lispy; see for example https://github.com/soam/blox/tree/master/samples/webservice https://github.com/soam/blox/tree/master/samples/webservice .)

90 points by chrislloyd 15y ago

I really, really like this idea. It's difficult enough to employ Ruby's metaprogramming constructs without making a mess, so the idea of instantly decomposing into nested lists is really appealing. I realize the library is only, like, three days old, but questions/feedback: - Do the DSL methods get turned into into proper Ruby methods eventually? I'd hate to see this whole thing built on the back of method_missing. - Picking apart nested arrays can lead to its own kind of spaghetti code. When people build DSLs, I recommend they immediately turn their high-level method calls into nicely testable objects. Can we see an example of that here? - The author specifically calls out libraries like ActiveRecord, which would suggest that the primary use case for this library is for integration as a DSL layer into your own libraries, but he doesn't provide any examples of that usage--just Lispy.new. I'd like a way to extend Lispy into my own class or module and get that same functionality.

ryan-allen 15y ago
G'day, Thanks for the feedback! I was hoping to do a little more work on it before it being posted to somewhere like HN, but oh well! A lot of the work I've done with this kind of Ruby, it often only is run once, so I tend to use method_missing. The reason why you'd want it converted to methods is for performance, right? I agree with your testing comments, there's a lot left out so far, specifically a guide on how you might use such a library, I'm working on that. And yes, you're right about AR being not the best example, it's use is for what you've described. Integration with your own code would be an ideal use. Implementing it as a module would be trivial.
- pi18n 15y ago
  Is there a specific reason you chose to do arrays of arrays instead of hashes for subelements? I'd want the lists to come out as (maybe using a pluralizer or maybe not) :items => [{:priority => :normal, :desc => '...'}, ...] Then you could do more intuitive things like todo_list.items.sort{|a,b| a[:priority] <=> b[:priority] } which is somewhat stupid for symbols but not for numbers.
  - ryan-allen 15y ago
    No, no reason why I chose arrays, it's just a first cut. I was thinking of converting the representation for objects as the meaning of the three possible elements isn't explicit at the moment.
    - alttab 15y ago
      One thing to keep in mind is different Ruby implementations, versions, and platforms run differently. An array is currently the safest way to maintain order when it matters. If order didn't matter, than a hash would be fine, but iterating over the data could cause unwanted behavior if order is important and hash keys are served to an iterator in a different order than was laid out by the DSL code.
- bguthrie 15y ago
  Performance and improved introspection. Once they're first-class they get included in that class's list of methods, which is helpful sometimes. Mostly I just like to avoid m_m when I can. AR is actually an okay example - it's as good an illustration as any of a place that employs these kinds of techniques, and the source of a lot of complexity in Rails.

sleight42 15y ago

The man has a point. We should consider using sexps more often when developing internal DSLs. That's not to say (stupid iPad) that we dont still need define_method and instance_eval; however, decoupling the language definition/API from the language impl is good sauce.

raganwald 15y ago

I like it, and I speak as someone who is slowly recovering from a year in a sanatorium gibbering about how I killed Mozart by using a library to decompile Ruby into sexps, then rearranged the sexps before reconstructing Ruby code: https://github.com/raganwald/rewrite_rails https://github.com/raganwald/rewrite_rails I'd still be in a padded cell if I hadn't run into a doctor who eschewed the more complex forms of therapy and simply told me to stop doing that.

extension 15y ago

I called this a Domain Agnostic Language: http://code.extension.ws/post/169602795/dal-rb http://code.extension.ws/post/169602795/dal-rb I don't actually like the approach though. Ruby makes DSLs as easy as "normal" interfaces. Why not take advantage of that? Generating and parsing an AST is exactly the kind of extra complexity that Ruby lets you avoid.

raganwald 15y ago
Here's a use case: Imagine that you want a form validation library, something like active record's domain-specific language, but it is not attached to models, just to forms being submitted. But anyhow, you write your DSL. Now you have two options: 1. Execute it directly in the controller of your Ruby server 2. Convert it to sexps, export those as JSON, and write a Javascript validator that validates forms on the client-side.
- extension 15y ago
  That is indeed a nifty use case, but also kind of a special case in which a serialized AST is the required output of your program. For the general case where the DSL is just a sexy API, I wouldn't recommend building a parse tree. EDIT: Actually, for your case I don't think I would use sexps. I would write some primitive validators in Ruby that can also generate equivalent JavaScript code, then use those to build higher order validators in Ruby alone. Compare that to maintaining complete validator engines in two different languages.
regularfry 15y ago
One advantage of this approach is that it lets you not care about the order of definitions if you don't want to. Doing things the usual way, ruby's execution model can have you tied in knots if you're not careful.

jamesbritt 15y ago

How does this compare to the 'sexp' library? http://www.artima.com/rubycs/articles/patterns_sexp_dslsP.html http://www.artima.com/rubycs/articles/patterns_sexp_dslsP.ht...

JonnieCache 15y ago

Ruby metaprogramming hell (it is a real place). But meta-sin is so much fun! Nice lib with a lot of potential. It will definitely come in handy and may well prompt me to take some unfinished projects off the back burner and get them in the oven as it were. Also, I enjoy your README writing style. It walks the fine line between amusing irreverence and technical specificity very well.

ryan-allen 15y ago
I'm glad you enjoyed the README, I had fun writing it (aided by some Jameson).
jamesbritt 15y ago
Also, I enjoy your README writing style. It walks the fine line between amusing irreverence and technical specificity very well. I found the README distracting and hard to follow. I'm still unclear just what the library does or why one might need it, possibly because of an over-emphasis on jokes over technical detail and sufficient examples. However, there seems to be an under-current of "Ruby is hard to use correctly, so use this." Do people really find Ruby meta-programming so hard?
bitwize 15y ago
Ruby metaprogramming hell is a myth used to frighten school children into doing the dishes. C++ metaprogramming hell -- ohhhh, that's real.

jcapote 15y ago

1000 internets to this man. This is excellent!

zwp 15y ago

From the Examples: fart => [ :fart, [] ] fart 1 => [ :fart, 1 ] fart 1, 2 => [ :fart, [ 1, 2 ] ] Why isn't the single argument call: fart 1 => [ :fart, [ 1 ] ] ? ps. this is also fun: http://parsetree.rubyforge.org/ruby_parser/ http://parsetree.rubyforge.org/ruby_parser/

dustmop 15y ago
That's funny, I was expecting the opposite behavior: fart => [ :fart ] fart 1 => [ :fart, 1 ] fart 1, 2 => [ :fart, 1, 2 ] Seems more lispy to me, and from a quick glance I can't see where it would break.
- zwp 15y ago
  Yes, I prefer this, definitely more lispy. I blame seeing too much of clojure's "defn fart [ args ]" syntax ;-)

KirinDave 15y ago

I'm a bit confused what this has to do with lisp. Seems more like a classic Smalltalk building pattern. If it's a reference to "code as data" from the Lisp tradition (which is a friendlier colloquialism for "homoiconicity"), then they bear only a tenuous relationship. Which is not to say that this library wouldn't be useful. But it is to say that it doesn't really bring any of the power of homoiconicity or consequential techniques thereof to Ruby.

deleted 15y ago

[deleted]

moron4hire 15y ago

"Code-as-data in Ruby, without the metaprogramming madness. — Read more http://www.youtube.com/watch?v=aD4bn5pp32w http://www.youtube.com/watch?v=aD4bn5pp32w link is a rickroll. What was the point of that? I don't think we need to be super serious about everything, but at least don't be anti-productive. I was hoping to have a link to a talk or screen-cast demonstrating what Lispy was about. Instead, backwards rickroll.

ryan-allen 15y ago
I'm sorry! I've updated the link to the RubyGem. I'm also working on a tutorial for intended usage, I'll post it when it's done.
bluedanieru 15y ago
It seems this hit the public consciousness a little earlier than the author expected is all. That might have just been a placeholder.

mardigan 15y ago

There was a recent ruby-talk announcement for live parse trees being available in 1.9 now. Presumably gives the functionality of ParseTree. http://quix.github.com/live_ast http://quix.github.com/live_ast

helium 15y ago

This is very close to something I built earlier this year...just nicer. https://github.com/michael-erasmus/Flippant https://github.com/michael-erasmus/Flippant

pavelludiq 15y ago

OT: I'm very tempted to be an asshole smug lisp weenie, and make some Greenspun's rule joke, but instead I'm going to try to be helpful. Ruby hackers pride themselves for being early adopters of new and cool technology, learning new skills, and libraries. What I've failed to notice is any interest in the old technologies, from which ruby got most of its good ideas(lisp, smalltalk). There is tremendous value in learning from the past(a lot of really good ideas didn't make it to the "cool kids languages" of today). This isn't specific to ruby, but its a good example. So my attempt at useful advice is: do learn lisp, instead of reinventing the wheel badly. Anyway, I've been thinking about this kind of stuff, and it seemed like a good opportunity to mention it.

erikpukinskis 15y ago
learn lisp, instead of reinventing the wheel badly When someone says something like this, my typical response is "That's true, I should dig more into Lisp. Oh, look! A bug in my software I need to fix. * forgets Lisp entirely *" If you were to say something like "the difference between Lispy and Lisp is that in Lisp you can do X", then my likelihood of actually going out and learning about X, pulling out the Lisp interpreter and playing around goes up about 1000x.
- pavelludiq 15y ago
  Yes, finding the time and motivation to educate ourselves is sometimes hard, especially when we have to deal with other crap, I've been meaning to look into smalltalk for ages now, myself. If you want an example of something cool you can do with lisp, in principal the common lisp object system is built with macros on top of the non-object oriented lisp. Could i use Lispy to add a CLOS style object system on top of ruby? What about a prototype based one? This isn't a rhetorical question, I'm actually curious if ruby can do these kinds of things.
  - erikpukinskis 15y ago
    Could i use Lispy to add a CLOS style object system on top of ruby What features of CLOS are you interested in? I just whipped up some CLOS-style parameter-matching method invocation in Ruby: https://gist.github.com/926156 https://gist.github.com/926156 I'm not sure how you'd do that with Lispy. But it's definitely possible in Ruby. If you Google "prototype-based object model in Ruby" you get at least one solution: http://www.google.com/search?hl=en&q=prototype-based+object+in+ruby#sclient=psy&hl=en&safe=off&source=hp&q=prototype-based+object+model+in+ruby&aq=f&aqi=&aql=f&oq=&pbx=1&bav=on.2,or.r_gc.r_pw.&fp=3826ddc6bde629ca http://www.google.com/search?hl=en&q=prototype-based+obj...

samps 15y ago

This looks pretty cool, but as a non-Rubyist, I remain confused about the purpose of all these "DSL" libraries that lispy aims to simplify. In the examples, at least, the language was only used to declaratively build hierarchical list/dictionary structures. The only constructs I can see used are blocks and method invocations; there aren't any loops or conditionals or anything else imperative. Can someone more experienced with Ruby DSLs explain what lispy accomplishes that something like YAML or a simple custom parser doesn't? (Other than a bunch of extra "do"s and ":"s.)

erikpukinskis 15y ago
There aren't any loops or conditionals in the example, but since it's using blocks, there could be arbitrary code. But yeah, if you're not doing any of that you might as well just use YAML, or a hash.

kleiba 15y ago

Saying that, I've written about 10 lines of LISP in my life. Pete keeps telling me to go read SICP and I keep meaning to but then I end up at the local bar listening to some rock band and I'm like "crap, it's 3am". Man, you're such a rebel!!

soamv 15y ago

I see that lispy is using instance_exec -- which means that the scope of code inside that code is the lispy instance, which might not be what the programmer expects. For example if he does this inside a method he'd find his instance variables missing. (Is that right, or am I misreading the code?) You could use some sort of global or thread local variable to maintain the context, and then you wouldn't lose the instance scope, but then this code will not mix well with yield (and continuations and fibers etc.) So then it looks like the only way to solve this problem really neatly is in a language that has macros. (I made some attempts at this problem too, at https://github.com/soam/blox https://github.com/soam/blox . It is slightly more complex than lispy; see for example https://github.com/soam/blox/tree/master/samples/webservice https://github.com/soam/blox/tree/master/samples/webservice .)