6 ms·

Guile-lips: Scheme as a generic macro language

40 points by ruste 10y ago

nothrabannosir 10y ago
<20 lines of parensoup> > How easy was that!!? ;) The problem comes mostly from lisp's dislike for text (my personal opinion). Lisps work best when everything is a list, always. As soon as you cross the boundary into... Well.. Anything else , really, you need a little bit of ugly glue. That's why it's so ugly, in case anyone was about to make the "ermagerd if lisp is so great explain this..." comment. :) [edit: which, btw, is not to lisp's credit, imho.]
- ruste 10y ago
  I completely agree. This is actually better than a lot of cases because Java import statements are just lisp symbols delimited by periods. This is mostly only useful if you're transforming lisp statements into code. Transforming java to java or c to c is just a mess. Still, If I need to do some ugly code transformation, I'd rather it be in scheme than c. :) Also, the example could and probably should be rewritten more cleanly. Hence the comment. That was the first thing I used the tool for and my scheme has improved since.
- duaneb 10y ago
  I don't get what you're suggesting. Add a hash type? The syntax isn't much different from a list, and sometimes it is a macro on top of existing lists. In other words, what is a concrete example of list syntax being inadequate? Literally every data format of which I can think can be expressed as a list or a derivative of lists (e.g. association lists).
- TheLarch 10y ago
  How is string processing worse or uglier in lisp than other languages?
PaulHoule 10y ago
Ouch! I never write those import statements by hand anyway, that's one of the many reasons I have an IDE.
- codemac 10y ago
  Meta-programming via your editor doing code-gen for you or lisp doing code-gen for you end with about the same code.
  - PaulHoule 10y ago
    Yeah, except IntelliJ Idea is a refined product that does that and 100 other things. I don't know what IntelliJ has inside of it, but I know Eclipse is based on EJC, the Eclipse Java Compiler, which can turn Java code into a Java abstract source tree, and then turn that either back to Java code or to byte code. It's not too different from the read function in LISP but it is a bit more complex...
agumonkey 10y ago
I don't like the style though. I'd write an accumulative tree traversal composed with a string formatting final step. ;; :: tree -> [flat-qualified-name] ;; :: flat-qualified-name -> string
- ruste 10y ago
  That does sound better. Build up the structure you want then convert to a string.
junke 10y ago
For those saying that the code is too verbose, here is another version in Common Lisp: (defun jim (arg &optional path) (etypecase arg (string (format t "import ~{~(~A~).~}~A;~%" (reverse path) arg)) (list (push (pop arg) path) (dolist (a arg) (jim a path))))) Test: (jim '(java (util "HashMap" "HashSet" "Map" "Random" "Set" "UUID" (concurrent "ExecutorService" "Executors" "Callable" "Future")) (awt "Color"))) Gives: import java.util.HashMap; import java.util.HashSet; import java.util.Map; import java.util.Random; import java.util.Set; import java.util.UUID; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Callable; import java.util.concurrent.Future; import java.awt.Color;
- ruste 10y ago
  If you'd prefer common lisp you should check out the project that inspired mine: https://github.com/zc1036/lips https://github.com/zc1036/lips
  - junke 10y ago
    You did a great job and I am sorry that the discussion is mostly about the example. I saw the Scheme version and found it quite verbose, like some other people on this thread. That's what made me want to implement a CL one. Maybe if you had a less controversial example where you produce e.g. Markdown text, it would not distract people from the tool.
    - ruste 10y ago
      It was verbose. I actually left a comment in the example about how it could have been written better. Someone else further down in the comments suggested a nicer way of writing it. I think most people got the idea though! Thanks for the feedback. :)
agentgt 10y ago
I think many will agree that the import statement example is fairly weak. If you are writing import statements by hand you should just seriously consider the benefits of using an IDE. That being said I use code generation instead of the IDE all the time but for harder things like generating fluent builders , copying objects, generating visitor pattern for pattern matching etc. What I do is write Groovy scripts and will either use Groovy GString templates or Squares Poet to generate code. What I do though instead of making some sort of DSL is use a subset of the language itself to help the code generator. For example copying a list of private fields I can quickly turn into either an immutable object with builder or an interface or just a plain object copy etc etc. I put my Groovy scripts in my ~/bin and then just keep my terminal running. I then copy from Eclipse/Intellij some code and then run: pbpaste | SomeGroovyTransformScript.groovy | pbcopy Then paste back the code into the IDE. Ideally I would use APT libraries (and in same cases I have converted Groovy code into APT libraries) but some times I just want something lightweight.
- outworlder 10y ago
  > I think many will agree that the import statement example is fairly weak. If you are writing import statements by hand you should just seriously consider the benefits of using an IDE. Agreed that the example is weak. I do agree that, for import statements, having and IDE fill those in for you is fair game. It can figure out what you are trying to use and offer to import those, that's fine. The problem here is the slippery slope. You get an IDE writing import statements. Than it will write getters and setters for you – which is already an aberration that prevents language evolution. For instance, C# has a syntax for default getters and setters (and for properties, in general). Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet). It's amazing the amount of machinery that gets created when people do not have access to a useful macro system.
  - ruste 10y ago
    I probably should have presented more than one example in my post. A lot of people are focusing on the example I presented and not the general utility. Live and learn I guess. :/
TheLarch 10y ago
In the case of Java you can use Clojure or Armed Bear Common Lisp anyway.
chriswarbo 10y ago
I really like the idea of a general-purpose text-replacement pre-processor like this. I especially like: - A "real" programming language (Scheme) - Only 1 character to avoid/escape (~) - Expression-oriented (as opposed to, for example, impure echo/print) - Completely orthogonal/independent of the code being manipulated (unlike the myriad Java/JVM examples cited by other comments) One thing that would be nice is to distinguish between pure expressions and impure ones. That way, pure expressions can always be processed automatically (e.g. in wrapper scripts around tools like static analysers) since they're incapable of performing any effects. Impure expressions would still be useful, e.g. to scan filenames in the current directory, but can be identified as such and skipped by tools/scripts.
- ruste 10y ago
  Hmm, I think it's a dangerous thing to be skipping portions of code in analysis just because they're impure. I think you'd be better off just taking the code produced after processing and running that through your static analyzer as is. That's what's going to be running in the end anyway. Maybe I'm not fully understanding the problem though.
  - chriswarbo 10y ago
    Well, you could choose to abort rather than skip anything. The problem is that impure code can do dangerous things (depending on which i/o primitives are provided). For example, a macro might create and delete a bunch of temporary files; this might work fine in one scenario (e.g. during compilation) but misbehave during another (e.g. static analysis). In which case "static analysis" would no longer be static.
Arelius 10y ago
Very cool! I had thrown together something very similar using s7 for a super small self-contained preprocessor. I'm using `@` as the symbol to consume the next-sexp and process it, otherwise it's mostly the same. I'll try to throw together some documentation, examples and a LICENSE in case anyone wants to use it. But I've just been using it internally as a generator for Ninja files for my builds. https://github.com/Arelius/s7pp https://github.com/Arelius/s7pp
Johnny_Brahms 10y ago
We have this at work, actually. I wrote a macro language for pascal in guile. At first it was a first draft, but it worked well enough and got to stay. It generates enormous amounts of code that would otherwise have been unmaintainable. As it is now, it is just a couple of hundreds of lines of scheme and pascal code that recursively generates pascal. We had some problems making it scale (as I said, it generates a lot of code), but the upcoming 2.2 guile solved most of our issues, and I rewrote parts of it to make it fast enough. Got it down from 1 minute to 10 seconds on a 10+ minute build when rebuilding everything. I might be able to opensource it. With minor tweaks, it can probably be made a bit more language-agnostic.
zeveb 10y ago
Here's the original lips on which this is based: https://github.com/zc1036/lips https://github.com/zc1036/lips Maybe I'll use this to hack generics into Go …
- ruste 10y ago
  Are you the author? I tried to find a way to get in contact with him, but github doesn't have a messaging feature and I couldn't find an email.
  - zeveb 10y ago
    Nope, I'm not. Good luck finding him — maybe you could open an issue? Bit of an abuse of the system, but it would work.
    - ruste 10y ago
      I considered that. I may do it yet.

40 points by ruste 10y ago

<20 lines of parensoup> > How easy was that!!? ;) The problem comes mostly from lisp's dislike for text (my personal opinion). Lisps work best when everything is a list, always. As soon as you cross the boundary into... Well.. Anything else , really, you need a little bit of ugly glue. That's why it's so ugly, in case anyone was about to make the "ermagerd if lisp is so great explain this..." comment. :) [edit: which, btw, is not to lisp's credit, imho.]

ruste 10y ago
I completely agree. This is actually better than a lot of cases because Java import statements are just lisp symbols delimited by periods. This is mostly only useful if you're transforming lisp statements into code. Transforming java to java or c to c is just a mess. Still, If I need to do some ugly code transformation, I'd rather it be in scheme than c. :) Also, the example could and probably should be rewritten more cleanly. Hence the comment. That was the first thing I used the tool for and my scheme has improved since.
duaneb 10y ago
I don't get what you're suggesting. Add a hash type? The syntax isn't much different from a list, and sometimes it is a macro on top of existing lists. In other words, what is a concrete example of list syntax being inadequate? Literally every data format of which I can think can be expressed as a list or a derivative of lists (e.g. association lists).
TheLarch 10y ago
How is string processing worse or uglier in lisp than other languages?

PaulHoule 10y ago

Ouch! I never write those import statements by hand anyway, that's one of the many reasons I have an IDE.

codemac 10y ago
Meta-programming via your editor doing code-gen for you or lisp doing code-gen for you end with about the same code.
- PaulHoule 10y ago
  Yeah, except IntelliJ Idea is a refined product that does that and 100 other things. I don't know what IntelliJ has inside of it, but I know Eclipse is based on EJC, the Eclipse Java Compiler, which can turn Java code into a Java abstract source tree, and then turn that either back to Java code or to byte code. It's not too different from the read function in LISP but it is a bit more complex...

agumonkey 10y ago

I don't like the style though. I'd write an accumulative tree traversal composed with a string formatting final step. ;; :: tree -> [flat-qualified-name] ;; :: flat-qualified-name -> string

ruste 10y ago
That does sound better. Build up the structure you want then convert to a string.

junke 10y ago

For those saying that the code is too verbose, here is another version in Common Lisp: (defun jim (arg &optional path) (etypecase arg (string (format t "import ~{~(~A~).~}~A;~%" (reverse path) arg)) (list (push (pop arg) path) (dolist (a arg) (jim a path))))) Test: (jim '(java (util "HashMap" "HashSet" "Map" "Random" "Set" "UUID" (concurrent "ExecutorService" "Executors" "Callable" "Future")) (awt "Color"))) Gives: import java.util.HashMap; import java.util.HashSet; import java.util.Map; import java.util.Random; import java.util.Set; import java.util.UUID; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Callable; import java.util.concurrent.Future; import java.awt.Color;

ruste 10y ago
If you'd prefer common lisp you should check out the project that inspired mine: https://github.com/zc1036/lips https://github.com/zc1036/lips
- junke 10y ago
  You did a great job and I am sorry that the discussion is mostly about the example. I saw the Scheme version and found it quite verbose, like some other people on this thread. That's what made me want to implement a CL one. Maybe if you had a less controversial example where you produce e.g. Markdown text, it would not distract people from the tool.
  - ruste 10y ago
    It was verbose. I actually left a comment in the example about how it could have been written better. Someone else further down in the comments suggested a nicer way of writing it. I think most people got the idea though! Thanks for the feedback. :)

agentgt 10y ago

I think many will agree that the import statement example is fairly weak. If you are writing import statements by hand you should just seriously consider the benefits of using an IDE. That being said I use code generation instead of the IDE all the time but for harder things like generating fluent builders , copying objects, generating visitor pattern for pattern matching etc. What I do is write Groovy scripts and will either use Groovy GString templates or Squares Poet to generate code. What I do though instead of making some sort of DSL is use a subset of the language itself to help the code generator. For example copying a list of private fields I can quickly turn into either an immutable object with builder or an interface or just a plain object copy etc etc. I put my Groovy scripts in my ~/bin and then just keep my terminal running. I then copy from Eclipse/Intellij some code and then run: pbpaste | SomeGroovyTransformScript.groovy | pbcopy Then paste back the code into the IDE. Ideally I would use APT libraries (and in same cases I have converted Groovy code into APT libraries) but some times I just want something lightweight.

outworlder 10y ago
> I think many will agree that the import statement example is fairly weak. If you are writing import statements by hand you should just seriously consider the benefits of using an IDE. Agreed that the example is weak. I do agree that, for import statements, having and IDE fill those in for you is fair game. It can figure out what you are trying to use and offer to import those, that's fine. The problem here is the slippery slope. You get an IDE writing import statements. Than it will write getters and setters for you – which is already an aberration that prevents language evolution. For instance, C# has a syntax for default getters and setters (and for properties, in general). Then you get IDE plugins for writing boilerplate (for, say, JEE). Then you end up with a XML preprocessing pass that writes Java code (XDoclet). It's amazing the amount of machinery that gets created when people do not have access to a useful macro system.
- ruste 10y ago
  I probably should have presented more than one example in my post. A lot of people are focusing on the example I presented and not the general utility. Live and learn I guess. :/

TheLarch 10y ago

In the case of Java you can use Clojure or Armed Bear Common Lisp anyway.

chriswarbo 10y ago

I really like the idea of a general-purpose text-replacement pre-processor like this. I especially like: - A "real" programming language (Scheme) - Only 1 character to avoid/escape (~) - Expression-oriented (as opposed to, for example, impure echo/print) - Completely orthogonal/independent of the code being manipulated (unlike the myriad Java/JVM examples cited by other comments) One thing that would be nice is to distinguish between pure expressions and impure ones. That way, pure expressions can always be processed automatically (e.g. in wrapper scripts around tools like static analysers) since they're incapable of performing any effects. Impure expressions would still be useful, e.g. to scan filenames in the current directory, but can be identified as such and skipped by tools/scripts.

ruste 10y ago
Hmm, I think it's a dangerous thing to be skipping portions of code in analysis just because they're impure. I think you'd be better off just taking the code produced after processing and running that through your static analyzer as is. That's what's going to be running in the end anyway. Maybe I'm not fully understanding the problem though.
- chriswarbo 10y ago
  Well, you could choose to abort rather than skip anything. The problem is that impure code can do dangerous things (depending on which i/o primitives are provided). For example, a macro might create and delete a bunch of temporary files; this might work fine in one scenario (e.g. during compilation) but misbehave during another (e.g. static analysis). In which case "static analysis" would no longer be static.

Arelius 10y ago

Very cool! I had thrown together something very similar using s7 for a super small self-contained preprocessor. I'm using `@` as the symbol to consume the next-sexp and process it, otherwise it's mostly the same. I'll try to throw together some documentation, examples and a LICENSE in case anyone wants to use it. But I've just been using it internally as a generator for Ninja files for my builds. https://github.com/Arelius/s7pp https://github.com/Arelius/s7pp

Johnny_Brahms 10y ago

We have this at work, actually. I wrote a macro language for pascal in guile. At first it was a first draft, but it worked well enough and got to stay. It generates enormous amounts of code that would otherwise have been unmaintainable. As it is now, it is just a couple of hundreds of lines of scheme and pascal code that recursively generates pascal. We had some problems making it scale (as I said, it generates a lot of code), but the upcoming 2.2 guile solved most of our issues, and I rewrote parts of it to make it fast enough. Got it down from 1 minute to 10 seconds on a 10+ minute build when rebuilding everything. I might be able to opensource it. With minor tweaks, it can probably be made a bit more language-agnostic.

zeveb 10y ago

Here's the original lips on which this is based: https://github.com/zc1036/lips https://github.com/zc1036/lips Maybe I'll use this to hack generics into Go …

ruste 10y ago
Are you the author? I tried to find a way to get in contact with him, but github doesn't have a messaging feature and I couldn't find an email.
- zeveb 10y ago
  Nope, I'm not. Good luck finding him — maybe you could open an issue? Bit of an abuse of the system, but it would work.
  - ruste 10y ago
    I considered that. I may do it yet.