6 ms·

Beautiful showcase of Julia's metaprogramming features. For those lispers around, these string macros are used for arbitrary syntax, but if your desired syntax

Beautiful showcase of Julia's metaprogramming features. For those lispers around, these string macros are used for arbitrary syntax, but if your desired syntax can be parsed by julia's parser, then you can just use regular macros which operate like Lisp1 macros with hygiene being something you opt out of instead of into.

dan-robertson 7y ago
I found Julia’s macros to be pretty difficult. Unlike lisp’s, it’s not obvious what the (somewhat lisp like) ast of a given piece of code will be. That ast also frequently changed between language versions. It also wasn’t documented when I last checked (ages ago). I recall the parsing rules around quoting were particularly confusing. It also seemed there were multiple valid asts for some expressions which would sometimes be magically canonicalised but maybe I was mistaken about that. I still think there’s a lot to like about Julia.
- eigenspace 7y ago
  Yes, I found while I was learning the ropes of metaprogramming, I did a lot of quoting and dumping of code and then just reading the output structure to teach myself how a given piece of code get's lowered. Honestly, even if there were good docs for this, I think just quoting code at the REPL is a faster and more direct way to get that information anyways. This f course does increase the cognitive overhead when you're writing macros. I think this additional complexity is a bit unavoidable given that Julia aims to provide something more like mathy syntax. > t also seemed there were multiple valid asts for some expressions which would sometimes be magically canonicalised but maybe I was mistaken about that. Yes, there are some things like that that happen, and to introspect that process you can use the Meta.@lower macro to see the canonicalization.
  - dan-robertson 7y ago
    > I think this additional complexity is a bit unavoidable given that Julia aims to provide something more like mathy syntax. I’m not sure I agree with regard to the canonicalisation. But also there are generally a few ways to do macros in different languages: - Macros that run in a preprocessor. This can be quite structured with macros written into the source code file (eg C) or they may exist inside the preprocessor (eg camlp4 would have a modified ocaml+macros grammar and output pure ocaml after expanding the macros) - CL style reader macros/Perl style language extensions dynamically modify the syntax/grammar of the language in somewhat arbitrary ways. These may operate at a text level (eg CL) or a higher parse tree level. - CL style: macros look similar to function calls, expansion works in a well defined way and they manipulate a regular syntax tree which is obvious from the syntax - scheme/rust style macros which are rule-based and automatically hygienic. In scheme the rules are easy to write because the AST is obvious. In rust these are a little harder to write but the ast is interesting: macro calls are distinguishable from other syntax at parse time and their arguments are passed on as “trees of tokens with balanced brackets”, a bit like sexps but there are a few types of paren pair. This way the rest of the language syntax need not apply inside the arguments to a macro and one doesn’t need to know the structure of the ast - Julia style macros: you get an untyped ast that isn’t obvious from the written code and you can modify it unhygienically like in CL macros. - ocaml ppx/rust proc macros: you get a typed ast (ie one in which the types let you know all possible ways some syntax you’re interested in may be represented) with certain extension points that may be used to call macros, and you can unhygienically modify that ast. I think in the rust case your ast input may be in the “tree of tokens with balanced parens” form, and you may only be able to look at the ast related to the proc macro call that invokes you. In ocaml ppxes, they get the whole ast of the file and have to find the extension points they care about. There’s no tree-of-tokens in the ast and the extension points need to contain valid expressions/types. The ppx must remove any references to these extension points from the ast (replacing them with valid syntax) or compilation will fail.

dan-robertson 7y ago

I found Julia’s macros to be pretty difficult. Unlike lisp’s, it’s not obvious what the (somewhat lisp like) ast of a given piece of code will be. That ast also frequently changed between language versions. It also wasn’t documented when I last checked (ages ago). I recall the parsing rules around quoting were particularly confusing. It also seemed there were multiple valid asts for some expressions which would sometimes be magically canonicalised but maybe I was mistaken about that. I still think there’s a lot to like about Julia.

eigenspace 7y ago
Yes, I found while I was learning the ropes of metaprogramming, I did a lot of quoting and dumping of code and then just reading the output structure to teach myself how a given piece of code get's lowered. Honestly, even if there were good docs for this, I think just quoting code at the REPL is a faster and more direct way to get that information anyways. This f course does increase the cognitive overhead when you're writing macros. I think this additional complexity is a bit unavoidable given that Julia aims to provide something more like mathy syntax. > t also seemed there were multiple valid asts for some expressions which would sometimes be magically canonicalised but maybe I was mistaken about that. Yes, there are some things like that that happen, and to introspect that process you can use the Meta.@lower macro to see the canonicalization.
- dan-robertson 7y ago
  > I think this additional complexity is a bit unavoidable given that Julia aims to provide something more like mathy syntax. I’m not sure I agree with regard to the canonicalisation. But also there are generally a few ways to do macros in different languages: - Macros that run in a preprocessor. This can be quite structured with macros written into the source code file (eg C) or they may exist inside the preprocessor (eg camlp4 would have a modified ocaml+macros grammar and output pure ocaml after expanding the macros) - CL style reader macros/Perl style language extensions dynamically modify the syntax/grammar of the language in somewhat arbitrary ways. These may operate at a text level (eg CL) or a higher parse tree level. - CL style: macros look similar to function calls, expansion works in a well defined way and they manipulate a regular syntax tree which is obvious from the syntax - scheme/rust style macros which are rule-based and automatically hygienic. In scheme the rules are easy to write because the AST is obvious. In rust these are a little harder to write but the ast is interesting: macro calls are distinguishable from other syntax at parse time and their arguments are passed on as “trees of tokens with balanced brackets”, a bit like sexps but there are a few types of paren pair. This way the rest of the language syntax need not apply inside the arguments to a macro and one doesn’t need to know the structure of the ast - Julia style macros: you get an untyped ast that isn’t obvious from the written code and you can modify it unhygienically like in CL macros. - ocaml ppx/rust proc macros: you get a typed ast (ie one in which the types let you know all possible ways some syntax you’re interested in may be represented) with certain extension points that may be used to call macros, and you can unhygienically modify that ast. I think in the rust case your ast input may be in the “tree of tokens with balanced parens” form, and you may only be able to look at the ast related to the proc macro call that invokes you. In ocaml ppxes, they get the whole ast of the file and have to find the extension points they care about. There’s no tree-of-tokens in the ast and the extension points need to contain valid expressions/types. The ppx must remove any references to these extension points from the ast (replacing them with valid syntax) or compilation will fail.