7 ms·

Julia Macros for Beginners

127 points by IlyaOrson 5y ago

Kalanos 5y ago
never understood why this was better/ how it was different from regular functions. just seems like bugs/vulns waiting to happen
- medo-bear 5y ago
  no one is proposing macros as a paradigm of programming. they simply give programmers expressive powers not afforded by use of regular functions. in a sense you can compare macros to c++ templating. know what you are doing and use sparingly
- IlyaOrson 5y ago
  It is not necessarily better in all cases and should not be overused: https://youtu.be/mSgXWpvQEHE?t=579 https://youtu.be/mSgXWpvQEHE?t=579 However, it is useful to provide a nicer syntax and DSLs. Some examples: https://stackoverflow.com/questions/58137512/why-use-macros-in-julia https://stackoverflow.com/questions/58137512/why-use-macros-... https://www.juliafordatascience.com/animations-with-plots-jl/ https://www.juliafordatascience.com/animations-with-plots-jl... https://gist.github.com/MikeInnes/8299575 https://gist.github.com/MikeInnes/8299575
- bobbylarrybobby 5y ago
  Being able to operate on expressions before they reach the compiler is very handy. Consider how ergonomic testing is thanks to macros: https://docs.julialang.org/en/v1/stdlib/Test/ https://docs.julialang.org/en/v1/stdlib/Test/ Here's an example of passing quasi-json to a plotting function: https://www.queryverse.org/VegaLite.jl/stable/userguide/vlplotmacro/ https://www.queryverse.org/VegaLite.jl/stable/userguide/vlpl... . This lets you essentially transliterate a VegaLite spec into Julia without needing to translate it into Julia. Finally, macros that operator on dataframes let you write code that looks kind of like SQL, and is much more pleasant than working with functions: https://dataframes.juliadata.org/stable/man/querying_frameworks/ https://dataframes.juliadata.org/stable/man/querying_framewo...
- srcreigh 5y ago
  Some macro systems can create variables in a loop for you. You could make this macro, (define-all i 5 0) ;; creates i1 i2 i3 i4 i5 initialized to 0 That's somewhat impossible with functions. The closest you get is either an array/dict with only runtime error checking, or an external codegen program. I wrote a post [0] about how to do this in Racket. The macro generates ORM code based given a SQLite DB. Aka the compiler queries SQLite and generates table-column functions automatically. More potential benefits are: Better static error messages (can implement a type system using macros, example here[1]), and controlling execution order (can add lazy computation semantics). [0]: http://tech.perpetua.io/2022/01/generating-sqlite-bindings-with-racket-macros/ http://tech.perpetua.io/2022/01/generating-sqlite-bindings-w... [1]: https://gist.github.com/srcreigh/f341b2adaa0fe37c241fdf15f37bdd2c https://gist.github.com/srcreigh/f341b2adaa0fe37c241fdf15f37...
leephillips 5y ago
This is an excellent, clear introduction to a topic that’s not easy to explain to beginners.
phoe-krk 5y ago
As always, I'm very glad to see that structural, Common Lisp-style macro systems with the whole language available for macro construction, have been successfully adopted in other languages to the point where it's possible to explain them without a single mention of Lisp in the article, or even better - where a mention of Lisp anywhere else except for the very beginning would make the article worse by making a unnecessary detour. pg's article on the topic, "What Made Lisp Different", [0] has aged poorly, and points 8 and 9 it makes (a notation for code using trees of symbols and the whole language always available) are no longer Lisp-specific. The final point, about "inventing a new dialect of Lisp", doesn't hold true either - as seen here, Julia is doing just fine not claiming to be another dialect of Lisp, even though many sources mention directly that it's Lisp-inspired. Congrats to Julia people for the macro system and to the author for the article! [0] http://www.paulgraham.com/diff.html http://www.paulgraham.com/diff.html
- sischoel 5y ago
  One could argue that Julia is just Lisp with a syntax that appeals more to popular taste. There is also a secret option to get into a lisp repl in Julia "julia --lisp".
  - phoe-krk 5y ago
    Sure, that's always an option - same as Dylan, in a way!
    - borodi 5y ago
      Julia does take inspiration from Dylan and Scheme(the parser is even written in it) so it makes sense I guess
  - dfghdfhs 5y ago
    > There is also a secret option to get into a lisp repl in Julia "julia --lisp". Wtf....... what? I just tried it, it's true. Is this some easter egg?
tlinker123 5y ago
Julia macros are pretty legit. Right up there with Rust macros.
xiaodai 5y ago
There are R meta programming techniques called non standard evaluation. It has many similarities to the macro system in Julia
- krumbie 5y ago
  That's true. However, I believe that many R programmers don't know when non-standard evaluation happens or what it is exactly. Functions with or without it cannot be told apart just by looking at the syntax. While NSE enables the dplyr syntax that many people enjoy, for me it's too magic and I have trouble reasoning about variable names in other people's code.
  - coop_solution 5y ago
    What does dplyr syntax look like?
    - pdeffebach 5y ago
      Let's say you have a data frame df = tibble(a = c(1, 2)) and you want to use a dplyr verb to modify it mutate(df, b = a + 1) the `a` in the above expression refers to the column in `df`, but this means it's hard to reference a variable in the outer scope named `a`. Furthermore, if you have a string referring to the column name `"a"`, you can't simply write mutate(df, b = a_var + 1) Contrast this with DataFramesMeta.jl, which is a dply-like library for Julia, written with macros. df = DataFrame(a = [1, 2]) @transform df :b = :a .+ 1 Because of the use of Symbols, there is no ambiguity about scopes. To work with a variable referring to column `a` you can write a_str = "a" @transform df :b = $a_str .+ 1 I won't pretend this isn't more complicated or harder to learn. Some of the complexity is due to Julia's high performance limiting non-standard evaluation in subtle ways. But a core strength of Julia's macros is that it's easy to inspect these expressions and understand exactly what's going on, with `@macroexpand` as shown in the blog post. DataFramesMeta.jl repo: https://github.com/JuliaData/DataFramesMeta.jl https://github.com/JuliaData/DataFramesMeta.jl
      - 5y ago
tombert 5y ago
Macros were part of the "holy shit" moment for me for Lisp, in particular the Common Lisp Object System. I hadn't fully realized that it was possible to add a whole new paradigm to a language as a library [1], and moreover a particularly nice implementation of that paradigm. After that, I realized that macros aren't always something that needs to be avoided; in the right hands they're immensely powerful. I've only played a little with Julia macros, but it seems like they learned a lot of Lisps lessons, so I support it wholly. [1] I wasn't aware of how Objective C was built at the time.
dan-robertson 5y ago
The frustration of Julia macros for me was never knowing what AST would be produced for a given expression. This is a bit more manageable if you have a typed AST (e.g. OCaml but ppxes have other issues) or an obvious one (e.g. lisp). I like the way rust handles it where the macros operate on a tree of non-delimiter leaves and [delimiter, subtree list, delimiter] nodes which can allow for figuring out what the input to a macro will be more easily and for more varied macro syntax. Other languages that want a full AST before macros force the macro input to be a bit more AST-like, e.g. the Julia parser picks operator precedence and OCaml won’t let you use _ as an identifier. Maybe it is better now but when I looked at macros ~5 years ago some language update changed the ast produced by the parser and I basically gave up. I like that Julia offers some macro-like techniques that replace a lot of the cases where one might use a macro for performance reasons.
- dunefox 5y ago
  > ~5 years ago Was the language even stable then?
  - adgjlsfhk1 5y ago
    5 years ago was right before 1.0, so a ton of stuff was broken since it was the last chance.
- StefanKarpinski 5y ago
  Tip: the first move when trying to write a macro is doing `Meta.@dump` on examples of argument expressions you want your macro to consume and produce. Then write code that transforms the inputs to the outputs.
  - dan-robertson 5y ago
    Right but it wasn’t obvious to me (at the time) what a change to the source code would do to the ast, so it wasn’t easy to know all the cases to handle, especially with quasiquotation (I think double-backtick style programming was basically impossible). For example, maybe you want to handle something that looks like: foo ~~> bar In lisp syntax (and recall that is what the Julia ast is: everything is a head and then arguments) it might look like: (~~> foo bar) ; or (op ~~> foo bar) But if you change to e.g. foo ~~> bar + 5 You might get (+ (~~> foo bar) 5) Or (~~> foo (progn (+ bar 5))) I don’t remember what you got or which cases were tricky, only that I could never guess what the output of dump would be.

127 points by IlyaOrson 5y ago

never understood why this was better/ how it was different from regular functions. just seems like bugs/vulns waiting to happen

medo-bear 5y ago
no one is proposing macros as a paradigm of programming. they simply give programmers expressive powers not afforded by use of regular functions. in a sense you can compare macros to c++ templating. know what you are doing and use sparingly
IlyaOrson 5y ago
It is not necessarily better in all cases and should not be overused: https://youtu.be/mSgXWpvQEHE?t=579 https://youtu.be/mSgXWpvQEHE?t=579 However, it is useful to provide a nicer syntax and DSLs. Some examples: https://stackoverflow.com/questions/58137512/why-use-macros-in-julia https://stackoverflow.com/questions/58137512/why-use-macros-... https://www.juliafordatascience.com/animations-with-plots-jl/ https://www.juliafordatascience.com/animations-with-plots-jl... https://gist.github.com/MikeInnes/8299575 https://gist.github.com/MikeInnes/8299575
bobbylarrybobby 5y ago
Being able to operate on expressions before they reach the compiler is very handy. Consider how ergonomic testing is thanks to macros: https://docs.julialang.org/en/v1/stdlib/Test/ https://docs.julialang.org/en/v1/stdlib/Test/ Here's an example of passing quasi-json to a plotting function: https://www.queryverse.org/VegaLite.jl/stable/userguide/vlplotmacro/ https://www.queryverse.org/VegaLite.jl/stable/userguide/vlpl... . This lets you essentially transliterate a VegaLite spec into Julia without needing to translate it into Julia. Finally, macros that operator on dataframes let you write code that looks kind of like SQL, and is much more pleasant than working with functions: https://dataframes.juliadata.org/stable/man/querying_frameworks/ https://dataframes.juliadata.org/stable/man/querying_framewo...
srcreigh 5y ago
Some macro systems can create variables in a loop for you. You could make this macro, (define-all i 5 0) ;; creates i1 i2 i3 i4 i5 initialized to 0 That's somewhat impossible with functions. The closest you get is either an array/dict with only runtime error checking, or an external codegen program. I wrote a post [0] about how to do this in Racket. The macro generates ORM code based given a SQLite DB. Aka the compiler queries SQLite and generates table-column functions automatically. More potential benefits are: Better static error messages (can implement a type system using macros, example here[1]), and controlling execution order (can add lazy computation semantics). [0]: http://tech.perpetua.io/2022/01/generating-sqlite-bindings-with-racket-macros/ http://tech.perpetua.io/2022/01/generating-sqlite-bindings-w... [1]: https://gist.github.com/srcreigh/f341b2adaa0fe37c241fdf15f37bdd2c https://gist.github.com/srcreigh/f341b2adaa0fe37c241fdf15f37...

leephillips 5y ago

This is an excellent, clear introduction to a topic that’s not easy to explain to beginners.

phoe-krk 5y ago

As always, I'm very glad to see that structural, Common Lisp-style macro systems with the whole language available for macro construction, have been successfully adopted in other languages to the point where it's possible to explain them without a single mention of Lisp in the article, or even better - where a mention of Lisp anywhere else except for the very beginning would make the article worse by making a unnecessary detour. pg's article on the topic, "What Made Lisp Different", [0] has aged poorly, and points 8 and 9 it makes (a notation for code using trees of symbols and the whole language always available) are no longer Lisp-specific. The final point, about "inventing a new dialect of Lisp", doesn't hold true either - as seen here, Julia is doing just fine not claiming to be another dialect of Lisp, even though many sources mention directly that it's Lisp-inspired. Congrats to Julia people for the macro system and to the author for the article! [0] http://www.paulgraham.com/diff.html http://www.paulgraham.com/diff.html

sischoel 5y ago
One could argue that Julia is just Lisp with a syntax that appeals more to popular taste. There is also a secret option to get into a lisp repl in Julia "julia --lisp".
- phoe-krk 5y ago
  Sure, that's always an option - same as Dylan, in a way!
  - borodi 5y ago
    Julia does take inspiration from Dylan and Scheme(the parser is even written in it) so it makes sense I guess
- dfghdfhs 5y ago
  > There is also a secret option to get into a lisp repl in Julia "julia --lisp". Wtf....... what? I just tried it, it's true. Is this some easter egg?

tlinker123 5y ago

Julia macros are pretty legit. Right up there with Rust macros.

xiaodai 5y ago

There are R meta programming techniques called non standard evaluation. It has many similarities to the macro system in Julia

krumbie 5y ago
That's true. However, I believe that many R programmers don't know when non-standard evaluation happens or what it is exactly. Functions with or without it cannot be told apart just by looking at the syntax. While NSE enables the dplyr syntax that many people enjoy, for me it's too magic and I have trouble reasoning about variable names in other people's code.
- coop_solution 5y ago
  What does dplyr syntax look like?
  - pdeffebach 5y ago
    Let's say you have a data frame df = tibble(a = c(1, 2)) and you want to use a dplyr verb to modify it mutate(df, b = a + 1) the `a` in the above expression refers to the column in `df`, but this means it's hard to reference a variable in the outer scope named `a`. Furthermore, if you have a string referring to the column name `"a"`, you can't simply write mutate(df, b = a_var + 1) Contrast this with DataFramesMeta.jl, which is a dply-like library for Julia, written with macros. df = DataFrame(a = [1, 2]) @transform df :b = :a .+ 1 Because of the use of Symbols, there is no ambiguity about scopes. To work with a variable referring to column `a` you can write a_str = "a" @transform df :b = $a_str .+ 1 I won't pretend this isn't more complicated or harder to learn. Some of the complexity is due to Julia's high performance limiting non-standard evaluation in subtle ways. But a core strength of Julia's macros is that it's easy to inspect these expressions and understand exactly what's going on, with `@macroexpand` as shown in the blog post. DataFramesMeta.jl repo: https://github.com/JuliaData/DataFramesMeta.jl https://github.com/JuliaData/DataFramesMeta.jl
    - 5y ago

tombert 5y ago

Macros were part of the "holy shit" moment for me for Lisp, in particular the Common Lisp Object System. I hadn't fully realized that it was possible to add a whole new paradigm to a language as a library [1], and moreover a particularly nice implementation of that paradigm. After that, I realized that macros aren't always something that needs to be avoided; in the right hands they're immensely powerful. I've only played a little with Julia macros, but it seems like they learned a lot of Lisps lessons, so I support it wholly. [1] I wasn't aware of how Objective C was built at the time.

dan-robertson 5y ago

The frustration of Julia macros for me was never knowing what AST would be produced for a given expression. This is a bit more manageable if you have a typed AST (e.g. OCaml but ppxes have other issues) or an obvious one (e.g. lisp). I like the way rust handles it where the macros operate on a tree of non-delimiter leaves and [delimiter, subtree list, delimiter] nodes which can allow for figuring out what the input to a macro will be more easily and for more varied macro syntax. Other languages that want a full AST before macros force the macro input to be a bit more AST-like, e.g. the Julia parser picks operator precedence and OCaml won’t let you use _ as an identifier. Maybe it is better now but when I looked at macros ~5 years ago some language update changed the ast produced by the parser and I basically gave up. I like that Julia offers some macro-like techniques that replace a lot of the cases where one might use a macro for performance reasons.

dunefox 5y ago
> ~5 years ago Was the language even stable then?
- adgjlsfhk1 5y ago
  5 years ago was right before 1.0, so a ton of stuff was broken since it was the last chance.
StefanKarpinski 5y ago
Tip: the first move when trying to write a macro is doing `Meta.@dump` on examples of argument expressions you want your macro to consume and produce. Then write code that transforms the inputs to the outputs.
- dan-robertson 5y ago
  Right but it wasn’t obvious to me (at the time) what a change to the source code would do to the ast, so it wasn’t easy to know all the cases to handle, especially with quasiquotation (I think double-backtick style programming was basically impossible). For example, maybe you want to handle something that looks like: foo ~~> bar In lisp syntax (and recall that is what the Julia ast is: everything is a head and then arguments) it might look like: (~~> foo bar) ; or (op ~~> foo bar) But if you change to e.g. foo ~~> bar + 5 You might get (+ (~~> foo bar) 5) Or (~~> foo (progn (+ bar 5))) I don’t remember what you got or which cases were tricky, only that I could never guess what the output of dump would be.