6 ms·

A Practical Intro to Macros in Rust

188 points by jxub 9y ago

solidsnack9000 9y ago
The colored diagrams use to explain contexts and hygienic macros are a neat idea.
kibwen 9y ago
Note that this dates back to Rust 1.0, and although the macro system hasn't changed since then, plans are in the works to succeed this original system ("macros 1.0") with an improved one ("macros 2.0"), culminating in the deprecation of the original. The two systems won't differ that much, so the documentation will remain mostly the same and the transition should be very straightforward. The impetus for this is that the macro system was unanimously considered the least mature feature of pre-1.0 Rust, but there was little will to delay Rust 1.0 even further solely for the sake of macros. Instead, some expedient band-aids were slapped over it during the run-up to 1.0, with the intent to eventually fix the whole thing properly when time allowed. To wit, the main problems with macros 1.0 were that macro "importing" is an ugly hack that stands out like a sore thumb from how modules work in the rest of the language; in addition, despite supposedly being hygenic, there are some edge cases where hygiene does not hold. The new system will make the language overall more consistent and reliable (and might bring some minor syntactic improvements), with the obvious downside that making the macro system nicer to use means that people might actually start using it. :P
- weinzierl 9y ago
  > plans are in the works to succeed this original system ("macros 1.0") with an improved one ("macros 2.0") Do you have any pointers where this work happens and I can follow along. Searching the RFC's for 'macro' turns up too much noise. Do you know about a timeline?
  - mzl 9y ago
    While not directly what you are looking for, the release milestone predictions for rust is a fairly nice page IMHO to see current plans: https://internals.rust-lang.org/t/rust-release-milestone-predictions/4591 https://internals.rust-lang.org/t/rust-release-milestone-pre... For example, one can see that "Macro use and rexport" (https://github.com/rust-lang/rust/issues/35896 https://github.com/rust-lang/rust/issues/35896) that will make macros a part of normal Rust naming and modules is planned for Rust 1.20 to be released end of August.
    - GolDDranks 9y ago
      Actually I think that prediction is now inaccurate, as they in your link "Macro use and rexport" say that they want to postpone the macro changes until Macros 2.0 are ready as a whole.
      - mzl 9y ago
        Oops, thanks for pointing it out. It was a while since I read the discussion on "Macro use and rexport". It is actually one feature I'm quite looking forward to, since I think proper naming for macros will make the whole eco-system simpler. Ah well, now we'll have to wait even longer.
- FrozenVoid 9y ago
  Unfortunate, because metaprogramming is very important to code design. I won't want to abandon code just because v2.0 will get deprecated and shiny v3.0 will become the new standard next year. If Rust team cared about backward compatibility they could preserve old code by adding something like -std=rust1.0 which would not break code. Forcing people to rewrite any code(esp. complex macros) will not gain Rust any goodwill longterm. https://github.com/rust-lang/rust/issues/39412 https://github.com/rust-lang/rust/issues/39412 > Eventually, I hope we can remove macro_rules!. That will take a long time, and would require a 2.0 version of Rust to strictly adhere to our stability guarantees.
  - scott_s 9y ago
    I don't think you're being charitable to the Rust team. They are quite vocal about how important they think API stability is. But they're also vocal about how important it is to get it right. For those reasons, they're upfront about what APIs are guaranteed to not change, and what APIs are still experimental. As an outsider, I think they do an excellent job of being transparent about the process.
    - im_down_w_otp 9y ago
      Having used Rust off and on for the better part of 4 years through a lot of tumult and change, I agree with you. In most cases the kinds of breaking changes and immaturities that the Rust core team communicates clearly to the ecosystem would be represented as nothing but silence and a giant dump of breaking changes suddenly in a lot of other ecosystems. Sometimes the Rust team seems to pay a price for transparency, in that it gives people visibility, and thus something to make an opinion about and/or complain about. In an information vacuum there's nothing to opine about except the vacuum itself. As annoyed as I often was, and occasionally still am, about different shapes of API stability in the language/libraries; I'm extremely happy the Rust team has maintained a commitment to developing everything out in the open.
  - steveklabnik 9y ago
    To be clear, the new macros will be different than the old ones, and the two systems will coexist. People will be encouraged not to use the older macro style, but it's not going to be removed, so your code will not break. There's no serious plans for a Rust 2.0, even though people bring it up from time to time.
Animats 9y ago
The intro to macros starts with a recursive macro. Not a good approach for an intro. Recursion in macros is usually more of a toy than a useful tool, anyway.
- Manishearth 9y ago
  No, it starts with a non recursive macro that does less stuff, and builds up to the recursive one. This resource is a "learning by example" resource, and it teaches the macro bit by bit starting from a simplified version and building up to the recursive and complex part, teaching concepts along the way. That's a great way of structuring it.
  - beefsack 9y ago
    I actually think it's great that it starts by showing something that looks really impressive, it immediately made me think of list comprehensions in Haskell. I was very keen to keep reading as soon as I saw that.
panic 9y ago
The concept of a "token tree" as a separate stage between ordinary tokenization and full parsing seems interesting on its own: was it based on another system or invented for Rust? For example, you could imagine implementing incremental parsing by writing your parser as a transformation on a token tree.
- nine_k 9y ago
  Isn't it how Lisp macros work?
  - db48x 9y ago
    Yep. Although in some lisps you can write reader macros that control tokenization to add new syntax.
  - panic 9y ago
    Yep! I have also been pointed at this PhD thesis: https://www.cs.utah.edu/~rafkind/papers/dissertation.pdf https://www.cs.utah.edu/~rafkind/papers/dissertation.pdf which defines a macro system on a uniform Algol-like syntax with intermediate "reading" and "enforestation" phases and corresponding syntax representations (the "read" representation being pretty much the same as the token tree representation here).
surrealize 9y ago
Cool! Any particular reason why you'd shuffle like that, as opposed to using a circular buffer? With IndexOffset, you've almost got a circular buffer going already.
shepmaster 9y ago
The author also created The Little Book of Rust Macros[1], which is an invaluable resource. They also are one of the top answerers for Rust questions on Stack Overflow[2]. Altogether a great asset for the community! [1]: https://danielkeep.github.io/tlborm/book/ https://danielkeep.github.io/tlborm/book/ [2]: https://stackoverflow.com/users/42353/dk?tab=answers https://stackoverflow.com/users/42353/dk?tab=answers
VermillionAzure 9y ago
Hi. I'm a lead developer on the Shaka Scheme project that hopes to accomplish implementing Scheme R7RS, which also has hygienic declarative macros. R7RS Scheme has `syntax-rules` which is very similar in structure to this macro system. It's quite elegant as it is, in my opinion, but I'm also wondering how good the debugging facilities for it are. Are they good?
- kibwen 9y ago
  Scheme's syntax-rules was the direct inspiration for Rust's macro system, actually. As for debugging, I don't write my own Rust macros very often (and when I do, they're not very extensive), but I'd characterize the experience as "okay". Rust reports errors in macros as originating from the unexpanded line of code, which is useful for macro users but less so for macro authors. But you can ask the Rust compiler to emit the code that it sees post-expansion, if you want to inspect the generated output directly.
  - Manishearth 9y ago
    There's a `trace_macros` macro that you can use for debugging. It's somewhat useful.
deleted 9y ago
[deleted]
jordigh 9y ago
I still find myself most comfortable with D's compile-time function evaluation (CTFE) and mixins. In D, one use of mixins is more or less an eval function: take any string and evaluate it as code at the invocation site. http://dlang.org/mixin.html http://dlang.org/mixin.html It's nice that you need very little new syntax to do compile-time tricks in D. The same syntax that works at runtime also works at compile time. If mixins worked with the AST instead of strings, it would be almost the same as Lisp macros. I'm also kind of happy to see that D seems to have inspired Rust macros a little.

188 points by jxub 9y ago

The colored diagrams use to explain contexts and hygienic macros are a neat idea.

Note that this dates back to Rust 1.0, and although the macro system hasn't changed since then, plans are in the works to succeed this original system ("macros 1.0") with an improved one ("macros 2.0"), culminating in the deprecation of the original. The two systems won't differ that much, so the documentation will remain mostly the same and the transition should be very straightforward. The impetus for this is that the macro system was unanimously considered the least mature feature of pre-1.0 Rust, but there was little will to delay Rust 1.0 even further solely for the sake of macros. Instead, some expedient band-aids were slapped over it during the run-up to 1.0, with the intent to eventually fix the whole thing properly when time allowed. To wit, the main problems with macros 1.0 were that macro "importing" is an ugly hack that stands out like a sore thumb from how modules work in the rest of the language; in addition, despite supposedly being hygenic, there are some edge cases where hygiene does not hold. The new system will make the language overall more consistent and reliable (and might bring some minor syntactic improvements), with the obvious downside that making the macro system nicer to use means that people might actually start using it. :P

weinzierl 9y ago
> plans are in the works to succeed this original system ("macros 1.0") with an improved one ("macros 2.0") Do you have any pointers where this work happens and I can follow along. Searching the RFC's for 'macro' turns up too much noise. Do you know about a timeline?
- mzl 9y ago
  While not directly what you are looking for, the release milestone predictions for rust is a fairly nice page IMHO to see current plans: https://internals.rust-lang.org/t/rust-release-milestone-predictions/4591 https://internals.rust-lang.org/t/rust-release-milestone-pre... For example, one can see that "Macro use and rexport" (https://github.com/rust-lang/rust/issues/35896 https://github.com/rust-lang/rust/issues/35896) that will make macros a part of normal Rust naming and modules is planned for Rust 1.20 to be released end of August.
  - GolDDranks 9y ago
    Actually I think that prediction is now inaccurate, as they in your link "Macro use and rexport" say that they want to postpone the macro changes until Macros 2.0 are ready as a whole.
    - mzl 9y ago
      Oops, thanks for pointing it out. It was a while since I read the discussion on "Macro use and rexport". It is actually one feature I'm quite looking forward to, since I think proper naming for macros will make the whole eco-system simpler. Ah well, now we'll have to wait even longer.
FrozenVoid 9y ago
Unfortunate, because metaprogramming is very important to code design. I won't want to abandon code just because v2.0 will get deprecated and shiny v3.0 will become the new standard next year. If Rust team cared about backward compatibility they could preserve old code by adding something like -std=rust1.0 which would not break code. Forcing people to rewrite any code(esp. complex macros) will not gain Rust any goodwill longterm. https://github.com/rust-lang/rust/issues/39412 https://github.com/rust-lang/rust/issues/39412 > Eventually, I hope we can remove macro_rules!. That will take a long time, and would require a 2.0 version of Rust to strictly adhere to our stability guarantees.
- scott_s 9y ago
  I don't think you're being charitable to the Rust team. They are quite vocal about how important they think API stability is. But they're also vocal about how important it is to get it right. For those reasons, they're upfront about what APIs are guaranteed to not change, and what APIs are still experimental. As an outsider, I think they do an excellent job of being transparent about the process.
  - im_down_w_otp 9y ago
    Having used Rust off and on for the better part of 4 years through a lot of tumult and change, I agree with you. In most cases the kinds of breaking changes and immaturities that the Rust core team communicates clearly to the ecosystem would be represented as nothing but silence and a giant dump of breaking changes suddenly in a lot of other ecosystems. Sometimes the Rust team seems to pay a price for transparency, in that it gives people visibility, and thus something to make an opinion about and/or complain about. In an information vacuum there's nothing to opine about except the vacuum itself. As annoyed as I often was, and occasionally still am, about different shapes of API stability in the language/libraries; I'm extremely happy the Rust team has maintained a commitment to developing everything out in the open.
- steveklabnik 9y ago
  To be clear, the new macros will be different than the old ones, and the two systems will coexist. People will be encouraged not to use the older macro style, but it's not going to be removed, so your code will not break. There's no serious plans for a Rust 2.0, even though people bring it up from time to time.

Animats 9y ago

The intro to macros starts with a recursive macro. Not a good approach for an intro. Recursion in macros is usually more of a toy than a useful tool, anyway.

Manishearth 9y ago
No, it starts with a non recursive macro that does less stuff, and builds up to the recursive one. This resource is a "learning by example" resource, and it teaches the macro bit by bit starting from a simplified version and building up to the recursive and complex part, teaching concepts along the way. That's a great way of structuring it.
- beefsack 9y ago
  I actually think it's great that it starts by showing something that looks really impressive, it immediately made me think of list comprehensions in Haskell. I was very keen to keep reading as soon as I saw that.

panic 9y ago

The concept of a "token tree" as a separate stage between ordinary tokenization and full parsing seems interesting on its own: was it based on another system or invented for Rust? For example, you could imagine implementing incremental parsing by writing your parser as a transformation on a token tree.

nine_k 9y ago
Isn't it how Lisp macros work?
- db48x 9y ago
  Yep. Although in some lisps you can write reader macros that control tokenization to add new syntax.
- panic 9y ago
  Yep! I have also been pointed at this PhD thesis: https://www.cs.utah.edu/~rafkind/papers/dissertation.pdf https://www.cs.utah.edu/~rafkind/papers/dissertation.pdf which defines a macro system on a uniform Algol-like syntax with intermediate "reading" and "enforestation" phases and corresponding syntax representations (the "read" representation being pretty much the same as the token tree representation here).

surrealize 9y ago

Cool! Any particular reason why you'd shuffle like that, as opposed to using a circular buffer? With IndexOffset, you've almost got a circular buffer going already.

shepmaster 9y ago

The author also created The Little Book of Rust Macros[1], which is an invaluable resource. They also are one of the top answerers for Rust questions on Stack Overflow[2]. Altogether a great asset for the community! [1]: https://danielkeep.github.io/tlborm/book/ https://danielkeep.github.io/tlborm/book/ [2]: https://stackoverflow.com/users/42353/dk?tab=answers https://stackoverflow.com/users/42353/dk?tab=answers

VermillionAzure 9y ago

Hi. I'm a lead developer on the Shaka Scheme project that hopes to accomplish implementing Scheme R7RS, which also has hygienic declarative macros. R7RS Scheme has `syntax-rules` which is very similar in structure to this macro system. It's quite elegant as it is, in my opinion, but I'm also wondering how good the debugging facilities for it are. Are they good?

kibwen 9y ago
Scheme's syntax-rules was the direct inspiration for Rust's macro system, actually. As for debugging, I don't write my own Rust macros very often (and when I do, they're not very extensive), but I'd characterize the experience as "okay". Rust reports errors in macros as originating from the unexpanded line of code, which is useful for macro users but less so for macro authors. But you can ask the Rust compiler to emit the code that it sees post-expansion, if you want to inspect the generated output directly.
- Manishearth 9y ago
  There's a `trace_macros` macro that you can use for debugging. It's somewhat useful.

deleted 9y ago

[deleted]

jordigh 9y ago

I still find myself most comfortable with D's compile-time function evaluation (CTFE) and mixins. In D, one use of mixins is more or less an eval function: take any string and evaluate it as code at the invocation site. http://dlang.org/mixin.html http://dlang.org/mixin.html It's nice that you need very little new syntax to do compile-time tricks in D. The same syntax that works at runtime also works at compile time. If mixins worked with the AST instead of strings, it would be almost the same as Lisp macros. I'm also kind of happy to see that D seems to have inspired Rust macros a little.