7 ms·

How to Write a Rust Syntax Extension

41 points by neverminder 12y ago

pron 12y ago
> Syntax extensions are a cool feature (and Rust is the only language I know of that supports something like this) that allows you to modify the AST at compile time, so you can generate your own code or modify existing code. Java has had this feature for about 10 years now. The standardized API (required by all conformant compilers) defines only reading a partial AST (just declaration) and generating new source files, but javac supports full AST transformations. Two famous Java projects making use of javac AST transformations are the good-old Lombok project and Checker, Java 8's new pluggable type systems.
- ajross 12y ago
  Lisp has had this feature for... never mind.
- nickmain 12y ago
  Haxe has a full AST available to its macros.
- dukerutledge 12y ago
  Haskell has it
JoshTriplett 12y ago
Great to see documentation on syntax extensions; hopefully this will end up in the official Rust book at some point. Also, this domain makes it really hard to take the content seriously...
- Argorak 12y ago
  Note that syntax extensions are behind a feature gate, that means they are an experimental feature usable on nightlies, but not on released versions of Rust.
  - steveklabnik 12y ago
    Yup, this is very much why I have put zero effort into documenting them: they deeply rely on compiler internals, and I have enough other work to do without helping create de-facto standardization, like what happened with macros... That said, still glad to see posts like this.
- gsingh93 12y ago
  Author here. I bought the domain my freshman year of college, when the "bro do you even lift" joke was going around. I think it's time to change it :)
  - JoshTriplett 12y ago
    It's a funny domain; it'd be great for memes or similar. It's just that using it for technical content makes you look like a "brogrammer", and I suspect you don't want that. Also, thank you for an awesome article. Any plans to try to get this document added to Rust documentation? Even if it's too unstable for the standard book, it'd be nice to have around as part of the documentation.
    - gsingh93 12y ago
      I don't personally have any plans to add it (too busy now, and not sure how to structure it), but since the documentation is part of the Rust repo, anyone can submit a pull request adding it. It'd probably good to contact Steve Klabnik and talk about how to structure it if you're interested in adding it.
ajross 12y ago
I can't help thinking that this is just the wrong way to do this. If I have a subtle metaprogramming problem that I'm going to solve with the C preprocessor, I can code it and document it right there in the same source where it's used. Because that's the natural place for it. The metacode is the fundamental thing I'm writing, and it belongs in the "code" files for the rest of the project. These Rust things are implemented as modules. I need to keep them separate. I need to build them separately. That breaks discoverability really badly. If a future maintainer needs to figure out what my strange macros do, they just look at them. If they get confused by my strange syntax extensions, they need to figure out where some other crate is coming from. Regardless of other concerns about the mechanics, I think this is going to put a really serious damper on this kind of code. Quite honestly, I'm looking at this and seriously thinking that more pedestrian options (python scripts to emit or preprocess rust code, for example) might be simpler and cleaner in practice.
- mbrubeck 12y ago
  Rust also has macros, which can easily be defined next to their use: http://doc.rust-lang.org/book/macros.html http://doc.rust-lang.org/book/macros.html Syntax extensions are more like plugins for the Rust compiler.
  - ajross 12y ago
    I know, but Rust macros seem to be fairly simple "expansion of a syntax tree" things, not intended as a programming environment. This is (or at least appears to be) the Rust answer for things like elaborate C macros or C++ template metaprogramming. There's a place for that kind of code in the world, even though it's needed rarely. Thinking about this as a "compiler plugin" is exactly what I think I'm getting at: if I need to implement some sort of DSL feature for my code, I want to do it as part of my code and not have to deploy it as a plugin to the compiler.
    - mbrubeck 12y ago
      Well, you can still have your syntax extension live in the same repo as the rest of your code. For example, the Servo repo has a "components/plugins" directory that contains our custom lints and other extensions. And other code in the repo just links to it like this: Cargo.toml: [dependencies.plugins] path = "../plugins" lib.rs: #![plugin(plugins)] This doesn't seem too different from the situation in C where you put your cpp macros in a header file so you can #include them in multiple source files.

41 points by neverminder 12y ago

pron 12y ago

> Syntax extensions are a cool feature (and Rust is the only language I know of that supports something like this) that allows you to modify the AST at compile time, so you can generate your own code or modify existing code. Java has had this feature for about 10 years now. The standardized API (required by all conformant compilers) defines only reading a partial AST (just declaration) and generating new source files, but javac supports full AST transformations. Two famous Java projects making use of javac AST transformations are the good-old Lombok project and Checker, Java 8's new pluggable type systems.

ajross 12y ago
Lisp has had this feature for... never mind.
nickmain 12y ago
Haxe has a full AST available to its macros.
dukerutledge 12y ago
Haskell has it

JoshTriplett 12y ago

Great to see documentation on syntax extensions; hopefully this will end up in the official Rust book at some point. Also, this domain makes it really hard to take the content seriously...

Argorak 12y ago
Note that syntax extensions are behind a feature gate, that means they are an experimental feature usable on nightlies, but not on released versions of Rust.
- steveklabnik 12y ago
  Yup, this is very much why I have put zero effort into documenting them: they deeply rely on compiler internals, and I have enough other work to do without helping create de-facto standardization, like what happened with macros... That said, still glad to see posts like this.
gsingh93 12y ago
Author here. I bought the domain my freshman year of college, when the "bro do you even lift" joke was going around. I think it's time to change it :)
- JoshTriplett 12y ago
  It's a funny domain; it'd be great for memes or similar. It's just that using it for technical content makes you look like a "brogrammer", and I suspect you don't want that. Also, thank you for an awesome article. Any plans to try to get this document added to Rust documentation? Even if it's too unstable for the standard book, it'd be nice to have around as part of the documentation.
  - gsingh93 12y ago
    I don't personally have any plans to add it (too busy now, and not sure how to structure it), but since the documentation is part of the Rust repo, anyone can submit a pull request adding it. It'd probably good to contact Steve Klabnik and talk about how to structure it if you're interested in adding it.

ajross 12y ago

I can't help thinking that this is just the wrong way to do this. If I have a subtle metaprogramming problem that I'm going to solve with the C preprocessor, I can code it and document it right there in the same source where it's used. Because that's the natural place for it. The metacode is the fundamental thing I'm writing, and it belongs in the "code" files for the rest of the project. These Rust things are implemented as modules. I need to keep them separate. I need to build them separately. That breaks discoverability really badly. If a future maintainer needs to figure out what my strange macros do, they just look at them. If they get confused by my strange syntax extensions, they need to figure out where some other crate is coming from. Regardless of other concerns about the mechanics, I think this is going to put a really serious damper on this kind of code. Quite honestly, I'm looking at this and seriously thinking that more pedestrian options (python scripts to emit or preprocess rust code, for example) might be simpler and cleaner in practice.

mbrubeck 12y ago
Rust also has macros, which can easily be defined next to their use: http://doc.rust-lang.org/book/macros.html http://doc.rust-lang.org/book/macros.html Syntax extensions are more like plugins for the Rust compiler.
- ajross 12y ago
  I know, but Rust macros seem to be fairly simple "expansion of a syntax tree" things, not intended as a programming environment. This is (or at least appears to be) the Rust answer for things like elaborate C macros or C++ template metaprogramming. There's a place for that kind of code in the world, even though it's needed rarely. Thinking about this as a "compiler plugin" is exactly what I think I'm getting at: if I need to implement some sort of DSL feature for my code, I want to do it as part of my code and not have to deploy it as a plugin to the compiler.
  - mbrubeck 12y ago
    Well, you can still have your syntax extension live in the same repo as the rest of your code. For example, the Servo repo has a "components/plugins" directory that contains our custom lints and other extensions. And other code in the repo just links to it like this: Cargo.toml: [dependencies.plugins] path = "../plugins" lib.rs: #![plugin(plugins)] This doesn't seem too different from the situation in C where you put your cpp macros in a header file so you can #include them in multiple source files.