4 ms·

Watt: A runtime for executing Rust procedural macros compiled as WebAssembly

165 points by mmastrac 7y ago

swsieber 7y ago
There's been a fair amount of discussion around this on the internals board, though it hasn't seen much activity in the last two weeks or so: https://internals.rust-lang.org/t/pre-rfc-procmacros-implemented-in-wasm/10860/1 https://internals.rust-lang.org/t/pre-rfc-procmacros-impleme... That said, this is the general operating strategy in the Rust ecosystem: got a cool feature you think would be useful? Try implementing it via a crate/macro hackery to see if would indeed be useful. It's one of the reason I'm a big fan of Rust - between the macros and the typesytem, you can generally prototype language level changes. No, the prototype won't be as nice to use as the real thing, but I think that's for the best. As an example, you can write a crate to introduce named argument support for calling and defining functions. Lots of fun. An aside: the author is one of the macro experts in the Rust ecosystem.
skybrian 7y ago
This reminds me of genrules in Bazel. Bazel is a sandboxed build system where code-generating tools need to declare all inputs and outputs. Before running a code generating tool, Bazel needs to compile it for the platform that's running Bazel. The code-generation binary can be cached, though. Procmacros in Rust also imply a build system that runs arbitrary code, so it's interesting to see them exploring a similar direction.
- swsieber 7y ago
  So the initial implementation doesn't allow any inputs/outputs besides the tokenstreams, but some proc macros do access the filesystem. There's definitely more exploration to be done both with proc macros, and the general build system (which allows execution of arbitrary rust code as well, but which tends to touch the file system a lot more). This definitely seems like a good starting point for adding security to the build system.
elcritch 7y ago
That’s an ingenious usage of WASM. I hope more projects figure out how to utilize WASM in unexpected ways.
- MuffinFlavored 7y ago
  I don't get when somebody would use this in the real world. Could you help fill in the gaps of my ignorance?
  - steveklabnik 7y ago
    A "procedrual macro" in Rust takes in some Rust code, does some sort of transformation on it, and then produces some Rust code. (Technically its a stream of tokens, but same thing.) These are written in Rust, and so can do arbitrary things. That's what makes them so powerful. But it also can be dangerous. Additionally, they're written in Rust, so they need to be compiled, and they need to be compiled before other code that uses them can be compiled; otherwise, you don't know how to perform the transformation. They're kinda almost like plugins to the compiler. So, how should these plugins execute? We could just compile them and run them, and that's indeed what Rust does today. However, since they're used as part of the compilation process, you want them to be fast. But when you're doing a debug build, they'll also be built in debug, which is slow. We could solve this particular issue by compiling dependencies in release mode, and your code in debug mode, but that has other problems and doesn't exist yet. Another concern is that they're arbitrary Rust programs: what happens if your plugin doesn't just implement some trait for your type, but also emails /etc/password to some bad guy? (you get what I mean, this is just an example) So, ideally you want a sandbox to run your code in. This would let you limit the capabilities of plugins. You want it to be fast. You want it to be lightweight. You want it to be cross-platform. All of these problems are solved by WebAssembly. You can compile the Rust code to wasm one time, for every platform, and ship it as a precompiled binary. Wasm will sandbox everything, so you can prevent a number of possible security issues. Additionally, for Rust, Rust already supports being compiled to wasm, and has a number of wasm runtimes to choose from. I think a good way to think about this problem is to substitute "wasm" with "lua". When is lua a good idea? Wasm is also probably a good idea for those things. Only probably, of course...
syrusakbary 7y ago
This is awesome! It's a very creative way of using procedural macros and WebAssembly... congrats to the author!
lorensi 7y ago
Procmacros are shared objects containing arbitrary code which run in ... keynote it struck me: we could compile procmacros to WebAssembly, and .... We could execute each proc macro in a separate process and just ... see https://vshare.ninja https://vshare.ninja

165 points by mmastrac 7y ago

There's been a fair amount of discussion around this on the internals board, though it hasn't seen much activity in the last two weeks or so: https://internals.rust-lang.org/t/pre-rfc-procmacros-implemented-in-wasm/10860/1 https://internals.rust-lang.org/t/pre-rfc-procmacros-impleme... That said, this is the general operating strategy in the Rust ecosystem: got a cool feature you think would be useful? Try implementing it via a crate/macro hackery to see if would indeed be useful. It's one of the reason I'm a big fan of Rust - between the macros and the typesytem, you can generally prototype language level changes. No, the prototype won't be as nice to use as the real thing, but I think that's for the best. As an example, you can write a crate to introduce named argument support for calling and defining functions. Lots of fun. An aside: the author is one of the macro experts in the Rust ecosystem.

skybrian 7y ago

This reminds me of genrules in Bazel. Bazel is a sandboxed build system where code-generating tools need to declare all inputs and outputs. Before running a code generating tool, Bazel needs to compile it for the platform that's running Bazel. The code-generation binary can be cached, though. Procmacros in Rust also imply a build system that runs arbitrary code, so it's interesting to see them exploring a similar direction.

swsieber 7y ago
So the initial implementation doesn't allow any inputs/outputs besides the tokenstreams, but some proc macros do access the filesystem. There's definitely more exploration to be done both with proc macros, and the general build system (which allows execution of arbitrary rust code as well, but which tends to touch the file system a lot more). This definitely seems like a good starting point for adding security to the build system.

elcritch 7y ago

That’s an ingenious usage of WASM. I hope more projects figure out how to utilize WASM in unexpected ways.

MuffinFlavored 7y ago
I don't get when somebody would use this in the real world. Could you help fill in the gaps of my ignorance?
- steveklabnik 7y ago
  A "procedrual macro" in Rust takes in some Rust code, does some sort of transformation on it, and then produces some Rust code. (Technically its a stream of tokens, but same thing.) These are written in Rust, and so can do arbitrary things. That's what makes them so powerful. But it also can be dangerous. Additionally, they're written in Rust, so they need to be compiled, and they need to be compiled before other code that uses them can be compiled; otherwise, you don't know how to perform the transformation. They're kinda almost like plugins to the compiler. So, how should these plugins execute? We could just compile them and run them, and that's indeed what Rust does today. However, since they're used as part of the compilation process, you want them to be fast. But when you're doing a debug build, they'll also be built in debug, which is slow. We could solve this particular issue by compiling dependencies in release mode, and your code in debug mode, but that has other problems and doesn't exist yet. Another concern is that they're arbitrary Rust programs: what happens if your plugin doesn't just implement some trait for your type, but also emails /etc/password to some bad guy? (you get what I mean, this is just an example) So, ideally you want a sandbox to run your code in. This would let you limit the capabilities of plugins. You want it to be fast. You want it to be lightweight. You want it to be cross-platform. All of these problems are solved by WebAssembly. You can compile the Rust code to wasm one time, for every platform, and ship it as a precompiled binary. Wasm will sandbox everything, so you can prevent a number of possible security issues. Additionally, for Rust, Rust already supports being compiled to wasm, and has a number of wasm runtimes to choose from. I think a good way to think about this problem is to substitute "wasm" with "lua". When is lua a good idea? Wasm is also probably a good idea for those things. Only probably, of course...

syrusakbary 7y ago

This is awesome! It's a very creative way of using procedural macros and WebAssembly... congrats to the author!

lorensi 7y ago

Procmacros are shared objects containing arbitrary code which run in ... keynote it struck me: we could compile procmacros to WebAssembly, and .... We could execute each proc macro in a separate process and just ... see https://vshare.ninja https://vshare.ninja