7 ms·
Can't you do this in Rust by writing a #[derivative] proc macro without having to modify the language? #[derivative] fn cube(x: f32) -> f32 { x * x * x
by fluffything 7y ago
Can't you do this in Rust by writing a #[derivative] proc macro without having to modify the language?
#[derivative]
fn cube(x: f32) -> f32 { x * x * x }
// expands to
// impl Derivative for cube {
// type Args = (f32,);
// type Return = f32;
// fn derivative(x: Self::Args) -> Self::Return {
// 3 * x * x
// }
// }
let cubeGrad = gradient(cube);
assert_eq!(cubeGrad(2), 12);
where `gradient` is just a normal Rust function:
fn gradient<F: Derivative>(_: F) -> impl Fn(F::Args) -> F::Return {
|x: F::Args| Derivative::derivative(x)
}
My original attempt was accidentally more powerful, in that it allowed `cube` to be a stateful function like a closure.
To implement this, the only thing one needs to do in the `#[derivative]` macro is:
* parse the function into an AST
* fold the AST using symbolic differentiation
* quasi quote the new AST
Swift can probably differentiate functions in other translation units because it keeps the bytecode of all functions around. A proc-macro based approach wouldn't be able to achieve that, at least, not in general, but a Rust plugin could, since plugins can drive compilation to, e.g., compile whatever function is being derived, and differentiate its bytecode instead of Rust, and if the function happens to be in a different translation unit, access their byte code.
- stefan_ 7y agoMaybe we can also make a #[returns] macro. This macro parses the function into an AST, invokes magic, and determines if the function will return on any input. I feel there is an impedance mismatch here.
- fluffything 7y agoWhat magic does need to be invoked by the #[derivative] proc macro ? For parsing Rust into an AST you just use syn, which also supports doing AST folds, and for semi-quoting you just use the quote crate. For the symbolic differentiation part, you can use any of the hundred symbolic differentiation libraries out there to obtain '3 * x * x' from 'x * x * x'. Pretty much any machine learning framework out there supports AD and symbolic differentiation in some form, by using some libraries already. Swift people are standardizing standard practice, not discovering the americas. If any of that looks like magic to you, then you might just be lacking the background.
- deleted 7y ago[deleted]
- MiroF 7y agoIs there a way to vectorize that and turn it into a GPU kernel using a macro?
- nestorD 7y agoAs Julia illustrates, you do not need to do the vectorization and GPU part within the differentiation. You can have a code using a GPU array library and just differentiate it (which ends up being more flexible / composable).
- fluffything 7y agoYou can write a #[cuda] proc macro that: - parses the Rust AST - folds it into a CUDA C AST - writes it to a temporary .cu file - compiles it with nvcc - links it into your Rust binary and that allows you to launch your function as a CUDA kernel. The Rust emu crate does this (more or less), but targets WebGPU instead of CUDA.
- nestorD 7y agoI would love to see a cleaned-up implementation of that (and I do have notes on the subject but no time to try and implement them). Note that to be useful you need backward differentiation (a function with one output but several outputs) and compatibility with tests and loops. See the Zygote.jl library (Julia) for a working example in a different language (their whitepaper is a great starting point if you want to implement something like that).
- jasode 7y ago>Can't you do this in Rust by writing a #[derivative] proc macro without having to modify the language? That question can be generalized to "since <Turing Complete language> can do <anything-we-want-via-library/function/macro/codegen/customcode/extension/whatever>, can't we just do that instead of modifying the core language?" Well yes, but the advantage of having 1st-class language syntax includes: + more helpful compiler error messages because it has intelligence of the semantics + more helpful IDE because the UI has intelligence about the intentions of the programmer + interoperability because programmers are all using the same syntax instead of Programmer Bob using his custom macro-dialect and Progammer Alice using her idiosyncratic macro-dialect.
- fluffything 7y ago> That question can be generalized to "since <Turing Complete language> can do <anything-we-want-via-library/function/macro/codegen/customcode/extension/whatever>, can't we just do that instead of modifying the core language?" Not really, since that usually requires using weird syntax or APIs, but the Rust proc-macro solution has the exact same syntax as the Swift one: @differentiable ...cube function let grad = gradient(at: cube); vs #[derivative] ...cube function.... let grad = gradient(cube);
- jasode 7y ago>, but the Rust proc-macro solution has the exact same syntax as the Swift one You're focusing on syntax but I was talking about the higher-level semantics. When programmers write custom code (e.g. custom macros), as far as the compiler/IDE is concerned, it's just an opaque string that it checks for valid syntax. Those tools have no "intelligence" of what the higher concepts of derivatives/deltas etc. E.g. the IntelliJ plugin for Rust will have no idea what a the higher semantics of a differentiable function is. It just sees a recursive "gradient()" which is just an arbitrary string of code to parse and it might as well be spelled "abcdefgzyxwvut123()". I.e. the blog post is talking about an symbiotic ecosystem of tools (e.g. compilers, IDEs, etc) that treats differentiable programming as a 1st-class language feature. The ultra flexibility of Rust or Lisp macros to reproduce the exact syntax doesn't really solve the semantics understood by the ecosystem of tools. edit reply to: >You seem to be at least suggesting that if one adds differentiable programming as a first class language feature, IDEs would automatically be intelligent about it, No, that's an uncharitable interpretation. I never claimed that IDEs will automagically understand higher level semantics with no extra work required to enhance the IDE's parser. I'm only saying that your macro proposal only recreates the syntax but not the semantics -- so you're really not fully solving what the blog author is talking about.