wsmoses

5 ms·

1.
by wsmoses 6y ago
The name of the LLVM AD tool is actually Enzyme [ http://enzyme.mit.edu/ ] (Zygote is a Julia tool)
2.
by wsmoses 6y ago
Oh for sure, any ML framework worth its salt should do some amount of graph rewriting / transformations. I was (perhaps poorly) trying to explain how while yes AD (regardless of implementation in Enzyme, PyTorch, etc) _can_ avoid cachi
3.
by wsmoses 6y ago
Regarding differentiating python via CPython, theoretically yes, though practically it is likely more wise to use something like Numba which takes Python to LLVM directly to avoid a bunch of abstraction overhead that would otherwise have to
4.
by wsmoses 6y ago
We go into more details in the Limitations section of the paper, but in short Enzyme requires the following properties: * IR of active functions must be accessible when Enzyme is called (e.g. cannot differentiate dlopen'd functions) *
5.
by wsmoses 6y ago
Whoops added one too many zero’s there, agreed that would be really nice :P
6.
by wsmoses 6y ago
Enzyme does indeed handle mutable arrays (both in Enzyme.jl and any other frontend)! If you want to try it out forewarned that we're currently upgrading Enzyme.jl for better JIT integration (dynamic re-entry, custom derivative passthro
7.
by wsmoses 6y ago
Yeah my best guess at that is that they were trying to say you'd only need to store one value: the sum, rather than the two individual values -- but I'm not completely sure.
8.
by wsmoses 6y ago
Say you have some existing virus simulation codebase that you want to use ML on to derive an effective policy on. Without an AD tool like Enzyme, you'd have to spend significant time and effort understanding and rewriting that obnoxiou
9.
by wsmoses 6y ago
Reverse mode AD can always get into situations where it needs to store original values (i.e. network state). One advantage, however, of doing a more whole-program approach to AD rather than individual operators is that one might be able to
10.
by wsmoses 6y ago
Enzyme needs to be able to access the IR of any potentially active functions (calls that it deduced could impact the gradient) to be able to differentiate them. If all of the code you care about is in one compilation unit, you're immed
11.
by wsmoses 6y ago
You don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s to 1.
12.
by wsmoses 6y ago
I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs of the program if you know that derivative computation won&
13.
by wsmoses 6y ago
Oh man that was a fun hack to write. Basically we demonstrated an easy-to-setup AD on rust by leveraging link-time optimization (LTO) as a way to make sure Enzyme's generate derivatives "optimization pass" was run. We're
14.
by wsmoses 6y ago
For GPU's, there's a couple of different things that you might want to do. You can use existing tools within LLVM to automatically generate GPU code out of existing code, and this works perfectly fine, even running Enzyme first to
15.
by wsmoses 6y ago
Adding onto this, numerical derivatives have two potential problems which is why they tend not to be used in big scientific/ML frameworks. First of all they suffer from accuracy decay. For example if you were to do the standard f'
16.
by wsmoses 6y ago
Enzyme is named such as it's a tool that "synthesizes derivatives" and also as a pun referencing Zygote (another AD tool) since Enzyme operates at a lower level (LLVM rather than Julia).
17.
by wsmoses 6y ago
Hi all, another author here and happy to answer any questions! Some more relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips.cc/paper/2020&#x
18.
by wsmoses 6y ago
Hi all, author here. A couple of relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b... Project: