3 ms·
This is huge and I have high hopes for zygote,jl. Observation/request: For higher-order derivatives (Hessian, Laplacian, etc), AD libraries typically provide A
by jknz 7y ago
This is huge and I have high hopes for zygote,jl.
Observation/request: For higher-order derivatives (Hessian, Laplacian, etc), AD libraries typically provide API shortcuts. I have found it difficult to control or predict the memory footprint and the time complexity of these API shortcuts.
Laplacian is case in point: it is sometimes computed by first computing the Hessian by forward-over-reverse and then taking the trace. In most libraries you would have to dig deep in the documentation or the code itself to understand how intermediary values are accumulated in memory for instance.
I would love to have a summary table of the complexity of all these API shortcuts, similar to https://wiki.python.org/moin/TimeComplexity https://wiki.python.org/moin/TimeComplexity for data-structures, but for these AD shortcuts (laplacian, divergence, hessian, etc). The table would show memory and time complexity in terms of the number of the input/intermediary/output dimensions and the complexity of evaluating the original function.
- KenoFischer 7y agoUnfortunately, this isn't super easy since the time complexity heavily depends on what optimizations the compiler will apply and how the higher order AD is exactly implemented (there's many ways to do so that all give you the same answer and you probably want the system to pick the best for you). What we could do fairly easily however is to have a compiler introspection tool that runs through the computation, but doesn't do any of the work, just record how much work it would have done. That would easily give you and idea and let you plot a parameter sweep over any dimension you care about with actual results corresponding to the capabilities of the system.
- jknz 7y agoThanks for the reply. Such introspection tool would be useful indeed--it seems already possible to do some inspection by hand with https://fluxml.ai/Zygote.jl/latest/adjoints/#Gradient-Reflection-1 https://fluxml.ai/Zygote.jl/latest/adjoints/#Gradient-Reflec... Anyway, great work and I look forward to what happens next!